Flatten views structure: reduce space complexity by 2 dimensions (#8258)

Part of #7585
Closes #5903

    Must be merged in coordination with Switch views repo docker-addonstore#36

Structure change

The current add-on views structure is too wide. It has storage complexity of O(numAddonFiles * numLanguages * numNVDAAPIVersions * numChannels). This should just be O(numAddonFiles).

This PR reduces the structure to O(numAddonFiles * numLanguages).

The current file structure for add-on views is:

views/en/2019.1.0/UIANotificationSwitch/stable.json

Generated from files committed to this branch as:

addons/AbsoluteYoutube/2026.2.0.json

File paths in the current views structure now serve as symlinks which point to files like addons/UIANotificationSwitch/2026.1/en.json which is generated from the translations in addons/AbsoluteYoutube/2026.2.0.json
Improvements
Size

The new views size is 264 MB, the old views size was 13GB.

Checked with: (Get-ChildItem . -Recurse -Force -File | Measure-Object Length -Sum).Sum).
Transform step speed

The old transform step took about 1.5-2h.
The new transform step takes about 10-15min.

Author: seanbudd

.github/workflows/testCode.yaml .github/workflows/testCode.yml.github/workflows/testCode.yaml renamed to .github/workflows/testCode.yml

@@ -10,7 +10,13 @@ concurrency:
 
 jobs:
   transformAndPush:
-    runs-on: windows-latest
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash
+    permissions:
+      contents: write
+      pull-requests: write
     env:
       # this is a git '--pretty=format' string
       # %h is SHA, %n is newline,
@@ -24,38 +30,72 @@ jobs:
       uses: actions/checkout@v5
       with:
         submodules: true
-    - name: Checkout views branch into separate folder
+    - name: Checkout addonstore-views repository
       uses: actions/checkout@v5
       with:
+        repository: nvaccess/addonstore-views
         path: views
-        ref: views
+        ref: main
+        token: ${{ secrets.VIEWS_PUSH_TOKEN }}
+    - name: Install system deps for lxml
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends \
+          libxml2-dev libxslt1-dev zlib1g-dev
     - name: Install the latest version of uv
       uses: astral-sh/setup-uv@v6
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
-        architecture: x86
     - name: Install requirements and run transformation
       run: |
+        set -euo pipefail
         uv sync
-        # empty the views git folder directory
-        Try { Remove-Item views/views/ -Recurse -ErrorAction stop }
-        Catch [System.Management.Automation.ItemNotFoundException] { $null }
-        uv run --directory transform python -m src.transform --loglevel ${{ env.logLevel }} nvdaAPIVersions.json ../addons ../views/views
+
+        # generate transformed data from this repository's addons metadata
+        rm -rf generated/
+        uv run --directory transform python -m src.transform --loglevel "${{ env.logLevel }}" nvdaAPIVersions.json ../addons ../generated
+
+        # replace generated data directories in addonstore-views
+        rm -rf views/addons/ views/views/
+        cp -R generated/addons views/addons
+        cp -R generated/views views/views
       env:
         logLevel: ${{ runner.debug && 'DEBUG' || 'INFO' }}
     - name: Copy files
       run: |
-        copy ./transform/nvdaAPIVersions.json ./views/nvdaAPIVersions.json
-        copy ./transform/docs/output.md ./views/output.md
-        copy ./readme.md ./views/readme.md
-    - name: Commit and push
+        set -euo pipefail
+        cp ./transform/nvdaAPIVersions.json ./views/nvdaAPIVersions.json
+        cp ./transform/docs/output.md ./views/output.md
+        cp ./README.md ./views/README.md
+    - name: Create PR and enable auto-merge
+      env:
+        GH_TOKEN: ${{ secrets.VIEWS_PUSH_TOKEN }}
       run: |
+        set -euo pipefail
         git log HEAD --pretty=format:"${{ env.COMMIT_FORMAT }}" -1 > commitMsg.txt
         cd views
         git config user.name github-actions
         git config user.email github-actions@github.com
         git add .
-        git commit -F ../commitMsg.txt
-        git push
+        if git diff --staged --quiet; then
+          echo "No generated changes; skipping PR creation."
+          exit 0
+        fi
+
+        branchName="transformViews${{ github.run_id }}-${{ github.run_attempt }}"
+        git checkout -b "$branchName"
+        # TODO: Temporarily suppress output during first commit as testing logs are too big.
+        # Remove quiet and gpg sign flags later.
+        git commit --quiet --no-gpg-sign -F ../commitMsg.txt
+        git push --set-upstream origin "$branchName"
+
+        prUrl=$(gh pr create \
+          --repo nvaccess/addonstore-views \
+          --base main \
+          --head "$branchName" \
+          --title "[Automated] Transform add-on data to views" \
+          --body-file ../commitMsg.txt)
+
+        gh pr merge --auto --squash --delete-branch "$prUrl"

.github/workflows/transformDataToViews.yml

@@ -1 +1 @@
-cpython-3.13-windows-x86_64-none
+cpython-3.13

.python-version

@@ -1,7 +1,7 @@
 # Add-on Store
 
-The addon-datastore repository is a data pipeline of submitting, validating and transforming add-on data to views.
-These views are hosted on the NV Access server and are available in the NVDA Add-on Store.
+The addon-datastore repository is a data pipeline for submitting and validating add-on data.
+Transformed add-on store data is produced with add-on files and symlinked views for hosting on the NV Access server.
 
 Please note: the NVDA project including the Add-on Store has a [Citizen and Contributor Code of Conduct](https://github.com/nvaccess/nvda/blob/master/CODE_OF_CONDUCT.md).
 NV Access expects that all contributors and other community members will read and abide by the rules set out in this document while participating in the project or contributing add-ons.

README.md

@@ -63,27 +63,34 @@ Aims:
   - While this is technically not necessary, it provides a good separation from implementation.
     If we wished to change our storage mechanism, we would not be breaking old versions of NVDA.
 
-
 ## API data generation details
 
-Triggered by a new commit to the `master` branch, [a GitHub workflow](../../.github/workflows/transformDataToViews.yml), [addon-datastore-transform](https://github.com/nvaccess/addon-datastore-transform), transforms the data into the required views.
+Triggered by a new commit to the `master` branch, [a GitHub workflow](../../.github/workflows/transformDataToViews.yml), and the [transform](../../transform/) module, transforms the data into add-on files and projected views.
 
 For each NVDA API version and channel, the add-on metadata with the highest version number is written.
 
-These views are then committed by the GitHub Action to the [views branch](https://github.com/nvaccess/addon-datastore/tree/views).
+This transformed data is then committed by the GitHub Action to the [addonstore-views](https://github.com/nvaccess/addonstore-views) main branch.
 
 ### Data views
-The following views will only be available on the [views branch](https://github.com/nvaccess/addon-datastore/tree/views) and located in a `views` folder.
-Required transformations of the data:
-- `/NVDA API Version/addon-1-ID/stable.json`
-- `/NVDA API Version/addon-1-ID/beta.json`
-- `/NVDA API Version/addon-2-ID/stable.json`
+
+The generated data is stored in two top-level folders:
+
+- `addons`: add-on data files by add-on version and language.
+- `views`: compatibility and latest projections as relative symlinks into `addons`.
+
+The following projected views are available in the `views` folder.
+
+The views folder is expected to have the following structure:
+
+`/views/<lang>/<NvdaAPIVersion>/<addonId>/<channel>.json`
 
 Notes:
+
 - 'NVDA API Version' will be something like '2019.3', there will be one folder for each NVDA API Version.
 - The `beta.json` and `stable.json` contain the information necessary for a store entry.
 - The contents for each add-on will include all the technical details required for NVDA to download, verify file integrity, and install.
-- The file will include translations (if available) for the displayable metadata.
+- View files are relative symlinks to files in `addons`.
+- Files use translations (if available) for displayable metadata.
 
 This simplifies the processing on the hosting (E.G NV Access) server.
 
@@ -116,16 +123,14 @@ Channel can be: all, dev, stable or beta.
 - Example: <https://addonStore.nvaccess.org/en/all/latest.json>
 
 ### `GET` cacheHash
+
 Returns a hash used for cache breaking.
 This hash will change whenever new add-on data is available.
-The hash should match the latest commit hash of the [views branch](https://github.com/nvaccess/addon-datastore/commits/views).
+For testing purposes, the hash should match the latest commit hash of the transformed [addonstore-views](https://github.com/nvaccess/addonstore-views) branch.
 
 - <https://addonStore.nvaccess.org/cacheHash.json>
 - Example return value: `"5fcf12f"`
 
-### Legacy
-This endpoint mirrors the legacy [get.php, from the addonFiles repository](https://github.com/nvaccess/addonFiles/blob/master/get.php).
-
 #### `addonslist` end-point
 
 The `addonslist` parameter generates a list of list of add-ons using the same logic as the [latest end-point](#get-latest) for all channels.

docs/design/designOverview.md

@@ -68,7 +68,6 @@ reportMissingTypeStubs = false
 
 [tool.uv]
 default-groups = "all"
-environments = ["sys_platform == 'win32'"]
 required-version = ">=0.8"
 
 [tool.setuptools]

pyproject.toml

@@ -1,10 +1,13 @@
 # Transforming data to views
 
-This repository primarily exists to transform data from [nvaccess/addon-datastore:master](https://github.com/nvaccess/addon-datastore) to views located at [nvaccess/addon-datastore:views](https://github.com/nvaccess/addon-datastore/tree/views).
+This module transforms add-on metadata into an output data layout with add-on files and views.
+The output is designed to be published from a single branch.
 
 For each NVDA version that needs to be supported by the add-on store, an entry must be added to [`nvdaAPIVersions.json`](./nvdaAPIVersions.json).
 This includes patch versions.
 
+This module should be run from linux, as symlinks are created for the server component.
+
 ## Overview
 
 For each version of NVDA, the meta-data of the most recent (the highest version number) of each add-on is automatically

transform/README.md

@@ -1,28 +1,55 @@
+# Output
+
 The output for running the transformation is described as follows.
-This is written to a given directory, and removes existing json data from the directory in the process.
+This is written to a given directory that must be new/empty; the transformation creates this directory and fails if it already exists.
+Callers are responsible for deleting any previous output directory before running the transformation.
 
 ## Output file structure
+
 The following subdirectories and files are created:
-- `/NVDA API Version/addon-1-ID/stable.json`
-- `/NVDA API Version/addon-1-ID/beta.json`
-- `/NVDA API Version/addon-2-ID/stable.json`
-eg: `/2020.3.0/nvdaOCR/stable.json`
+
+Translated add-on files:
+
+- `/addons/addon-1-ID/<addonVersion>/en.json`
+- `/addons/addon-1-ID/<addonVersion>/ar.json` (when a translation exists)
+- `/addons/addon-2-ID/<addonVersion>/en.json`
+
+Symlink end views which point to translated files:
+
+- `/views/en/<NvdaAPIVersion>/addon-1-ID/stable.json`
+- `/views/en/<NvdaAPIVersion>/addon-1-ID/beta.json`
+- `/views/en/<NvdaAPIVersion>/addon-2-ID/stable.json`
+- `/views/ar/<NvdaAPIVersion>addon-1-ID/stable.json`
+
+Examples:
+
+- `/addons/nvdaOCR/1.3.5/en.json`
+- `/views/en/2020.3.0/nvdaOCR/stable.json`
 
 Where `NVDA API Version` may be:
+
 - `2022.1.0`: A major release.
 - `2022.1.3`: A patch release.
 
-The system differentiates patch releases from major releases to cater to the (very unlikely) event of requireing a breaking change or introduction to the NVDA add-on API.
+The system differentiates patch releases from major releases to cater to the (very unlikely) event of requiring a breaking change or introduction to the NVDA add-on API.
 
 ## Output file data
-Each addon file is the addon data taken from input that is the latest compatible version, with the given requirements `(NVDA API Version, addon-ID, stable|beta|dev)`.
-The transformed data file content will be the same as the input.
-The contents for each addon file includes all the technical details required for NVDA to download, verify file integrity, and install.
+
+Add-on files in `/addons` contain transformed add-on metadata by add-on version and language.
+Views in `/views` are relative symlinks to files in `/addons`.
+
+For each required view `(language, NVDA API Version, addon-ID, channel)`, the view symlink points at a single file:
+
+- Prefer exact language translation
+- Otherwise prefer language without locale (`pt_BR` -> `pt`)
+- Otherwise fallback to `en`
+
+Each transformed add-on file includes all technical details required for NVDA to download, verify file integrity, and install.
 It also contains the information necessary for a store entry.
-Later, translated versions will become available.
 
 ## Output notes
+
 This structure simplifies the processing on the hosting (e.g. NV Access) server.
-To fetch the latest add-ons for `<NVDA API Version X>`, the server can concatenate the appropriate JSON files that match a glob: `/<NVDA API Version X>/*/stable.json`.
-Similarly, to fetch the latest version of an add-on with `<Addon-ID>` for `<NVDA API Version X>`. The server can return the data at `/<NVDA API Version X>/<addon-ID>/stable.json`.
+To fetch the latest add-ons for `<NVDA API Version X>`, the server can concatenate the appropriate JSON files that match a glob: `/views/<lang>/<NVDA API Version X>/*/stable.json`.
+Similarly, to fetch the latest version of an add-on with `<Addon-ID>` for `<NVDA API Version X>`, the server can return the data at `/views/<lang>/<NVDA API Version X>/<addon-ID>/stable.json`.
 Using the NV Access server as the endpoint for this is important in case the implementation has to change or be migrated away from GitHub for some reason.

transform/docs/output.md

@@ -25,7 +25,7 @@ def test_toStr_patch_optional(self):
 		"""Confirm that versions as string always include the patch number, 0 by default.
 
 		Even if the patch isn't specified, it should be included
-		so that the output is consistent - e.g. /views/2021.1.3/stable.json"""
+		so that the output is consistent - e.g. /views/en/2021.1.3/addonId/stable.json"""
 		self.assertEqual(str(MajorMinorPatch(13, 2)), "13.2.0")
 
 	def test_fromDict(self):