-
-
Notifications
You must be signed in to change notification settings - Fork 3.9k
refactor(nix): Managed Reproducibility. Managed Cache. #1601
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+1,412
−1,124
Closed
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
ddb3d47
feat(nix): replace monolithic pnpm.fetchDeps with per-package CAFS la…
lietblue 3e5bb34
fix(nix): use rm -rf to clean temp dir in cafs-add to handle read-onl…
lietblue baa58ae
fix(nix): chmod -R u+rwX before rm to handle 0555 tarball directories
lietblue 653a878
fix(nix): extract tarballs with --no-same-permissions to prevent mode…
lietblue f1d6842
fix(nix): chmod -R u+rwX after extraction to restore write bits strip…
lietblue f711ad7
fix(nix): two-pass extraction to handle tarballs with non-executable …
lietblue c0d20f4
perf(nix): stream-hash files in cafs-add to avoid loading entire file…
lietblue e75787c
fix(nix): loop tar+chmod until success to handle nested no-execute di…
lietblue efa2cc7
fix(nix): fix three merge-phase failures in pnpm store build
lietblue 393f079
fix(nix): align CAFS output with pnpm v10 store format
lietblue 886179a
fix(nix): add trailing newline to pkgStoresList to ensure last packag…
lietblue 55f867b
docs(nix): document pnpm CAFS store merging and trailing newline issue
lietblue 987b312
fix(nix): set execute permissions on CAFS -exec files in pnpmDeps
lietblue 7e0306c
fix: move chmod to buildPhase to ensure -exec files get execute permi…
lietblue 7e96b7a
chore: update nixpkgs
lietblue ad0f58d
fix(nix): align nix:gen pnpm-store with CAFS v10 merge layout
lietblue d7ffc42
chore(nix): remove nix.md
lietblue d209c32
refactor(nix): replace custom CAFS with @pnpm/store.cafs, output JSON
lietblue 85a3f98
refactor(nix): use @pnpm/constants for STORE_VERSION, remove pnpm fro…
lietblue 0f17442
refactor(nix): remove bundled .mjs, run .ts directly with node_modules
lietblue dd6b1c2
fix(nix): replace deprecated pnpm.configHook with pnpmConfigHook
lietblue f902008
fix(nix): include pnpm binary where pnpmConfigHook runs
lietblue 856cfb8
refactor(nix): replace generated JSON with IFD from pnpm-lock.yaml
lietblue 3364ab3
refactor(nix): derive cafsScript deps from lockfile graph, drop hardc…
lietblue 64e8c62
[autofix.ci] apply automated fixes
autofix-ci[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
| # Hand-written Nix derivation for per-package pnpm CAFS store. | ||
| # | ||
| # Architecture: | ||
| # 1. IFD: convert pnpm-lock.yaml → JSON via yj (tiny, cached after first eval) | ||
| # 2. Pure Nix: extract package metadata (name, version, url, integrity) | ||
| # 3. cafsScript: bundles the .ts source with its node_modules (from lockfile graph) | ||
| # 4. makePkgStore: per-package derivation that runs the .ts script via Node | ||
| # 5. Merge derivation: combines all fragments with cp -rn | ||
| # | ||
| # All pnpm internal format knowledge is in pnpm-cafs-add.ts (via @pnpm/store.cafs | ||
| # and @pnpm/constants). This Nix file is completely format-agnostic. | ||
| # | ||
| # NOTICE: This file uses IFD (Import From Derivation) to parse pnpm-lock.yaml | ||
| # at evaluation time. nix build allows IFD by default; --pure-eval (used by | ||
| # nix flake check) disables it — override with --allow-import-from-derivation | ||
| # or allow-import-from-derivation = true in nix.conf. | ||
| { | ||
| stdenvNoCC, | ||
| lib, | ||
| fetchurl, | ||
| nodejs, | ||
| runCommand, | ||
| yj, | ||
| }: | ||
| let | ||
| # --------------------------------------------------------------------------- | ||
| # IFD: parse pnpm-lock.yaml → JSON (sub-second, cached by Nix store) | ||
| # --------------------------------------------------------------------------- | ||
| lockfileJson = runCommand "pnpm-lock-json" { | ||
| nativeBuildInputs = [ yj ]; | ||
| } '' | ||
| yj < ${../pnpm-lock.yaml} > $out | ||
| ''; | ||
| lockfile = builtins.fromJSON (builtins.readFile lockfileJson); | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Pure Nix: extract package metadata from lockfile | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
| # Parse "name@version" key — last @ is the separator (handles @scope/name@version) | ||
| parseKey = key: | ||
| let | ||
| # builtins.match returns null on no match, or a list of capture groups | ||
| # "@scope/name@1.0.0" → ["@scope/name" "1.0.0"] | ||
| # "lodash@4.17.21" → ["lodash" "4.17.21"] | ||
| m = builtins.match "(.+)@([^@]+)" key; | ||
| in { | ||
| name = builtins.elemAt m 0; | ||
| version = builtins.elemAt m 1; | ||
| }; | ||
|
|
||
| # Compute npm registry tarball URL from package name and version. | ||
| # Scoped: @scope/name → registry.npmjs.org/@scope/name/-/name-version.tgz | ||
| # Unscoped: name → registry.npmjs.org/name/-/name-version.tgz | ||
| npmTarballUrl = name: version: | ||
| let | ||
| hasScope = lib.hasInfix "/" name; | ||
| unscopedName = if hasScope | ||
| then lib.last (lib.splitString "/" name) | ||
| else name; | ||
| in "https://registry.npmjs.org/${name}/-/${unscopedName}-${version}.tgz"; | ||
|
|
||
| # Build package entry from lockfile key + entry, or null if skipped | ||
| mkPkgEntry = key: entry: | ||
| let | ||
| parsed = parseKey key; | ||
| integrity = entry.resolution.integrity or null; | ||
| url = entry.resolution.tarball or (npmTarballUrl parsed.name parsed.version); | ||
| in | ||
| if (entry.bundled or false) || integrity == null | ||
| then null | ||
| else { | ||
| inherit (parsed) name version; | ||
| inherit url integrity; | ||
| }; | ||
|
|
||
| # Extract all non-null package entries from lockfile | ||
| rawEntries = lib.mapAttrs mkPkgEntry (lockfile.packages or {}); | ||
|
lietblue marked this conversation as resolved.
|
||
| packages = lib.filterAttrs (_: v: v != null) rawEntries; | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Runtime dependencies of nix/scripts/pnpm-cafs-add.ts, derived automatically | ||
| # from the lockfile. The nix/scripts workspace declares @pnpm/constants and | ||
| # @pnpm/store.cafs as direct deps; this section walks the lockfile's snapshot | ||
| # graph to compute the full transitive closure. No manual version list needed. | ||
| # --------------------------------------------------------------------------- | ||
| snapshots = lockfile.snapshots or {}; | ||
|
|
||
| # Read the nix/scripts workspace's resolved dependencies from the lockfile | ||
| scriptImporter = lockfile.importers."nix/scripts" or {}; | ||
| scriptDirectDeps = | ||
| lib.mapAttrsToList (name: entry: "${name}@${entry.version}") | ||
| (scriptImporter.dependencies or {}); | ||
|
|
||
| # Walk the snapshot dependency graph to collect the transitive closure. | ||
| # Each snapshot entry maps a package key to its runtime dependencies. | ||
| # Returns an attrset used as a visited set (keys = "name@version"). | ||
| depClosure = visited: queue: | ||
| if queue == [] then visited | ||
| else | ||
| let | ||
| key = builtins.head queue; | ||
| rest = builtins.tail queue; | ||
| in | ||
| if visited ? ${key} then depClosure visited rest | ||
| else | ||
| let | ||
| snap = snapshots.${key} or {}; | ||
| newDeps = lib.mapAttrsToList (n: v: "${n}@${v}") (snap.dependencies or {}); | ||
| in | ||
| depClosure (visited // { ${key} = true; }) (rest ++ newDeps); | ||
|
|
||
| # Full transitive closure, filtered to packages that exist in the lockfile | ||
| # (skips @types/* and other declaration-only packages without tarballs). | ||
| cafsScriptDeps = | ||
| builtins.filter (key: packages ? ${key}) | ||
| (builtins.attrNames (depClosure {} scriptDirectDeps)); | ||
|
|
||
| # Fetch a package tarball by its key | ||
| fetchPkg = key: | ||
| let pkg = packages.${key}; | ||
| in fetchurl { url = pkg.url; hash = pkg.integrity; }; | ||
|
|
||
| # Build a node_modules directory with the CAFS script's runtime dependencies. | ||
| # Extracts each transitive dep into a flat node_modules tree. | ||
| cafsScript = stdenvNoCC.mkDerivation { | ||
| name = "pnpm-cafs-add"; | ||
| dontUnpack = true; | ||
| dontConfigure = true; | ||
| dontInstall = true; | ||
| dontFixup = true; | ||
| buildPhase = '' | ||
| # Extract each runtime dependency into a flat node_modules | ||
| ${lib.concatStringsSep "\n" (map (key: | ||
| let | ||
| pkg = packages.${key}; | ||
| modPath = pkg.name; | ||
| parentDir = | ||
| if lib.hasPrefix "@" modPath | ||
| then "$out/node_modules/${lib.head (lib.splitString "/" modPath)}" | ||
| else "$out/node_modules"; | ||
| in '' | ||
| mkdir -p "${parentDir}" | ||
| mkdir -p "$out/node_modules/${modPath}" | ||
| tar xzf ${fetchPkg key} --strip-components=1 -C "$out/node_modules/${modPath}" | ||
| '' | ||
| ) cafsScriptDeps)} | ||
| cp ${./scripts/pnpm-cafs-add.ts} $out/pnpm-cafs-add.ts | ||
| ''; | ||
| }; | ||
|
|
||
| # Each package gets its own derivation: fetch tarball → write CAFS fragment. | ||
| # Runs the .ts source directly via Node's built-in TypeScript stripping. | ||
| makePkgStore = key: pkg: | ||
| let | ||
| drv = fetchurl { | ||
| url = pkg.url; | ||
| hash = pkg.integrity; | ||
| }; | ||
| in | ||
| stdenvNoCC.mkDerivation { | ||
| name = "airi-pnpm-pkg-${lib.replaceStrings [ "@" "/" ] [ "" "-" ] key}"; | ||
| nativeBuildInputs = [ nodejs ]; | ||
|
Weathercold marked this conversation as resolved.
|
||
| buildPhase = '' | ||
| node --experimental-strip-types \ | ||
| ${cafsScript}/pnpm-cafs-add.ts \ | ||
| "${drv}" \ | ||
| "$out" \ | ||
| "${key}" \ | ||
| "${pkg.integrity}" | ||
| ''; | ||
| dontConfigure = true; | ||
| dontUnpack = true; | ||
| dontInstall = true; | ||
| dontFixup = true; | ||
| }; | ||
|
|
||
| pkgStores = lib.mapAttrs makePkgStore packages; | ||
|
|
||
| in | ||
| # Merge all per-package CAFS fragments into one complete pnpm store. | ||
| # CAFS is content-addressed: cp -rn is safe (same hash = same file, no conflicts). | ||
| # | ||
| # NOTICE: pkgStoresList is passed via passAsFile instead of being interpolated | ||
| # directly into buildPhase. With ~3000 packages each path is ~50 chars, the | ||
| # concatenated string exceeds ARG_MAX (~2 MB) when bash is invoked with all | ||
| # environment variables, causing E2BIG. passAsFile writes the content to a | ||
| # temp file and sets pkgStoresListPath, keeping the env small. Nix still | ||
| # tracks the string context (i.e. dependencies on all pkgStores) correctly. | ||
| stdenvNoCC.mkDerivation { | ||
| name = "airi-pnpm-deps"; | ||
| passAsFile = [ "pkgStoresList" ]; | ||
| pkgStoresList = (lib.concatStringsSep "\n" (lib.attrValues pkgStores)) + "\n"; | ||
| buildPhase = '' | ||
| while IFS= read -r store; do | ||
| [ -z "$store" ] && continue | ||
| cp -rn --no-preserve=mode "$store/." "$out/" | ||
| done < "$pkgStoresListPath" | ||
| # NOTICE: CAFS files with -exec suffix denote executable binaries (e.g. turbo, esbuild). | ||
| # Set their permissions to 555 (r-xr-xr-x) so pnpm can execute them during install. | ||
| find "$out" -type f -name "*-exec" -exec chmod 555 {} + | ||
| ''; | ||
| dontConfigure = true; | ||
| dontUnpack = true; | ||
| dontInstall = true; | ||
| dontFixup = true; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "name": "@proj-airi/nix-scripts", | ||
| "version": "0.0.0", | ||
| "private": true, | ||
| "dependencies": { | ||
| "@pnpm/constants": "catalog:", | ||
| "@pnpm/store.cafs": "1000.1.4" | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.