refactor(nix): Managed Reproducibility. Managed Cache.#1601
refactor(nix): Managed Reproducibility. Managed Cache.#1601lietblue wants to merge 25 commits intomoeru-ai:mainfrom
Conversation
lietblue
changed the title
⏳ Approval required for deploying to Cloudflare Workers (Preview) for stage-web.
Hey, maintainers, kindly take some time to review and approve this deployment when you are available. Thank you! 🙏 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new, more granular Nix-based pnpm dependency management system that generates individual derivations for each package to improve caching. It includes a documentation file explaining the pnpm CAFS store merging process and scripts to automate the generation of these Nix expressions. Several improvements were identified in the review: the openssl dependency needs to be explicitly added to the Nix environment for tarball hashing, the find command in the merge phase should be made more robust using -exec to avoid potential failures with xargs, and the cafs-add.mjs script should use idiomatic Node.js writeFileSync instead of spawning shell processes for file operations.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a0f35679f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
Address PR moeru-ai#1601 review feedback from Weathercold: - Replace custom cafs-add.mjs with pnpm-cafs-add.ts using @pnpm/store.cafs (pnpm's own CAFS library), eliminating fragile reimplementation of pnpm internals that would break on store format changes - Generator now outputs pnpm-packages.json instead of generated .nix files; pnpm-store.nix is hand-written Nix that reads JSON via builtins.fromJSON - Use `find -exec {} +` instead of `xargs` in merge phase (Gemini review) - CI workflow: add --ignore-scripts to pnpm install (Codex review) - Bundle pnpm-cafs-add.ts with tsdown for self-contained Nix builds Deleted: nix/scripts/cafs-add.mjs, nix/pnpm-packages.nix Added: nix/scripts/pnpm-cafs-add.{ts,mjs}, nix/pnpm-packages.json
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42ecc54aa9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
I reworked the PR per your feedback: 1. Replaced custom CAFS logic with
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83b7684410
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
I prefer with IFD. Indeed, the overhead should be minimal (we currently have IFD as well) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b12b95d82
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
Rebase. |
Address PR moeru-ai#1601 review feedback from Weathercold: - Replace custom cafs-add.mjs with pnpm-cafs-add.ts using @pnpm/store.cafs (pnpm's own CAFS library), eliminating fragile reimplementation of pnpm internals that would break on store format changes - Generator now outputs pnpm-packages.json instead of generated .nix files; pnpm-store.nix is hand-written Nix that reads JSON via builtins.fromJSON - Use `find -exec {} +` instead of `xargs` in merge phase (Gemini review) - CI workflow: add --ignore-scripts to pnpm install (Codex review) - Bundle pnpm-cafs-add.ts with tsdown for self-contained Nix builds Deleted: nix/scripts/cafs-add.mjs, nix/pnpm-packages.nix Added: nix/scripts/pnpm-cafs-add.{ts,mjs}, nix/pnpm-packages.json
lietblue
force-pushed
the
nix-build-redesign-merge
branch
from
7b12b95 to
b0b52de
Compare
done |
…yers Previously, `pnpm.fetchDeps` was a single fixed-output derivation that downloaded all ~2990 packages in one shot. Any change to pnpm-lock.yaml invalidated the entire layer, forcing a full re-download. New approach: each package gets two independent Nix derivations — one `fetchurl` for the tarball download and one `makePkgStore` for assembling it into a pnpm CAFS v10 store fragment. The merge derivation combines all ~2990 fragments with `cp -rn` (safe since CAFS is content-addressed). Only changed packages need to be rebuilt. New files: - nix/scripts/cafs-add.mjs: pure Node.js script that extracts a tarball and writes pnpm CAFS v10 files/ and index/ entries - nix/scripts/gen-pnpm-packages.ts: parses pnpm-lock.yaml and generates nix/pnpm-packages.nix (~2990 fetchurl entries) and nix/pnpm-store.nix - nix/pnpm-packages.nix: auto-generated fetchurl derivations (DO NOT EDIT) - nix/pnpm-store.nix: auto-generated CAFS assembly + merge (DO NOT EDIT) Developer workflow after pnpm-lock.yaml changes: pnpm run nix:gen git add nix/pnpm-packages.nix nix/pnpm-store.nix nix build .#airi.pnpmDeps Removes nix/pnpm-deps-hash.txt and nix/update-pnpm-deps-hash.sh. Updates CI workflow to run pnpm run nix:gen instead.
… 0555 extraction failures
- cafs-add: chmod -R u+rwX unconditionally after tar extraction so that tarballs whose top-level dir is stored with a non-executable mode (e.g. 0600) don't cause EACCES in statSync/readdirSync during the walk phase. The extraction loop only called chmod on failure; tar can exit 0 while leaving dirs non-traversable because it applies final permissions after populating children. - cafs-add: chmod -R u+rwX in the finally block before rm -rf so that cleanup never hits EPERM on non-writable extracted directories. - pnpm-store: use passAsFile for the store-paths list to avoid E2BIG. Interpolating ~3000 store paths directly into buildPhase produces a ~150 KB env var that exceeds ARG_MAX when bash is exec'd. passAsFile writes the content to a temp file and exposes $pkgStoresListPath. - pnpm-store: add --no-preserve=mode to cp -rn so that Nix store dirs (mode 0555) are not copied verbatim into $out; the first package to create a shared subdir would otherwise lock out all subsequent packages with "Permission denied".
Two bugs caused pnpm to fail with ERR_PNPM_NO_OFFLINE_TARBALL even
though packages were present in the store:
1. Missing -exec suffix on executable CAFS files.
pnpm v10 CAFS requires executable files to be stored with a "-exec"
filename suffix (e.g. {hash}-exec). Without it, pnpm cannot locate
bin scripts and falls back to downloading, which fails offline.
Fix: append "-exec" to cafsName when isExec is true.
2. Stale checkedAt timestamp in package index JSON.
Setting checkedAt: 1 (epoch ms) caused pnpm to treat the entry as
never-verified and trigger integrity re-checks that could fail in the
sandbox. nixpkgs pnpm.fetchDeps removes this field entirely via
`jq "del(.. | .checkedAt?)"`. Fix: omit checkedAt from index entries.
3. Dynamic pnpm CAFS version subdirectory in merge phase.
Previously hardcoded "files/" at root; pnpm actually stores CAFS
under a versioned subdir (e.g. "v10/"). The merge derivation now adds
pnpm as a nativeBuildInput, runs `pnpm store path` once to detect the
version, and writes output under $out/$storeVer/. Per-package drv
hashes are unaffected (pnpm is not in makePkgStore's nativeBuildInputs).
…e is processed The while IFS= read -r loop skips the last line if it has no trailing newline. This caused the last package (zwitch@2.0.4) to not be copied into the merged pnpm store, resulting in only 2989/2990 packages being available. Adding a trailing newline to lib.concatStringsSep output ensures all packages are correctly merged into the final CAFS store.
CAFS v10 files with -exec suffix denote executable binaries (turbo, esbuild, etc). These need execute permission (mode 555) so pnpm can run them during install. Without this, pnpm install fails with EACCES when turbo or other native binaries are executed in the build sandbox.
- Generate passAsFile pkgStoresList loop (avoid ARG_MAX/E2BIG) - Detect $storeVer via pnpm store path; merge under $out/$storeVer - cp -rn --no-preserve=mode; chmod *-exec for native bins - Regenerate pnpm-packages.nix from lockfile (3014 packages) - Bump nix/assets-hash.txt
Made-with: Cursor
Address PR moeru-ai#1601 review feedback from Weathercold: - Replace custom cafs-add.mjs with pnpm-cafs-add.ts using @pnpm/store.cafs (pnpm's own CAFS library), eliminating fragile reimplementation of pnpm internals that would break on store format changes - Generator now outputs pnpm-packages.json instead of generated .nix files; pnpm-store.nix is hand-written Nix that reads JSON via builtins.fromJSON - Use `find -exec {} +` instead of `xargs` in merge phase (Gemini review) - CI workflow: add --ignore-scripts to pnpm install (Codex review) - Bundle pnpm-cafs-add.ts with tsdown for self-contained Nix builds Deleted: nix/scripts/cafs-add.mjs, nix/pnpm-packages.nix Added: nix/scripts/pnpm-cafs-add.{ts,mjs}, nix/pnpm-packages.json
…m merge Move all pnpm internal format knowledge into pnpm-cafs-add.ts: - Import STORE_VERSION from @pnpm/constants (currently "v10") - Write store version prefix and .fetcher-version inside the script - Merge derivation is now completely format-agnostic (just cp -rn) - Remove pnpm dependency and runtime detection from merge derivation - Remove storeVersion from pnpm-packages.json (no longer needed)
- Build node_modules from fetchurl tarballs in a Nix derivation (cafsScript) - Run pnpm-cafs-add.ts directly via Node's --experimental-strip-types - Delete the 142KB unauditable bundled pnpm-cafs-add.mjs - Generator now emits cafsScriptDeps list in JSON for Nix to consume - Remove nix:bundle-cafs script and eslint ignore for bundle
Add pnpm to nativeBuildInputs in common and package derivations so pnpmConfigHook can execute in the Nix sandbox without missing binary errors. Made-with: Cursor
Parse pnpm-lock.yaml directly in Nix via yj (Import From Derivation) instead of maintaining a committed 18K-line JSON file with a TypeScript generator and CI regeneration workflow. - Rewrite pnpm-store.nix to use IFD + pure Nix package extraction - Delete nix/pnpm-packages.json (~18,200 lines) - Delete nix/scripts/gen-pnpm-packages.ts (generator script) - Delete .github/workflows/update-nix-pnpm-deps-hash.yaml (CI workflow) - Remove nix:gen script from package.json pnpm-lock.yaml is now the single source of truth. The IFD derivation (yj < lockfile > $out) is sub-second and cached after first evaluation.
…oded list Replace the manually maintained cafsScriptDeps version list with a pure Nix lockfile graph walk (depClosure). The nix/scripts workspace declares @pnpm/constants and @pnpm/store.cafs; the Nix evaluator reads the lockfile importers + snapshots sections to compute the full transitive closure automatically. No manual updates needed when deps change.
lietblue
force-pushed
the
nix-build-redesign-merge
branch
from
b0b52de to
3364ab3
Compare
|
Why do we need such a complicated nix build to set up a pnpm project? We could just grab the binary directly, then use elfpatch to make nix happy, and that's it. This PR might remain just a PoC for now. Even though modifying a dependency only requires fetching a single package, still — why should we use nix to manage such a complex pnpm project? In a few days, I'll just use nix patch to package the binary — no need for assets, and no need for this huge blob of pnpmDeps. |
|
my bad |
One of the points of using nix is reproducible builds. Packaging binary build defeats that point You can add a |
4 participants
Super Nix. Our build system
REPRODUCIBILITY — IMMUTABILITY — INCREMENTAL BUILDS.
Our righteous path to build.
but build doesn't come free.
LOOKS FAMILIAR?
Full-store rebuilds. Multi‑gigabyte pnpm trees. CI nodes burning hours because one line in the lockfile moved. Things like these were happening all over the galaxy right now. Your monorepo could be next. Unless you can make the most important decision of your life.
Join the Nix Diver.
FREEDOM IS NOT FREE. DISK IS NOT EITHER.
The old world asked you to pay for the whole stack every time. We asked a different question: what if each package were its own verdict?
Prove to yourself that you have the strength and courage to be free — free from all-or-nothing rebuilds, free from shipping the same tarball work twice, free from watching Nix re-materialize civilization because someone bumped a patch version.
CI/CD changes
.github/workflows/update-nix-pnpm-deps-hash.yaml