chore(nix): add bun node_modules normalization scripts#1256
chore(nix): add bun node_modules normalization scripts#1256cjpais merged 3 commits intocjpais:mainfrom
Conversation
Add canonicalize-node-modules.ts and normalize-bun-binaries.ts to .nix/scripts/ for use by nixpkgs and other non-flake Nix builds. These scripts ensure deterministic symlink ordering inside node_modules/.bun/, which is required for reproducible fixed-output derivation hashes when using bun install --cpu="*" --os="*" to fetch all platform variants. Adapted from the opencode package (MIT license).
y0usaf
left a comment
There was a problem hiding this comment.
Tried these locally against Handy's current Bun install and found two spots where the scripts seem to change the install tree, rather than just normalize its ordering.
| list.sort((a, b) => { | ||
| const aValid = isValidSemver(a.version); | ||
| const bValid = isValidSemver(b.version); | ||
| if (aValid && bValid) return -Bun.semver.order(a.version, b.version); |
There was a problem hiding this comment.
I think this goes a bit further than normalization. This loop doesn't preserve the targets Bun picked; it re-selects them by highest semver / string order. When I tried it against Handy's current bun.lock, it rewired entries under node_modules/.bun/node_modules like minimatch (3.1.2 -> 9.0.5) and brace-expansion (1.1.12 -> 2.0.2). Since that directory is still on the upward resolution path, that can change what actually gets resolved at runtime. Could we make the link creation deterministic without changing which target each slug points to?
There was a problem hiding this comment.
Confirmed — reproduces exactly on a fresh bun install --linker=isolated (minimatch 3.1.2 → 9.0.5, brace-expansion 1.1.12 → 2.0.2). Fixed in 4236f7b: the script now reads the existing symlinks in .bun/node_modules/, removes them, and recreates them in sorted order with bun's exact targets — no re-selection.
| for (const name of topLevel) { | ||
| if (name === ".bin" || name === ".bun") continue; | ||
| const full = join(modulesRoot, name); | ||
| if (!(await isDirectory(full))) continue; |
There was a problem hiding this comment.
I think this is skipping most packages in an isolated Bun install. isDirectory() uses lstat, so symlinked package dirs don't count as directories here, and collectPackages() only walks the small set of real directories. On a fresh install of Handy's current lockfile, running this took the nested .bin links from 86 down to 27 and dropped entries like vite, parser, tsc, and eslint. Was the intent to follow package symlinks here as well?
There was a problem hiding this comment.
Confirmed — reproduced 86 → 27 on the current lockfile (losing vite, eslint, tsc, parser etc.). Fixed in 4236f7b by a simpler primitive: instead of re-deriving bins from package manifests, the script now reads the existing .bin/ symlinks bun produced and recreates them in sorted order with the same targets. This also avoids the side-effect of adding .bin/ entries bun deliberately omits for peer deps.
canonicalize-node-modules.ts previously re-selected .bun/node_modules/ targets by highest semver, silently rewiring runtime resolution (e.g. minimatch 3.1.2 -> 9.0.5, brace-expansion 1.1.12 -> 2.0.2). normalize-bun-binaries.ts used lstat in collectPackages(), which skipped symlinked peer-dep packages and dropped most .bin/ entries (86 -> 27 on Handy's current lockfile; vite, eslint, tsc, parser etc. were missing from nested node_modules/.bin/). Both scripts now read existing symlinks, remove them, and recreate them in lexicographic order with bun's exact targets. Verified on a fresh `bun install --linker=isolated` that all 958 symlinks are preserved byte-for-byte and NAR hash is unchanged. Reported by @y0usaf in cjpais#1256.
|
@kakapt could you take a look when you have a moment? @philocalyst could you verify the scripts produce the same hash on macOS? On Linux (x86_64, bun 1.3.3) I get: bun install --linker=isolated --force --frozen-lockfile --ignore-scripts --no-progress --cpu="*" --os="*"
bun --bun .nix/scripts/canonicalize-node-modules.ts
bun --bun .nix/scripts/normalize-bun-binaries.ts
nix hash path node_modules
# sha256-gREugUONMaECdqeEgLqzn6P/JNIq7zfUtRhclXhqdZc=If a fresh run on your mac yields a different hash, that tells us bun still picks different targets across platforms and we need a stronger normalization step. |
|
I got bun install v1.3.11 (af24e281)
+ @playwright/test@1.58.0
+ @tauri-apps/cli@2.10.0
+ @types/node@24.9.1
+ @types/react@18.3.26
+ @types/react-dom@18.3.7
+ @types/react-select@5.0.1
+ @typescript-eslint/eslint-plugin@8.49.0
+ @typescript-eslint/parser@8.49.0
+ @vitejs/plugin-react@4.7.0
+ eslint@9.39.1
+ eslint-plugin-i18next@6.1.3
+ prettier@3.6.2
+ typescript@5.6.3
+ vite@6.4.1
+ @tailwindcss/vite@4.1.16
+ @tauri-apps/api@2.10.1
+ @tauri-apps/plugin-autostart@2.5.1
+ @tauri-apps/plugin-clipboard-manager@2.3.2
+ @tauri-apps/plugin-dialog@2.6.0
+ @tauri-apps/plugin-fs@2.4.4
+ @tauri-apps/plugin-global-shortcut@2.3.1
+ @tauri-apps/plugin-opener@2.5.2
+ @tauri-apps/plugin-os@2.3.2
+ @tauri-apps/plugin-process@2.3.1
+ @tauri-apps/plugin-sql@2.3.1
+ @tauri-apps/plugin-store@2.4.1
+ @tauri-apps/plugin-updater@2.10.0
+ i18next@25.7.2
+ immer@11.1.3
+ lucide-react@0.542.0
+ react@18.3.1
+ react-dom@18.3.1
+ react-i18next@16.4.1
+ react-select@5.10.2
+ sonner@2.0.7
+ tailwindcss@4.1.16
+ tauri-plugin-macos-permissions-api@2.3.0
+ zod@3.25.76
+ zustand@5.0.8
658 packages installed [641.00ms]
[canonicalize-node-modules] rebuilt 320 links
[normalize-bun-binaries] rebuilt 87 links |
Temporary workflow for debugging cross-platform hash divergence in PR cjpais#1256. Runs a fresh isolated bun install on ubuntu-latest and macos-latest with pinned bun 1.3.11, executes our normalize scripts, and uploads file/symlink/.bin listings + NAR hash as artifacts so we can diff exactly what differs between the two platforms. Will be removed after data collection.
Temporary workflow for debugging cross-platform hash divergence in PR cjpais#1256. Runs a fresh isolated bun install on ubuntu-latest and macos-latest with pinned bun 1.3.11, executes our normalize scripts, and uploads file/symlink/.bin listings + NAR hash as artifacts so we can diff exactly what differs between the two platforms. Will be removed after data collection.
Makes `bun install --linker=isolated` bit-reproducible for the nixpkgs FOD by wiring three small passes behind a single entry point: canonicalize-node-modules.ts — sort .bun/node_modules/ symlinks heal-peer-dep-bins.ts — add missing peer-dep .bin entries normalize-bun-binaries.ts — sort per-package .bin/ symlinks The new heal pass is the non-obvious one: bun's isolated installer creates `.bin/<name>` entries by checking whether the target file exists at the moment the linker runs. For circular peer dependency pairs (the consumer's wait on the provider is skipped to avoid a deadlock), the two sides race, and the losing side silently drops one or more `.bin/` links. In practice this hits `update-browserslist-db`/`browserslist` and eslint ↔ eslint-utils, and makes the resulting `node_modules/` tree differ between Linux and macOS runs of the same bun.lock with the same bun version. We heal (add the intended entries) rather than strip, so the output matches what bun would produce without the race — and the pass becomes a no-op if upstream ever fixes it. Related upstream issue: oven-sh/bun#28147. Orchestrator `normalize-install.ts` is the single call-site used by package.nix; each sub-script is still runnable standalone for debugging. Verified cross-platform on bun 1.3.11 via GH Actions: x86_64-linux: sha256-DMhAdAzgh0O9cOSXDVspSykEfOF3TwNtbjOhPuFUOyw= aarch64-darwin: sha256-J7Z1CAZXy5+VNdYGusQgg1Pw1XZjAPdgpkdcVNSnghA= Diffs between the two are exclusively legitimate native-binary packages (@esbuild/*, @rollup/rollup-*, @tailwindcss/oxide-*, @tauri-apps/cli-*, lightningcss-*); the `.bin/` sets now match byte-for-byte.
|
Update: pushed a new revision that adds a third normalization pass — The two original scripts canonicalized symlink order, but it turned out Bun's isolated installer can also produce a different Verified on bun 1.3.11 via a GH Actions matrix:
Both sides now have 88 @philocalyst — this should match the hash you'll get if you run @y0usaf — if you have a minute, another pass would be appreciated. The two issues you originally flagged are addressed in @kakapt — would appreciate another look when you have a moment. |
The orchestrator at .nix/scripts/normalize-install.ts runs three small passes after `bun install --linker=isolated`: - canonicalize-node-modules.ts — sort .bun/node_modules/ symlinks - heal-peer-dep-bins.ts — add missing peer-dep .bin entries - normalize-bun-binaries.ts — sort per-package .bin/ symlinks Without this, bun's installer produces a node_modules/ tree that is not bit-reproducible for a fixed-output derivation: a timing race around circular peer dependencies (browserslist ↔ update-browserslist-db, eslint ↔ eslint-utils) drops random `.bin/` entries, and .bun symlink creation order is not stable. The orchestrator fixes both passes. Refresh x86_64-linux hash accordingly, reset the other three systems to lib.fakeHash so the hash-update workflow can repopulate them. src is temporarily pointed at xilec/Handy's PR branch so the scripts are present in the source tree; revert to the upstream tag once cjpais/Handy#1256 lands and a Handy release containing the scripts is cut.
The orchestrator at .nix/scripts/normalize-install.ts runs three small passes after `bun install --linker=isolated`: - canonicalize-node-modules.ts — sort .bun/node_modules/ symlinks - heal-peer-dep-bins.ts — add missing peer-dep .bin entries - normalize-bun-binaries.ts — sort per-package .bin/ symlinks Without this, bun's installer produces a node_modules/ tree that is not bit-reproducible for a fixed-output derivation: a timing race around circular peer dependencies (browserslist ↔ update-browserslist-db, eslint ↔ eslint-utils) drops random `.bin/` entries, and .bun symlink creation order is not stable. The orchestrator fixes both passes. Refresh x86_64-linux hash accordingly, reset the other three systems to lib.fakeHash so the hash-update workflow can repopulate them. src is temporarily pointed at xilec/Handy's PR branch so the scripts are present in the source tree; revert to the upstream tag once cjpais/Handy#1256 lands and a Handy release containing the scripts is cut.
GitHub exposes PR commits on the base repository, so we can pin the scripts-bearing commit directly via `owner = "cjpais"` rather than indirecting through xilec/Handy. The tarball is byte-identical either way, but the more direct path is less confusing for reviewers: any reader of package.nix only needs to know about cjpais/Handy#1256.
`bun install --linker=isolated` is not bit-reproducible on its own: symlink creation order in `.bun/node_modules/` and in each package's `.bin/` directory is not stable, and a timing race in bun's installer silently drops some `.bin/<peer>` entries around circular peer dependencies (e.g. `browserslist ↔ update-browserslist-db`, `eslint ↔ eslint-utils`). See `oven-sh/bun#28147` for the closest upstream tracker. Handy ships a tiny post-install orchestrator at `.nix/scripts/normalize-install.ts` (added in cjpais/Handy#1256) that runs three passes — canonicalize, heal, normalize — producing a stable, idempotent `node_modules/` tree. The buildPhase invokes that single entry point right after `bun install`. Within a platform the tree is now deterministic, but bun still only downloads the host-matching native binaries (esbuild, rollup, tauri-cli, lightningcss, tailwindcss-oxide, ...), so the hash differs between systems. Store one hash per `hostPlatform.system`; missing platforms `throw` rather than fall back to a placeholder. Currently populated: x86_64-linux sha256-tJ6LK99dELOiR0BcsTRTt/vLyNamntujLxhBy5Xl/lc= aarch64-linux sha256-S+dX6ZVgv9dexxIHoa5PxP7e0nxf/d7cKUGty5eEi8A= aarch64-darwin sha256-DQbogNBQ9izK5GPmoOudqiB2lJvct1vZI2U5lp3WFy8= `x86_64-darwin` is the only hole: GitHub Actions no longer hosts a free `macos-13` Intel image, so we cannot compute it without a self-hosted or paid runner. The throw-on-missing surfaces that fact immediately when the build is requested there. Also expose `frontendDeps` via `passthru` so the update script and other tooling can build it directly without dragging in the full handy compile. `src` is temporarily pinned to the HEAD commit of cjpais/Handy#1256 so the orchestrator scripts are actually present in the source tree; reverts to `tag = "v${finalAttrs.version}"` once that PR merges and a Handy release containing the scripts is cut.
|
Hi @cjpais — quick status update:
From my side the PR is ready to merge whenever you have a moment. |
|
Okay thanks for confirmation and update merging |
Before Submitting This PR
Human Written Description
These scripts are needed for the nixpkgs upstream packaging. nixpkgs stores bun dependencies as a fixed-output derivation, which requires
node_modules/to be byte-identical every time it is produced from the samebun.lock. Out of the boxbun install --linker=isolateddoes not give us that — two runs on different machines (and even two runs on the same machine) can differ in subtle ways. These scripts patch the output so the hash is stable.Having them live in the Handy repo — rather than in the nixpkgs tree — is the preferred approach and matches how opencode does it: any future bun/Handy bump can update the scripts in one place.
What the scripts do
Everything is wired behind a single entry point,
normalize-install.ts, which runs three small passes in order:canonicalize-node-modules.ts— rebuildsnode_modules/.bun/node_modules/symlinks in sorted order, preserving the exact targets bun picked.heal-peer-dep-bins.ts— adds any.bin/<peer>entries that bun's installer intended to create but skipped.normalize-bun-binaries.ts— rebuilds each package's private.bin/directory in sorted order, again preserving bun's targets.The orchestrator is what
package.nixin nixpkgs calls. Each sub-script is still runnable standalone for debugging.Why the heal pass exists (the non-obvious part)
Bun's isolated installer decides whether to create a
.bin/<name>link by checking, at the moment the linker runs, whether the target file exists on disk. For most dependencies that's fine because the installer waits for the provider package to finish before linking the consumer. But for circular peer dependencies (e.g.browserslist↔update-browserslist-db,eslint↔eslint-utils), that wait is skipped to avoid a deadlock — so the two sides race, and whichever one finishes last silently drops one or more.bin/entries.The outcome depends on CPU load, thread scheduling, filesystem latency, etc., so even the same
bun.lockwith the same bun version can produce different.bin/sets between Linux and macOS (and in principle between two runs on the same host). We've observed this concretely in Handy's install.The heal pass reads each package's
peerDependencies, finds the bins the peer declares, and creates any missing symlinks — producing the complete.bin/set that bun would have produced without the race. Advantages:Closest upstream tracker for the family of races: oven-sh/bun#28147.
Cross-platform verification
Verified on bun 1.3.11 via a temporary GitHub Actions matrix (ubuntu-latest + macos-latest):
node_modulesNAR hashsha256-DMhAdAzgh0O9cOSXDVspSykEfOF3TwNtbjOhPuFUOyw=sha256-J7Z1CAZXy5+VNdYGusQgg1Pw1XZjAPdgpkdcVNSnghA=Both sides have 88
.bin/entries (matching byte-for-byte); the remaining differences are only genuine native-binary packages (@esbuild/*,@rollup/rollup-*,@tailwindcss/oxide-*,@tauri-apps/cli-*,lightningcss-*). Re-running the orchestrator on the same tree does not change the hash.Because bun still downloads only the host-matching native binary set, the nixpkgs side stores one hash per system (linux + darwin) rather than a single cross-platform hash. That part lives in philocalyst/nixpkgs#1.
Related Issues/Discussions
Required by NixOS/nixpkgs#507754
Needed by philocalyst/nixpkgs#1
Upstream reference: oven-sh/bun#28147
Testing
AI Assistance
If AI was used: