Skip to content

chore(nix): add bun node_modules normalization scripts#1256

Merged
cjpais merged 3 commits intocjpais:mainfrom
xilec:nix/add-bun-normalize-scripts
Apr 19, 2026
Merged

chore(nix): add bun node_modules normalization scripts#1256
cjpais merged 3 commits intocjpais:mainfrom
xilec:nix/add-bun-normalize-scripts

Conversation

@xilec
Copy link
Copy Markdown
Contributor

@xilec xilec commented Apr 8, 2026

Before Submitting This PR

  • I have searched existing issues and pull requests (including closed ones) to ensure this isn't a duplicate
  • I have read CONTRIBUTING.md

Human Written Description

These scripts are needed for the nixpkgs upstream packaging. nixpkgs stores bun dependencies as a fixed-output derivation, which requires node_modules/ to be byte-identical every time it is produced from the same bun.lock. Out of the box bun install --linker=isolated does not give us that — two runs on different machines (and even two runs on the same machine) can differ in subtle ways. These scripts patch the output so the hash is stable.

Having them live in the Handy repo — rather than in the nixpkgs tree — is the preferred approach and matches how opencode does it: any future bun/Handy bump can update the scripts in one place.

What the scripts do

Everything is wired behind a single entry point, normalize-install.ts, which runs three small passes in order:

  1. canonicalize-node-modules.ts — rebuilds node_modules/.bun/node_modules/ symlinks in sorted order, preserving the exact targets bun picked.
  2. heal-peer-dep-bins.ts — adds any .bin/<peer> entries that bun's installer intended to create but skipped.
  3. normalize-bun-binaries.ts — rebuilds each package's private .bin/ directory in sorted order, again preserving bun's targets.

The orchestrator is what package.nix in nixpkgs calls. Each sub-script is still runnable standalone for debugging.

Why the heal pass exists (the non-obvious part)

Bun's isolated installer decides whether to create a .bin/<name> link by checking, at the moment the linker runs, whether the target file exists on disk. For most dependencies that's fine because the installer waits for the provider package to finish before linking the consumer. But for circular peer dependencies (e.g. browserslistupdate-browserslist-db, eslinteslint-utils), that wait is skipped to avoid a deadlock — so the two sides race, and whichever one finishes last silently drops one or more .bin/ entries.

The outcome depends on CPU load, thread scheduling, filesystem latency, etc., so even the same bun.lock with the same bun version can produce different .bin/ sets between Linux and macOS (and in principle between two runs on the same host). We've observed this concretely in Handy's install.

The heal pass reads each package's peerDependencies, finds the bins the peer declares, and creates any missing symlinks — producing the complete .bin/ set that bun would have produced without the race. Advantages:

  • It matches bun's intended behavior, not a stripped-down version of it.
  • It's idempotent: if bun ever fixes the race upstream, the script becomes a no-op and the hash stays the same.
  • It does not depend on Handy's own manifest — works for any bun install with peer dependencies.

Closest upstream tracker for the family of races: oven-sh/bun#28147.

Cross-platform verification

Verified on bun 1.3.11 via a temporary GitHub Actions matrix (ubuntu-latest + macos-latest):

system node_modules NAR hash
x86_64-linux sha256-DMhAdAzgh0O9cOSXDVspSykEfOF3TwNtbjOhPuFUOyw=
aarch64-darwin sha256-J7Z1CAZXy5+VNdYGusQgg1Pw1XZjAPdgpkdcVNSnghA=

Both sides have 88 .bin/ entries (matching byte-for-byte); the remaining differences are only genuine native-binary packages (@esbuild/*, @rollup/rollup-*, @tailwindcss/oxide-*, @tauri-apps/cli-*, lightningcss-*). Re-running the orchestrator on the same tree does not change the hash.

Because bun still downloads only the host-matching native binary set, the nixpkgs side stores one hash per system (linux + darwin) rather than a single cross-platform hash. That part lives in philocalyst/nixpkgs#1.

Related Issues/Discussions

Required by NixOS/nixpkgs#507754
Needed by philocalyst/nixpkgs#1
Upstream reference: oven-sh/bun#28147

Testing

  • Scripts are only used during Nix builds; no impact on regular development
  • Verified on Linux (local NixOS) and macOS (GitHub Actions): reproducible, idempotent, hashes stable
  • Each sub-script is standalone-runnable for debugging

AI Assistance

  • No AI was used in this PR
  • AI was used (please describe below)

If AI was used:

  • Tools used: Claude Code
  • How extensively: AI helped write and debug the scripts; the approach (healing circular peer-dep bins and canonicalizing symlink order) and the cross-platform verification methodology were designed and validated interactively.

Add canonicalize-node-modules.ts and normalize-bun-binaries.ts to
.nix/scripts/ for use by nixpkgs and other non-flake Nix builds.

These scripts ensure deterministic symlink ordering inside
node_modules/.bun/, which is required for reproducible fixed-output
derivation hashes when using bun install --cpu="*" --os="*" to fetch
all platform variants.

Adapted from the opencode package (MIT license).
Copy link
Copy Markdown
Contributor

@y0usaf y0usaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried these locally against Handy's current Bun install and found two spots where the scripts seem to change the install tree, rather than just normalize its ordering.

list.sort((a, b) => {
const aValid = isValidSemver(a.version);
const bValid = isValidSemver(b.version);
if (aValid && bValid) return -Bun.semver.order(a.version, b.version);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this goes a bit further than normalization. This loop doesn't preserve the targets Bun picked; it re-selects them by highest semver / string order. When I tried it against Handy's current bun.lock, it rewired entries under node_modules/.bun/node_modules like minimatch (3.1.2 -> 9.0.5) and brace-expansion (1.1.12 -> 2.0.2). Since that directory is still on the upward resolution path, that can change what actually gets resolved at runtime. Could we make the link creation deterministic without changing which target each slug points to?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — reproduces exactly on a fresh bun install --linker=isolated (minimatch 3.1.2 → 9.0.5, brace-expansion 1.1.12 → 2.0.2). Fixed in 4236f7b: the script now reads the existing symlinks in .bun/node_modules/, removes them, and recreates them in sorted order with bun's exact targets — no re-selection.

Comment thread .nix/scripts/normalize-bun-binaries.ts Outdated
for (const name of topLevel) {
if (name === ".bin" || name === ".bun") continue;
const full = join(modulesRoot, name);
if (!(await isDirectory(full))) continue;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is skipping most packages in an isolated Bun install. isDirectory() uses lstat, so symlinked package dirs don't count as directories here, and collectPackages() only walks the small set of real directories. On a fresh install of Handy's current lockfile, running this took the nested .bin links from 86 down to 27 and dropped entries like vite, parser, tsc, and eslint. Was the intent to follow package symlinks here as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — reproduced 86 → 27 on the current lockfile (losing vite, eslint, tsc, parser etc.). Fixed in 4236f7b by a simpler primitive: instead of re-deriving bins from package manifests, the script now reads the existing .bin/ symlinks bun produced and recreates them in sorted order with the same targets. This also avoids the side-effect of adding .bin/ entries bun deliberately omits for peer deps.

canonicalize-node-modules.ts previously re-selected .bun/node_modules/
targets by highest semver, silently rewiring runtime resolution
(e.g. minimatch 3.1.2 -> 9.0.5, brace-expansion 1.1.12 -> 2.0.2).

normalize-bun-binaries.ts used lstat in collectPackages(), which
skipped symlinked peer-dep packages and dropped most .bin/ entries
(86 -> 27 on Handy's current lockfile; vite, eslint, tsc, parser etc.
were missing from nested node_modules/.bin/).

Both scripts now read existing symlinks, remove them, and recreate
them in lexicographic order with bun's exact targets. Verified on a
fresh `bun install --linker=isolated` that all 958 symlinks are
preserved byte-for-byte and NAR hash is unchanged.

Reported by @y0usaf in cjpais#1256.
@xilec
Copy link
Copy Markdown
Contributor Author

xilec commented Apr 14, 2026

@kakapt could you take a look when you have a moment?

@philocalyst could you verify the scripts produce the same hash on macOS? On Linux (x86_64, bun 1.3.3) I get:

bun install --linker=isolated --force --frozen-lockfile --ignore-scripts --no-progress --cpu="*" --os="*"
bun --bun .nix/scripts/canonicalize-node-modules.ts
bun --bun .nix/scripts/normalize-bun-binaries.ts
nix hash path node_modules
# sha256-gREugUONMaECdqeEgLqzn6P/JNIq7zfUtRhclXhqdZc=

If a fresh run on your mac yields a different hash, that tells us bun still picks different targets across platforms and we need a stronger normalization step.

@philocalyst
Copy link
Copy Markdown

I got sha256-H54AsWPGku8qHx1OvkhD6fWFRU4QcRAnTbHa+gFWi18= ... @xilec

bun install v1.3.11 (af24e281)

+ @playwright/test@1.58.0
+ @tauri-apps/cli@2.10.0
+ @types/node@24.9.1
+ @types/react@18.3.26
+ @types/react-dom@18.3.7
+ @types/react-select@5.0.1
+ @typescript-eslint/eslint-plugin@8.49.0
+ @typescript-eslint/parser@8.49.0
+ @vitejs/plugin-react@4.7.0
+ eslint@9.39.1
+ eslint-plugin-i18next@6.1.3
+ prettier@3.6.2
+ typescript@5.6.3
+ vite@6.4.1
+ @tailwindcss/vite@4.1.16
+ @tauri-apps/api@2.10.1
+ @tauri-apps/plugin-autostart@2.5.1
+ @tauri-apps/plugin-clipboard-manager@2.3.2
+ @tauri-apps/plugin-dialog@2.6.0
+ @tauri-apps/plugin-fs@2.4.4
+ @tauri-apps/plugin-global-shortcut@2.3.1
+ @tauri-apps/plugin-opener@2.5.2
+ @tauri-apps/plugin-os@2.3.2
+ @tauri-apps/plugin-process@2.3.1
+ @tauri-apps/plugin-sql@2.3.1
+ @tauri-apps/plugin-store@2.4.1
+ @tauri-apps/plugin-updater@2.10.0
+ i18next@25.7.2
+ immer@11.1.3
+ lucide-react@0.542.0
+ react@18.3.1
+ react-dom@18.3.1
+ react-i18next@16.4.1
+ react-select@5.10.2
+ sonner@2.0.7
+ tailwindcss@4.1.16
+ tauri-plugin-macos-permissions-api@2.3.0
+ zod@3.25.76
+ zustand@5.0.8

658 packages installed [641.00ms]
[canonicalize-node-modules] rebuilt 320 links
[normalize-bun-binaries] rebuilt 87 links

xilec added a commit to xilec/Handy that referenced this pull request Apr 14, 2026
Temporary workflow for debugging cross-platform hash divergence in
PR cjpais#1256. Runs a fresh isolated bun install on ubuntu-latest and
macos-latest with pinned bun 1.3.11, executes our normalize scripts,
and uploads file/symlink/.bin listings + NAR hash as artifacts so we
can diff exactly what differs between the two platforms.

Will be removed after data collection.
xilec added a commit to xilec/Handy that referenced this pull request Apr 14, 2026
Temporary workflow for debugging cross-platform hash divergence in
PR cjpais#1256. Runs a fresh isolated bun install on ubuntu-latest and
macos-latest with pinned bun 1.3.11, executes our normalize scripts,
and uploads file/symlink/.bin listings + NAR hash as artifacts so we
can diff exactly what differs between the two platforms.

Will be removed after data collection.
Makes `bun install --linker=isolated` bit-reproducible for the nixpkgs FOD
by wiring three small passes behind a single entry point:

  canonicalize-node-modules.ts  — sort .bun/node_modules/ symlinks
  heal-peer-dep-bins.ts         — add missing peer-dep .bin entries
  normalize-bun-binaries.ts     — sort per-package .bin/ symlinks

The new heal pass is the non-obvious one: bun's isolated installer creates
`.bin/<name>` entries by checking whether the target file exists at the
moment the linker runs. For circular peer dependency pairs (the consumer's
wait on the provider is skipped to avoid a deadlock), the two sides race,
and the losing side silently drops one or more `.bin/` links. In practice
this hits `update-browserslist-db`/`browserslist` and eslint ↔
eslint-utils, and makes the resulting `node_modules/` tree differ between
Linux and macOS runs of the same bun.lock with the same bun version.

We heal (add the intended entries) rather than strip, so the output
matches what bun would produce without the race — and the pass becomes a
no-op if upstream ever fixes it. Related upstream issue: oven-sh/bun#28147.

Orchestrator `normalize-install.ts` is the single call-site used by
package.nix; each sub-script is still runnable standalone for debugging.

Verified cross-platform on bun 1.3.11 via GH Actions:
  x86_64-linux:   sha256-DMhAdAzgh0O9cOSXDVspSykEfOF3TwNtbjOhPuFUOyw=
  aarch64-darwin: sha256-J7Z1CAZXy5+VNdYGusQgg1Pw1XZjAPdgpkdcVNSnghA=

Diffs between the two are exclusively legitimate native-binary packages
(@esbuild/*, @rollup/rollup-*, @tailwindcss/oxide-*, @tauri-apps/cli-*,
lightningcss-*); the `.bin/` sets now match byte-for-byte.
@xilec
Copy link
Copy Markdown
Contributor Author

xilec commented Apr 15, 2026

Update: pushed a new revision that adds a third normalization pass — heal-peer-dep-bins.ts — and a single orchestrator entry point (normalize-install.ts).

The two original scripts canonicalized symlink order, but it turned out Bun's isolated installer can also produce a different .bin/ set across machines. Cause is a timing race in how it links .bin/<peer> entries for circular peer-dependency pairs (e.g. browserslist ↔ update-browserslist-db, eslint ↔ eslint-utils) — whichever side finishes last silently drops some entries. See the header of .nix/scripts/heal-peer-dep-bins.ts for the full write-up, and oven-sh/bun#28147 for the closest upstream tracker. The new pass heals the set by reading peerDependencies and adding back the intended entries — idempotent, and becomes a no-op if bun ever fixes the race.

Verified on bun 1.3.11 via a GH Actions matrix:

  • x86_64-linux: sha256-DMhAdAzgh0O9cOSXDVspSykEfOF3TwNtbjOhPuFUOyw=
  • aarch64-darwin: sha256-J7Z1CAZXy5+VNdYGusQgg1Pw1XZjAPdgpkdcVNSnghA=

Both sides now have 88 .bin/ entries matching byte-for-byte; remaining diffs are only genuine native-binary packages (@esbuild/*, @rollup/rollup-*, @tailwindcss/oxide-*, @tauri-apps/cli-*, lightningcss-*). This is the reason philocalyst/nixpkgs#1 stores one hash per system rather than a single universal hash.

@philocalyst — this should match the hash you'll get if you run bun --bun .nix/scripts/normalize-install.ts after bun install --linker=isolated on aarch64-darwin. The previous 87-entry output you saw lines up with one .bin/browserslist that the race was dropping.

@y0usaf — if you have a minute, another pass would be appreciated. The two issues you originally flagged are addressed in 4236f7b (the scripts now read and preserve bun's chosen targets instead of re-deriving them), and the new commit on top adds the heal pass for the cross-platform race we later found. Would be good to confirm the three-pass flow looks sensible end-to-end.

@kakapt — would appreciate another look when you have a moment.

xilec added a commit to xilec/nixpkgs that referenced this pull request Apr 15, 2026
The orchestrator at .nix/scripts/normalize-install.ts runs three small
passes after `bun install --linker=isolated`:
  - canonicalize-node-modules.ts — sort .bun/node_modules/ symlinks
  - heal-peer-dep-bins.ts        — add missing peer-dep .bin entries
  - normalize-bun-binaries.ts    — sort per-package .bin/ symlinks

Without this, bun's installer produces a node_modules/ tree that is not
bit-reproducible for a fixed-output derivation: a timing race around
circular peer dependencies (browserslist ↔ update-browserslist-db,
eslint ↔ eslint-utils) drops random `.bin/` entries, and .bun symlink
creation order is not stable. The orchestrator fixes both passes.

Refresh x86_64-linux hash accordingly, reset the other three systems to
lib.fakeHash so the hash-update workflow can repopulate them.

src is temporarily pointed at xilec/Handy's PR branch so the scripts
are present in the source tree; revert to the upstream tag once
cjpais/Handy#1256 lands and a Handy release containing the scripts
is cut.
xilec added a commit to xilec/nixpkgs that referenced this pull request Apr 15, 2026
The orchestrator at .nix/scripts/normalize-install.ts runs three small
passes after `bun install --linker=isolated`:
  - canonicalize-node-modules.ts — sort .bun/node_modules/ symlinks
  - heal-peer-dep-bins.ts        — add missing peer-dep .bin entries
  - normalize-bun-binaries.ts    — sort per-package .bin/ symlinks

Without this, bun's installer produces a node_modules/ tree that is not
bit-reproducible for a fixed-output derivation: a timing race around
circular peer dependencies (browserslist ↔ update-browserslist-db,
eslint ↔ eslint-utils) drops random `.bin/` entries, and .bun symlink
creation order is not stable. The orchestrator fixes both passes.

Refresh x86_64-linux hash accordingly, reset the other three systems to
lib.fakeHash so the hash-update workflow can repopulate them.

src is temporarily pointed at xilec/Handy's PR branch so the scripts
are present in the source tree; revert to the upstream tag once
cjpais/Handy#1256 lands and a Handy release containing the scripts
is cut.
xilec added a commit to xilec/nixpkgs that referenced this pull request Apr 15, 2026
GitHub exposes PR commits on the base repository, so we can pin the
scripts-bearing commit directly via `owner = "cjpais"` rather than
indirecting through xilec/Handy. The tarball is byte-identical either
way, but the more direct path is less confusing for reviewers: any
reader of package.nix only needs to know about cjpais/Handy#1256.
xilec added a commit to xilec/nixpkgs that referenced this pull request Apr 15, 2026
`bun install --linker=isolated` is not bit-reproducible on its own:
symlink creation order in `.bun/node_modules/` and in each package's
`.bin/` directory is not stable, and a timing race in bun's installer
silently drops some `.bin/<peer>` entries around circular peer
dependencies (e.g. `browserslist ↔ update-browserslist-db`,
`eslint ↔ eslint-utils`). See `oven-sh/bun#28147` for the closest
upstream tracker.

Handy ships a tiny post-install orchestrator at
`.nix/scripts/normalize-install.ts` (added in cjpais/Handy#1256) that
runs three passes — canonicalize, heal, normalize — producing a stable,
idempotent `node_modules/` tree. The buildPhase invokes that single
entry point right after `bun install`.

Within a platform the tree is now deterministic, but bun still only
downloads the host-matching native binaries (esbuild, rollup,
tauri-cli, lightningcss, tailwindcss-oxide, ...), so the hash differs
between systems. Store one hash per `hostPlatform.system`; missing
platforms `throw` rather than fall back to a placeholder.

Currently populated:
  x86_64-linux   sha256-tJ6LK99dELOiR0BcsTRTt/vLyNamntujLxhBy5Xl/lc=
  aarch64-linux  sha256-S+dX6ZVgv9dexxIHoa5PxP7e0nxf/d7cKUGty5eEi8A=
  aarch64-darwin sha256-DQbogNBQ9izK5GPmoOudqiB2lJvct1vZI2U5lp3WFy8=

`x86_64-darwin` is the only hole: GitHub Actions no longer hosts a
free `macos-13` Intel image, so we cannot compute it without a
self-hosted or paid runner. The throw-on-missing surfaces that fact
immediately when the build is requested there.

Also expose `frontendDeps` via `passthru` so the update script and
other tooling can build it directly without dragging in the full
handy compile.

`src` is temporarily pinned to the HEAD commit of cjpais/Handy#1256
so the orchestrator scripts are actually present in the source tree;
reverts to `tag = "v${finalAttrs.version}"` once that PR merges and
a Handy release containing the scripts is cut.
Copy link
Copy Markdown
Contributor

@y0usaf y0usaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked after 4236f7b — both prior issues fixed, heal pass for the peer-dep race looks correct, three-pass ordering makes sense. LGTM.

@xilec
Copy link
Copy Markdown
Contributor Author

xilec commented Apr 19, 2026

Hi @cjpais — quick status update:

  • Scripts verified: although these scripts aren't exercised by Handy's own CI, I validated them separately in my fork via custom GitHub Actions workflows on both Ubuntu and macOS runners — all green.
  • Review: approved by @y0usaf.

From my side the PR is ready to merge whenever you have a moment.

@cjpais
Copy link
Copy Markdown
Owner

cjpais commented Apr 19, 2026

Okay thanks for confirmation and update merging

@cjpais cjpais merged commit af6ec6c into cjpais:main Apr 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants