Skip to content

perf(root): retain native Darwin batch roots - #1095

Open
agent03-miq wants to merge 1 commit into
openclaw:mainfrom
agent03-miq:perf/darwin-root-open-batch
Open

agent03-miq wants to merge 1 commit into
openclaw:mainfrom
agent03-miq:perf/darwin-root-open-batch

Conversation

@agent03-miq

Copy link
Copy Markdown

Darwin file batches currently use one-shot admission for every file. Retain the existing native root descriptor for canonical, strict inputs while continuing shared strict path resolution before each native open. POSIX openBeneath permits contained ancestor symlinks, so Darwin must retain that path-policy check.

The Windows route, native-capability fallback, final identity/content checks, and originating native closer from #1071 remain intact. The changelog records the public behavior.

Validation against main 2147295fc553399964c151668a05c9edbce5aa7b:

  • The final regression tests against unchanged source produced 6 failures, 13 passes, with one Windows-only assertion unexecuted on Darwin.
  • The candidate passed 72 tests across six focused suites, with that Windows-only assertion still unexecuted. The suites cover batch opens, native close ownership, pinned opens/close, root files, and final admission.
  • Actual Darwin tests cover replaced roots and ancestors, leaf replacement after native open, descriptor cleanup, hardlinks, oversize inputs, NUL validation, opaque native I/O failure without pathname retry, capability-absent auto fallback and require-mode refusal, and cleanup of every retained root when one close throws.
  • Both baseline and candidate source/declaration compilation, touched-test typechecking, file-size and filesystem-boundary lint, and git diff --check pass. Native tests used the published Darwin ARM64 0.28.1 helper; native sources are unchanged since that release.
  • A separate network-denied full-content capture qualification preserved the complete 5,934-file / 26,072,397-byte path/hash/mode manifest and ordered receipt. A generated 101-file dependency fixture also preserved its computed asset and receipt.
  • pnpm check is incomplete: source compilation passes, then the archive WASM build stops because a wasm32-capable clang is unavailable. Allocator, docs, full-suite and package checks were not reached. The Windows helper toolchain is also unavailable locally; ordinary upstream CI can supply those existing checks.

Measured JavaScript filesystem calls during the complete capture:

Operation Baseline Candidate
openSync 17,866 6,121
realpathSync.native 61,005 37,392
lstatSync 175,852 187,843
readSync 41,528 41,528

Strict path admission increases lstatSync work. These are JavaScript call counts, not kernel syscall counts or a latency guarantee. No checks, timeouts, dependency versions, package exports or release settings are changed.

@agent03-miq
agent03-miq requested a review from a team as a code owner November 18, 2026 06:34
@clawsweeper

clawsweeper Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Nov 18, 2026
@clawsweeper

clawsweeper Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

What this changes

This PR reuses native root descriptors for strict, canonical Darwin file batches.

Example: A Darwin caller opens /Users/alice/project/a.ts and /Users/alice/project/b.ts in one batch.

  • Before: Each file uses ordinary one-shot root admission.
  • After: Both opens share a retained root descriptor while checking each path and file separately.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) Focused reuse and relevant tests are promising, but compatibility defects, unverified runtime evidence and an outstanding owner decision prevent acceptance.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is necessary before merge. See Before merge.
Patch quality 🦪 silver shellfish (2/6) 3 actionable review findings remain.

Product

Kind: Performance · Worth it: Needs a maintainer decision · Fix scope: Complete
User problem: Repeated synchronous Darwin file reads repeat root admission during large captures.
Reason: Reported reductions in JavaScript open and realpath calls support potential value, but expanding the owner's documented platform behavior needs agreement.

Merge readiness

⛔ Blocked before merge - 3 items remain

Keep open: current main lacks this optimization, and the PR contains useful work, but two compatibility defects need repair and the expanded platform contract needs owner agreement.

Priority: P2
Reviewed head: 79a0ef029bb78b1089aad8f98e73a467189f428f
Owner decision: Required. See Decision needed.

Decision needed

  • Question: Should strict canonical Darwin batches adopt retained native roots in place of the documented non-Windows one-shot path?
  • Recommendation: Approve the Darwin expansion after repairs: Retain the optimization while preserving existing permission and failure behavior.
  • Why: The feature owner's original implementation and documentation deliberately scope retention to Windows, and no owner approval of this expansion appears in the discussion.

Before merge

  • Add real behavior proof - The body reports native Darwin tests and a 5,934-file capture with matching manifests, but provides no inspectable public-entry-point transcript or artifact connecting that capture to the changed batch owner; provide that evidence with sensitive paths, credentials and endpoints redacted. After adding proof, update the PR body, then request a new review with @clawsweeper re-review. The PR author or someone with repository write access can request it.
  • Preserve reads through search-only Darwin ancestors (P2) - On macOS before 15.4, use a canonical root containing a directory with mode 0111 and a readable file beneath it, running as a non-root user. Ordinary openRootFileSync can lstat and traverse that directory without listing it. This new branch instead reaches the native fallback in native/src/unix.rs:1634, which opens every intermediate directory with O_RDONLY and fails with EACCES. There is no ordinary-path retry, so a previously successful batch read now fails. Preserve search-only traversal in the shared native owner, or retain ordinary admission where that contract cannot be met; add a focused regression for this input.
  • Keep strict resolver failures classified as validation (P2) - For a canonical Darwin root /Users/alice/project containing a regular file named plain, opening /Users/alice/project/plain/child makes the shared resolver throw ENOTDIR. Previously, openRootFileSync caught that exception and returned reason validation. The added resolver call now escapes to failure(), which returns reason path; an EACCES during resolution similarly changes from validation to io. This changes dispatch through matchRootFileOpenFailure despite the documented shared result contract. Preserve the ordinary resolver exception mapping around this call and test parity for these failures.

Findings

  • [P2] Preserve reads through search-only Darwin ancestors — src/root-file-open-batch-sync.ts:30-37
  • [P2] Keep strict resolver failures classified as validation — src/root-file-open-batch-sync.ts:100-104
  • [P3] Update the documented batch platform behavior — src/root-file-open-batch-sync.ts:56

Tests

  • Missing end-to-end proof: Supply redacted terminal output or a linked artifact showing the public batch API performing the claimed Darwin capture with native retention and matching content results.
Agent review details

How this fits together

The advanced synchronous batch API accepts root-confined file paths, performs path and native admission, and returns independently owned file descriptors through the existing pinned-file checks.

flowchart TD
A[Batch caller] --> B[Path eligibility]
B --> C[Ordinary one-shot admission]
B --> D[Retained root and strict path checks]
D --> E[Native beneath open]
E --> F[Pinned file and final identity checks]
C --> G[Caller owned descriptor]
F --> G
Loading

Technical review

Best possible solution:

Reuse the existing owners while preserving search-only directory access and resolver failure categories, then align the Darwin documentation with the approved contract.

Do we have a high-confidence way to reproduce the issue?

Current-main source conclusively routes Darwin batches through one-shot admission; the reported performance measurements were not independently executed.

Is this the best way to solve the issue?

Extending the existing retained-root owner is appropriately small, but its native permission requirements and resolver error handling must preserve ordinary Darwin read semantics.

Full review comments:

  • [P2] Preserve reads through search-only Darwin ancestors — src/root-file-open-batch-sync.ts:30-37
    On macOS before 15.4, use a canonical root containing a directory with mode 0111 and a readable file beneath it, running as a non-root user. Ordinary openRootFileSync can lstat and traverse that directory without listing it. This new branch instead reaches the native fallback in native/src/unix.rs:1634, which opens every intermediate directory with O_RDONLY and fails with EACCES. There is no ordinary-path retry, so a previously successful batch read now fails. Preserve search-only traversal in the shared native owner, or retain ordinary admission where that contract cannot be met; add a focused regression for this input.
    Confidence: 0.96
  • [P2] Keep strict resolver failures classified as validation — src/root-file-open-batch-sync.ts:100-104
    For a canonical Darwin root /Users/alice/project containing a regular file named plain, opening /Users/alice/project/plain/child makes the shared resolver throw ENOTDIR. Previously, openRootFileSync caught that exception and returned reason validation. The added resolver call now escapes to failure(), which returns reason path; an EACCES during resolution similarly changes from validation to io. This changes dispatch through matchRootFileOpenFailure despite the documented shared result contract. Preserve the ordinary resolver exception mapping around this call and test parity for these failures.
    Confidence: 0.99
  • [P3] Update the documented batch platform behavior — src/root-file-open-batch-sync.ts:56
    The new Darwin branch contradicts docs/reading.md:214-222, which says non-Windows platforms use ordinary per-open admission, and docs/advanced.md:82 still describes Windows-only retention. Update both descriptions to cover Darwin eligibility and its strict-path check, including the auto-mode fallback and require-mode refusal distinction.
    Confidence: 1

Overall correctness: patch is incorrect
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 2147295fc553.

Provenance checked

  • Platform eligibility and strict component admission changes intended behavior with a stated reason (feat(root): batch native Windows file opens #984: Windows retained-root opens reject reparse traversal; other platforms retain ordinary admission. The new body explains why Darwin requires a separate shared strict-path check.)
  • Descriptor close ownership keeps the original intent (fix(root): retain native batch-read close ownership #1071: Native-opened descriptors must retain their originating closer rather than close through Node's descriptor tracker.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

Likely related people:

  • Peter Steinberger: Raw commit 85972db adds src/root-file-open-batch-sync.ts:25 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 85972db8cd4d; files: src/root-file-open-batch-sync.ts)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Review metrics

Metric Value Why it matters
Patch scope Production +21/-1; tests +90/-7; changelog +4 The production change reuses existing owners; tests cover the newly enabled platform and native lifecycle behavior.

Labels

Label changes:

  • add P2: This is a bounded performance improvement with compatibility defects, without evidence of an urgent current-user outage.
  • add merge-risk: 🚨 compatibility: Enabling the native route can reject readable files beneath search-only directories on older macOS and changes public failure categories.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Label justifications:

  • P2: This is a bounded performance improvement with compatibility defects, without evidence of an urgent current-user outage.
  • merge-risk: 🚨 compatibility: Enabling the native route can reject readable files beneath search-only directories on older macOS and changes public failure categories.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 18, 2026, 1:48 AM ET / 06:48 UTC.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants