Support serial dependency ordering (3.14.3) (#552)

Cargo.toml

@@ -99,6 +99,7 @@ serde-saphyr = "0.0.6"
 minijinja = { version = "2.12.0", features = ["loader"] }
 cap-primitives = "3.4.4"
 cap-std = { version = "3.4.4", features = ["fs_utf8"] }
+fs4 = "1.1.0"
 camino = "1.2.0"
 semver = { version = "1", features = ["serde"] }
 anyhow = "1"

docs/adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md

@@ -0,0 +1,133 @@
+# Architecture decision record (ADR): Use Ninja dyndep for serial dependency ordering
+
+## Status
+
+Accepted.
+
+## Date
+
+2026-08-11
+
+## Context and problem statement
+
+Manifest authors need an explicit way to run the direct `deps` of an action or
+target in declaration order. The existing Ninja dependency classes preserve
+freshness and scheduling constraints, but ordinary implicit dependencies are
+all visible to the scheduler and may run concurrently.
+
+The implementation must retain a single Ninja invocation so a shared dependency
+runs once, propagate a failing early dependency to stop later work through the
+annotated path, and leave unrelated branches available for normal concurrent
+scheduling. It must also leave the Intermediate Representation (IR)
+backend-agnostic and make generated Ninja output executable in every command
+path.
+
+## Decision
+
+**Y-statement:** In the context of declaration-ordered direct dependencies,
+and facing the forces of shared-work reuse, failure short-circuiting,
+unrelated-branch concurrency, backend-neutral IR, and executable generated
+output, we decided to use staged Ninja dyndep sidecars for
+`dependency_order: serial`, accepting the Ninja 1.10 floor, reserved generated
+state beneath `.netsuke`, and a path-scoped ordering guarantee.
+
+In the context of a `dependency_order: serial` manifest `deps` list, Netsuke
+will use staged Ninja dyndep sidecars to reveal one direct dependency at a time
+and will materialize those sidecars atomically beneath `.netsuke/dyndep`. The
+runner materializes every sidecar file before Ninja starts; no Ninja edge
+produces sidecar content.
+
+`dependency_order` is a closed `parallel`/`serial` enum on the shared action
+and target AST shape. It is copied to `BuildEdge`, where it remains a logical
+graph annotation. Only the Ninja generator lowers a serial list containing two
+or more direct dependencies into synthetic phony gates beneath
+`.netsuke/serial` and content-addressed dyndep sidecars beneath
+`.netsuke/dyndep`.
+
+The gate edge associated with the next sidecar depends on the preceding gate,
+so each later direct dependency remains unavailable until earlier work
+succeeds.
+
+The main generated build file declares `ninja_required_version = 1.10` only
+when staged serial lowering is present. The generator exposes a complete bundle
+containing main-file text and every required sidecar. String-only generation
+rejects a graph requiring sidecars instead of returning an incomplete file.
+
+The serial guarantee is deliberately path-scoped: each direct dependency in the
+annotated list becomes schedulable only after its predecessor succeeds. A later
+dependency independently reachable through another requested path remains free
+to run through that other path.
+
+## Rationale
+
+- **One scheduler preserves shared work.** The generated gates stay inside one
+  Ninja invocation, so Ninja continues to deduplicate a repeated or diamond
+  dependency.
+- **Dyndep controls visibility.** A later real dependency is absent from the
+  relevant graph path until its sidecar is revealed, unlike an order-only edge
+  whose transitive inputs are already visible to Ninja.
+- **The IR remains portable.** Gates, sidecar paths, and Ninja version syntax
+  are backend mechanics rather than manifest graph concepts.
+- **Bundle ownership prevents incomplete output.** Treating sidecars as part of
+  the generated artefact makes every runner path materialize them before Ninja
+  loads the main file.
+- **Content addressing makes state reusable.** Existing matching sidecars are
+  safely reused; mismatching content is corruption and is reported rather than
+  overwritten.
+
+## Consequences
+
+- Serial lists with zero or one dependency use ordinary Ninja lowering; no
+  relative order needs enforcing and no dyndep version floor is emitted.
+- User graph paths in outputs, inputs, implicit dependencies, and order-only
+  dependencies cannot use `.netsuke/serial` or `.netsuke/dyndep`, because those
+  names are reserved generated state.
+- `build`, `clean`, and `generate` each materialize sidecars relative to the
+  effective Ninja working directory. `clean` may leave the immutable,
+  content-addressed sidecars in place.
+- `src/ninja_gen/dyndep.rs` owns staging and naming. The command-boundary
+  module `src/runner/dyndep_publication.rs` opens the effective capability and
+  orchestrates publication and retention; `src/runner/process/dyndep_files.rs`
+  owns atomic sidecar writes and verification, while
+  `src/runner/process/dyndep_retention.rs` owns the lease and cleanup. Neither
+  side of the boundary may broaden the path-scoped guarantee with a global
+  scheduler.
+- Tests must continue to use real Ninja for ordered starts, failure
+  short-circuiting, shared-work reuse, and unrelated-branch concurrency.
+
+## Alternatives considered
+
+### Order-only phony gate chain
+
+Rejected. Ninja eagerly schedules already-visible transitive inputs, so an
+order-only chain can order gate completion without preventing later real
+dependencies from starting early.
+
+### Ninja pool with depth one
+
+Rejected. A pool provides mutual exclusion, not declaration order, and would
+serialize unrelated work outside the annotated dependency list.
+
+### Recursive Ninja or Netsuke invocation per dependency
+
+Rejected. Separate child schedulers lose the enclosing build's memoization and
+can execute a shared dependency more than once.
+
+### A Netsuke-owned global scheduler
+
+Rejected for this feature. It would change the execution architecture and
+global reachability semantics rather than implement the requested scoped
+manifest policy. It requires a separately approved design.
+
+## Implementation references
+
+- Manifest and IR contract: [`src/ast/mod.rs`](../src/ast/mod.rs),
+  [`src/ir/graph.rs`](../src/ir/graph.rs), and
+  [`src/ir/from_manifest.rs`](../src/ir/from_manifest.rs)
+- Ninja bundle generation:
+  [`src/ninja_gen/dyndep.rs`](../src/ninja_gen/dyndep.rs)
+- Atomic sidecar materialization:
+  [`src/runner/process/dyndep_files.rs`](../src/runner/process/dyndep_files.rs)
+- User contract: [user's guide](users-guide.md#run-direct-dependencies-serially)
+- Implementation history:
+  [issue #552 ExecPlan](execplans/issue-552-support-serial-dependency-ordering-for-actions-and-targets.md)

docs/adr-012-bound-dyndep-sidecar-retention.md

@@ -0,0 +1,107 @@
+# Architecture decision record (ADR): Bound dyndep sidecar retention
+
+## Status
+
+Accepted.
+
+## Date
+
+2026-08-15
+
+## Context and problem statement
+
+Serial dependency ordering uses immutable, content-addressed Ninja dyndep
+sidecars beneath `.netsuke/dyndep`. A changed manifest therefore produces new
+`.dd` files without changing or overwriting files that an existing generated
+Ninja manifest references. Without a cleanup policy, obsolete sidecars would
+accumulate indefinitely. Cleanup must also avoid removing a sidecar that a
+concurrent Netsuke command is still consuming.
+
+The policy must preserve every sidecar in the current generated bundle,
+remove stale temporary files left by interrupted atomic writes, and bound
+obsolete storage deterministically. It must define the failure boundary for
+`clean` and make the compatibility consequence of retaining an old
+`generate --output` manifest explicit.
+
+## Decision
+
+Netsuke will retain immutable, content-addressed dyndep sidecars. Each
+sidecar-capable `build`, `generate`, or `clean` command materializes every
+sidecar in its current bundle before writing or invoking the generated Ninja
+file. Publication and cleanup use a capability-scoped, exclusive lease for
+the `.netsuke/dyndep` directory. The lease remains held through Ninja
+consumption for `build` and `clean`, or through generated-output consumption
+for `generate`.
+
+While that lease is held, Netsuke removes stale `.tmp` files and applies the
+following deterministic policy to obsolete `.dd` files:
+
+- every sidecar in the current bundle is retained;
+- at most 32 obsolete `.dd` files are retained; and
+- at most 1 MiB of obsolete `.dd` bytes is retained.
+
+Obsolete files are considered in deterministic path order. A sidecar's
+content is never changed in place. `build` and `generate` prune after
+materialization. `clean` prunes only after `ninja -t clean` succeeds; a failed
+clean does not prune sidecars.
+
+## Rationale
+
+- **Content addressing preserves active bundles.** A matching sidecar can be
+  reused and a mismatching file is corruption, not permission to overwrite
+  it.
+- **The lease protects consumption.** Publication, temporary-file cleanup,
+  and pruning share one directory lease, so cleanup cannot remove files while
+  another serial command is using its bundle.
+- **Fixed budgets are predictable.** File-count and byte limits provide a
+  bounded cache without relying on filesystem timestamps or an age-based
+  policy.
+- **`clean` keeps failure evidence.** Deferring cleanup until successful
+  `ninja -t clean` avoids deleting historical state when the requested clean
+  did not complete.
+
+## Consequences
+
+An old arbitrary manifest written by `generate --output` may lose referenced
+sidecars after a later Netsuke command applies retention. Such a manifest
+must be regenerated before use when its sidecars have been pruned. Generated
+manifests should therefore be treated as command outputs paired with the
+current sidecar cache, not as permanently self-contained artefacts.
+
+The policy does not use sidecar age, and it does not make sidecars mutable.
+There is no guarantee that an obsolete sidecar remains available merely
+because its manifest was generated successfully in an earlier command.
+
+## Alternatives considered
+
+### Retain sidecars by age
+
+Rejected. Wall-clock age is not deterministic and does not bound storage.
+
+### Keep every content-addressed sidecar
+
+Rejected. Immutable files would accumulate without bound as manifests change.
+
+### Mutate or overwrite existing sidecars
+
+Rejected. A content-addressed path must continue to identify one byte
+sequence, and overwriting it could change the graph seen by an existing
+manifest.
+
+### Prune without a directory lease
+
+Rejected. Publication and cleanup could race with a command that is consuming
+the current bundle, removing a required sidecar between materialization and
+Ninja or output consumption.
+
+## Implementation references
+
+- Runner publication boundary:
+  [`src/runner/dyndep_publication.rs`](../src/runner/dyndep_publication.rs)
+- Atomic sidecar materialization:
+  [`src/runner/process/dyndep_files.rs`](../src/runner/process/dyndep_files.rs)
+- Retention and lease implementation:
+  [retention implementation](../src/runner/process/dyndep_retention.rs)
+- User contract: [user's guide](users-guide.md#run-direct-dependencies-serially)
+- Serial dyndep architecture:
+  [ADR-011](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md)

docs/contents.md

@@ -57,14 +57,20 @@ operator, user, and contributor references are easier to find.
 - [adr-010-scope-glob-capability-to-literal-prefix.md](adr-010-scope-glob-capability-to-literal-prefix.md):
   Glob capability-scoping decision record, opening the metadata capability at
   a pattern's literal directory prefix instead of an ambient root.
+- [adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md](adr-011-use-ninja-dyndep-for-serial-dependency-ordering.md):
+  Serial `deps` ordering decision record, covering staged Ninja dyndep bundles,
+  their scoped execution guarantee, and generated-state ownership.
+- [ADR-012](adr-012-bound-dyndep-sidecar-retention.md):
+  Deterministic retention, lease, and failure-boundary policy for generated
+  dyndep sidecars.
 
 ## User and operator guides
 
 - [quickstart.md](quickstart.md): First-run walkthrough for building with
   Netsuke.
 - [v0-1-0-migration-guide.md](v0-1-0-migration-guide.md): Migration notes for
-  the v0.1.0 child-environment API additions and glob behaviour, and the
-  stability caveat that covers them.
+  the v0.1.0 child-environment API, glob behaviour, and serial-dependency
+  additions, plus the stability caveat that covers them.
 - [users-guide.md](users-guide.md): End-user reference for authoring and
   running Netsuke manifests, including executable discovery and
   `command_available` branch selection.