diff --git a/docs/adr-001-lock-file-naming.md b/docs/adr-001-lock-file-naming.md new file mode 100644 index 0000000..dd21254 --- /dev/null +++ b/docs/adr-001-lock-file-naming.md @@ -0,0 +1,204 @@ +# Architectural decision record (ADR) 001: Lock file naming + +## Status + +Accepted on 2026-06-29. `mpsc-log` reserves coordination filenames and derives +one journal lock by appending `.lock` to the complete journal filename in the +same directory. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` serializes repair, the authoritative sidecar read, rotation, +compression, and append through one advisory lock per journal. Every safety +claim in the design depends on all cooperating invocations choosing the same +lock for the same journal and choosing different locks for different journals. + +The obvious rule, `.lock` in the same directory, is simple +but needs collision boundaries. Names such as `run`, `run.jsonl`, +`run.jsonl.lock`, and sidecar paths derived by extension replacement can +otherwise create hidden ambiguity: + +- `run` should not silently collide with `run.jsonl`; +- `run.jsonl.lock` is naturally the lock path for `run.jsonl`, but it could + also be supplied as a journal path unless reserved; +- `run.toml` can be both a journal path and the extension-replaced sidecar path + for `run` unless `.toml` journal paths are rejected; +- journals that share a stem, such as `run` and `run.jsonl`, share the sidecar + `run.toml` under the design's extension-replacement rule. + +The project needs this decision before implementing filesystem safety, because +lock naming, sidecar derivation, and reserved suffixes define what the writer +can safely create under contention. + +## Decision drivers + +- Lock naming must be deterministic from the caller-provided journal path. +- The lock path must not use extension replacement; otherwise `run` and + `run.jsonl` can become ambiguous. +- Coordination artefacts must be reserved so callers cannot accidentally append + JSONL records into another journal's lock file. +- The sidecar derivation rule is already part of the design and should remain + easy to explain. +- The rule must work before the journal exists and while several first writers + create parent directories concurrently. + +## Requirements + +### Functional requirements + +- Derive one lock path for each accepted journal path. +- Ensure accepted journal paths cannot name another journal's lock file. +- Ensure accepted journal paths cannot equal their own sidecar path. +- Keep `run` and `run.jsonl` lock paths distinct. +- Keep the sidecar extension-replacement rule explicit when multiple journal + names share one stem. + +### Technical requirements + +- Create parent directories before opening the lock file. +- Open the lock file with create semantics and acquire the exclusive lock + before the authoritative sidecar configuration read, repairing tails, + rotating, compressing, or appending. +- Permit an unlocked, advisory pre-read of the sidecar `timeout_ms` value + solely to choose the lock-acquisition timeout; it never substitutes for the + authoritative read. +- Treat `.lock` journal paths as invalid because `.lock` is reserved for + coordination artefacts. +- Treat `.toml` journal paths as invalid when the derived sidecar path would be + the journal path itself. +- Keep lock naming local to the journal directory; do not introduce a global + lock directory or daemon. + +## Options considered + +### Option A: Append `.lock` to the full journal filename and reserve suffixes + +For an accepted journal path, derive the lock path by appending `.lock` to the +complete filename in the same directory. Reject journal filenames ending in +`.lock`, and reject journal filenames whose derived sidecar path equals the +journal path. + +Examples: + +| Journal path | Lock path | Sidecar path | Accepted | +| ---------------- | ----------------- | ------------ | ------------------------------- | +| `run` | `run.lock` | `run.toml` | Yes | +| `run.jsonl` | `run.jsonl.lock` | `run.toml` | Yes | +| `run.ndjson` | `run.ndjson.lock` | `run.toml` | Yes | +| `run.jsonl.lock` | None | None | No, `.lock` is reserved | +| `run.toml` | None | None | No, sidecar would equal journal | + +_Table 1: Filename outcomes for the accepted naming policy._ + +This option keeps lock derivation simple, avoids lock collisions between +extension variants, and makes reserved coordination/configuration names +explicit. + +### Option B: Replace the journal extension with `.lock` + +This option would make `run.jsonl` use `run.lock`. It is short, but it collides +with the lock path for `run` and repeats the ambiguity already present in +extension-replaced sidecar names. + +### Option C: Use a hidden lock filename + +This option would use a name such as `.run.jsonl.lock` in the same directory. +It avoids some visible filename clutter, but it does not remove the need for +reserved names, makes operator inspection harder, and adds platform-specific +hidden-file expectations without improving the safety claim. + +### Option D: Use a central lock directory + +This option would hash canonical journal paths into a shared lock directory. It +can avoid adjacent artefacts, but it introduces global state, path canonicalize +questions, lifecycle cleanup, and permission behaviour that are +disproportionate for a local CLI. + +| Topic | Option A | Option B | Option C | Option D | +| ---------------------------------- | -------- | -------- | -------- | -------- | +| Same-stem collision risk | Low | High | Low | Low | +| Operator inspectability | High | High | Lower | Lower | +| Requires global state | No | No | No | Yes | +| Handles `run.jsonl.lock` ambiguity | Yes | No | Partly | Partly | +| Complexity | Low | Low | Medium | High | + +_Table 2: Comparison of lock naming options._ + +## Decision outcome / proposed direction + +Adopt Option A. + +The writer derives lock paths by appending `.lock` to the complete accepted +journal filename in the same directory: + +```plaintext +/.lock +``` + +The operation appends to the complete filename; it does not replace an +extension. The lock for `run` is `run.lock`. The lock for `run.jsonl` is +`run.jsonl.lock`. + +The suffix `.lock` is reserved for lock files. A caller-supplied journal path +whose final filename ends in `.lock` fails before append because it can be +mistaken for another journal's coordination artefact. A caller-supplied journal +path whose derived sidecar path equals the journal path also fails; +practically, that reserves `.toml` journal filenames under the current sidecar +rule. + +Sidecar configuration still uses extension replacement: `run.jsonl` and +`run.ndjson` both use `run.toml`, while `run` also uses `run.toml`. That shared +sidecar is intentional under the current design. Callers needing independent +configuration must choose distinct stems or directories, such as +`run-jsonl.jsonl` and `run-raw.ndjson`. + +## Goals and non-goals + +- Goals: + - Make lock naming deterministic and collision-resistant for accepted journal + paths. + - Reserve coordination and configuration artefact names before implementation. + - Preserve the simple adjacent lock-file model. + - Keep sidecar sharing by stem explicit rather than accidental. +- Non-goals: + - Provide distributed locking across hosts. + - Canonicalize paths across symlinks or mount aliases beyond normal + filesystem API behaviour. + - Replace sidecar extension derivation with a different configuration scheme. + - Permit arbitrary journal filenames when they collide with coordination + artefacts. + +## Migration plan + +1. Add journal-path validation before opening the lock file. + - Reject final filenames ending in `.lock`. + - Reject paths whose derived sidecar path equals the journal path. +2. Implement lock-path derivation as a pure function and test the examples in + Table 1. +3. Use the derived lock path for the complete critical section: the + authoritative sidecar read, tail repair, rotation, compression, and append. +4. Document reserved suffixes and sidecar sharing in the users' guide when the + CLI implementation lands. + +## Known risks and limitations + +- The rule does not prevent two accepted journal paths with the same stem from + sharing a sidecar. That is documented as intentional configuration sharing. +- The rule does not solve symlink aliasing or path canonicalization across + mount points. Those remain outside the local-filesystem correctness claim. +- Operators may still choose confusing stems, such as `run.locked.jsonl`; those + are accepted because they do not use the reserved final `.lock` suffix. + +## Architectural rationale + +The accepted rule keeps the lock path adjacent to the data it protects and +removes hidden ambiguity around lock artefacts. Appending `.lock` to the full +filename keeps `run` and `run.jsonl` separate, while reserving `.lock` journal +paths prevents a caller from treating another journal's lock as data. The +decision is deliberately conservative because every later concurrency, repair, +rotation, and timeout guarantee assumes all cooperating invocations agree on +the same coordination boundary. diff --git a/docs/adr-002-testing-strategy.md b/docs/adr-002-testing-strategy.md new file mode 100644 index 0000000..6d79f10 --- /dev/null +++ b/docs/adr-002-testing-strategy.md @@ -0,0 +1,217 @@ +# Architectural decision record (ADR) 002: Testing strategy + +## Status + +Accepted on 2026-06-29. The project will apply the testing prongs mandated in +`AGENTS.md` as a layered strategy tied to the `mpsc-log` design's contracts, +not as a separate testing phase. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` is a short-lived Rust CLI that appends one JSON object to a shared +JSON Lines journal while handling sidecar configuration, type coercion, +locking, repair, rotation, compression, and bounded lock waiting. The design's +highest-risk behaviours are externally observable: concurrent writers must not +lose records, rotation must preserve every successful entry, partial tails must +be repaired, and diagnostics must let non-interactive agents distinguish usage, +data, configuration, I/O, and timeout failures. + +`AGENTS.md` mandates several testing prongs for this repository: unit tests, +behavioural tests, snapshots, end-to-end tests, property tests, bounded model +checking, and Verus proofs where contractual business logic warrants exhaustive +proof. The project needs one decision record explaining how those prongs apply +to the proposed design so implementers do not either under-test concurrency and +persistence or create standalone testing theatre that does not improve +confidence. + +## Decision drivers + +- The CLI's main promise is durable append under contention, so process-level + and filesystem-level evidence matter more than line coverage alone. +- The design already separates pure domain logic from clock and filesystem + adapters, which should make deterministic and fault-injection tests practical. +- `AGENTS.md` requires `rstest`, `rstest-bdd` where applicable, `insta` where + output variants matter, end-to-end tests for observable workflows, and + property-based or formal methods for invariants over ranges of states. +- Tests must remain review-sized with development tasks. Unit and behavioural + tests belong with their implementation work; only cross-feature end-to-end, + combinatorial, or formal hardening suites should stand alone. +- Direct environment mutation in tests is forbidden unless guarded through + shared helpers as a last resort. The current design should avoid relying on + mutable process environment state. + +## Requirements + +### Functional requirements + +- Verify that each successful invocation appends exactly one complete JSON + object line. +- Verify that concurrent successful invocations produce the same number of + complete records as successful process exits. +- Verify that size-only rotation, scheduled rotation, gzip compression, and + retention preserve successful records. +- Verify that malformed unterminated tails and injected write failures do not + leave partial final records after the next invocation. +- Verify that newline-terminated corrupt final records fail closed without + truncation. +- Verify the selected `jo` field syntax, object-root enforcement, sidecar + precedence, schema-guided coercion, explicit coercion flags, default + timestamp insertion, and stable exit-code mapping. +- Verify df12-build fixture records against the JSON Schema and sidecar + coercion contract. + +### Technical requirements + +- Use `rstest` fixtures and parameterized cases for unit and integration tests. +- Use `rstest-bdd` for externally observable CLI scenarios where the + Given/When/Then shape clarifies user-facing behaviour. +- Use `insta` snapshots only for stable, reviewer-useful output boundaries, + pairing snapshots with semantic assertions and normalizing nondeterministic + fields. +- Use process-level end-to-end tests for CLI workflows, concurrent appends, + rotation, and df12-build fixture invocations. +- Use `proptest` for generated field words, object-path merges, sidecar/CLI + precedence, timestamp override cases, tail-repair inputs, and rotation-plan + invariants. +- Use `kani` for bounded state-machine checks when a pure planner has a compact + state space, such as generation shifting, scheduled period grouping, or + append/repair transitions. +- Use Verus only for introduced lemmas or contractual business logic whose + safety property is important enough to justify proof maintenance. +- Keep filesystem, time, and failure injection behind explicit adapters rather + than mutating global state. + +## Options considered + +### Option A: Unit and CLI smoke tests only + +This option would test parsers, record construction, and a few happy-path CLI +commands. It is fast and easy to maintain, but it does not prove the +concurrency, repair, or rotation promises that justify the tool. + +### Option B: Apply every testing tool to every feature + +This option would require unit, behavioural, snapshot, end-to-end, +property-based, model-checking, and proof coverage for every change. It is +maximally strict, but it would slow review, produce weak tests for features +that do not benefit from a given prong, and make formal tools feel ceremonial. + +### Option C: Risk-based layered strategy + +This option maps each testing prong to the design surface it can validate best. +Pure logic gets unit and property tests. User-facing workflows get behavioural +and end-to-end tests. Stable text or file-shape contracts get narrow snapshots. +Concurrency, rotation, and repair get stress, fault-injection, and model-based +coverage. Formal proof is reserved for compact contractual invariants. + +| Topic | Option A | Option B | Option C | +| -------------------------- | ------------- | ----------------------- | ----------------------------- | +| Concurrency confidence | Weak | Strong but costly | Strong where required | +| Review size | Small | Often too large | Review-sized by risk | +| Formal methods | Absent | Over-applied | Applied to compact invariants | +| Alignment with `AGENTS.md` | Incomplete | Literal but inefficient | Complete and targeted | +| Maintenance cost | Low initially | High | Proportional to risk | + +_Table 1: Comparison of testing strategy options._ + +## Decision outcome / proposed direction + +Adopt Option C, the risk-based layered testing strategy. + +Every implementation task must include the unit, behavioural, property, or +snapshot coverage needed for the behaviour it introduces. Dedicated roadmap +tasks remain appropriate for cross-feature evidence that exceeds one pull +request, such as multi-process stress testing, pairwise CLI/configuration +coverage, concurrent rotation end-to-end tests, and broader formal hardening of +shared planners. + +The testing prongs apply as follows: + +- Unit tests with `rstest` cover pure domain logic in `args`, `fields`, + `config`, `record`, `errors`, `clock`, and rotation-planning helpers. +- Behavioural tests with `rstest-bdd` cover user-visible CLI behaviours: + successful append, unsupported syntax, sidecar precedence, diagnostics, + timeout, repair, corrupt-final-line, and rotation scenarios. +- Snapshot tests with `insta` cover stable outputs that reviewers benefit from + seeing as artefacts: help text, one-line diagnostics, canonical fixture + records after nondeterministic normalization, and rotation filename matrices. +- End-to-end tests cover real binary invocations against temporary directories, + including parent-directory creation, simultaneous first writes, concurrent + appends, forced size rotation, scheduled rotation, compression, and + df12-build fixture commands. +- Property tests with `proptest` cover input and state ranges for field + parsing, object-path insertion, type coercion, merge precedence, partial-tail + repair, final-line validation, and rotation-plan invariants. +- Bounded model checks with `kani` cover compact pure planners where exhaustive + small-state exploration is more useful than randomized testing, especially + generation shifting and scheduled-period retention. +- Verus proofs are reserved for explicit lemmas or contractual business logic + introduced during implementation. They are not required for ordinary glue + code, parser plumbing, or tests that can be more clearly expressed with + `rstest`, `proptest`, or `kani`. + +## Goals and non-goals + +- Goals: + - Tie each required testing prong to a concrete `mpsc-log` design surface. + - Keep tests deterministic through clock and filesystem injection. + - Make concurrency, rotation, repair, and failure-mode evidence release + blockers. + - Keep implementation tasks review-sized by embedding ordinary tests in the + task that introduces the behaviour. +- Non-goals: + - Mandate every testing tool for every implementation task. + - Treat coverage percentage as a substitute for contract evidence. + - Prove network-filesystem correctness before the design accepts that + platform promise. + - Add `max_age` retention tests while `max_age` remains out of scope. + +## Migration plan + +1. Add test support before the first feature slice lands. + - Create shared `rstest` fixtures for temporary directories, fixed clocks, + filesystem adapters, command invocation, and decoded JSONL records. + - Add helper modules under `tests/` or `src/` behind `#[cfg(test)]` where + they preserve ownership boundaries. +2. Add domain tests with each implementation module. + - Pair parser, sidecar, record, error, and rotation-planner work with + `rstest` cases and `proptest` strategies for their accepted input spaces. +3. Add behavioural and end-to-end suites once the CLI can append a record. + - Use `rstest-bdd` for user-facing scenarios and process-level tests for + real binary execution. +4. Add concurrency, repair, and rotation hardening as the filesystem adapter + lands. + - Exercise multi-process contention, lock timeout, partial-tail repair, + newline-terminated corrupt final records, injected I/O failures, size + rotation, scheduled rotation, gzip, and retention. +5. Add model checking or proofs only when the implementation introduces a pure + invariant-bearing planner or lemma. + - Prefer `kani` for bounded state machines and Verus for explicit lemmas + whose proof is clearer than a large generated test matrix. + +## Known risks and limitations + +- Multi-process stress tests can be timing-sensitive. They must assert durable + outcomes, not exact scheduling order. +- Snapshot tests can become brittle if they capture broad objects or raw + timestamps. They must stay narrow and normalize nondeterministic fields. +- Formal tools add maintenance cost. They should protect compact invariants, + not duplicate straightforward example tests. +- The initial correctness claim remains limited to local filesystems until a + separate platform verification matrix names and validates other filesystem + behaviours. + +## Architectural rationale + +The chosen strategy follows the design's architecture. Pure record-building and +rotation-planning code can be tested exhaustively and deterministically. +Filesystem effects pass through adapters, allowing fault injection without +global state mutation. User-facing CLI behaviour is verified at the process +boundary because agents experience the tool through exit codes, diagnostics, +and files on disk. The result keeps the repository's mandated testing prongs +connected to real product risk instead of distributing them mechanically across +the codebase. diff --git a/docs/adr-003-jo-field-syntax-and-duplicate-keys.md b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md new file mode 100644 index 0000000..a8a1434 --- /dev/null +++ b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md @@ -0,0 +1,181 @@ +# Architectural decision record (ADR) 003: `jo` field syntax and duplicate keys + +## Status + +Accepted on 2026-06-29. `mpsc-log` uses selected `jo`-inspired field syntax for +object records, but does not promise textual JSON compatibility with `jo`. +Duplicate writes to the same object path resolve with last-wins semantics. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` should feel familiar to workflow authors who already use `jo` to +build JSON from shell arguments. The product goal is not to be a drop-in `jo` +replacement. It is a reliable multi-process JSON Lines writer whose root record +is always an object, held in the domain as a `Record` object map of domain +`Value` values and serialized by the JSON adapter at the output boundary. + +That distinction matters for duplicate object keys. Textual JSON can contain +repeated object names, and `jo` can produce those names in its output. An +object map cannot preserve repeated textual keys. Once `mpsc-log` chooses a map +as its domain representation, duplicate writes must either be rejected or +collapsed into one value. + +The project needs an explicit compatibility statement before implementing the +argument parser, merge rules, sidecar schema interaction, and tests. + +## Decision drivers + +- The CLI should preserve the ergonomic field words that make `jo` useful for + shell callers. +- The root record must remain a JSON object so defaults, telemetry fields, and + object-path writes have predictable names. +- Consumers of JSON Lines logs usually parse records into maps, so repeated + textual keys would be fragile even if the writer could emit them. +- The implementation should use structured JSON serialization rather than + manually assembling JSON text. +- Duplicate-path handling must match sidecar merge precedence and CLI + left-to-right processing. + +## Requirements + +### Functional requirements + +- Accept selected `jo`-inspired field words for object records. +- Support explicit coercion flags, schema-guided coercion, object paths, and + file-value forms that preserve an object root. +- Reject `jo` features that would produce a non-object root or require textual + output compatibility. +- Define duplicate writes to the same object path deterministically. +- Document that `mpsc-log` is `jo`-inspired, not fully `jo` compatible. + +### Technical requirements + +- Store records in a domain `Record` object map whose values are domain + `Value` values, and keep serialization crates out of domain modules. +- Serialize that `Record` to compact JSON through `serde_json` in the JSON + adapter at the output boundary. +- Process CLI field words in argument order. +- Let later writes to the same object path replace earlier writes. +- Pair the duplicate-key behaviour with parameterized tests covering top-level + keys, nested object paths, sidecar defaults, and explicit coercion flags. + +## Options considered + +### Option A: Selected `jo`-inspired syntax with last-wins duplicate paths + +This option accepts the object-record syntax needed by `mpsc-log` and resolves +duplicate writes by replacing the earlier value at the same object path. + +Examples: + +| Arguments | Result | +| ---------------------------------------------- | ------------------- | +| `status=started status=done` | `{"status":"done"}` | +| `task.id=1 task.id=2` with `-d .` | `{"task":{"id":2}}` | +| sidecar `status = "queued"`, CLI `status=done` | `{"status":"done"}` | + +_Table 1: Last-wins examples for duplicate paths._ + +This option matches the existing merge model: sidecar defaults seed the record, +CLI fields are processed left to right, and each later CLI write wins at its +path. + +### Option B: Reject duplicate paths + +This option would fail when a CLI field writes a path already set by a sidecar +default or earlier CLI field. It is strict, but it makes deliberate overrides +awkward and complicates the existing precedence model. + +### Option C: Preserve textual duplicate JSON object names + +This option would attempt closer textual compatibility with `jo` by emitting +duplicate JSON object names. It conflicts with the domain `Record` object map, +makes schema coercion and object-path updates harder to reason about, and +produces records that many consumers collapse differently. + +### Option D: Full `jo` compatibility + +This option would expand the CLI toward all `jo` behaviours, including array +roots and formatting options. It conflicts with the object-root logging +contract and would pull the tool away from its append-safe journal purpose. + +| Topic | Option A | Option B | Option C | Option D | +| -------------------------- | -------- | -------- | -------- | -------- | +| Object-root fit | High | High | Medium | Low | +| Familiar shell syntax | High | Medium | High | High | +| Textual `jo` compatibility | Partial | Partial | Higher | High | +| Structured JSON safety | High | High | Low | Medium | +| Implementation complexity | Low | Medium | High | High | + +_Table 2: Comparison of field syntax and duplicate-key options._ + +## Decision outcome / proposed direction + +Adopt Option A. + +`mpsc-log` describes its CLI as `jo`-inspired or as selected `jo` field syntax. +It does not claim full `jo` compatibility. The `jo` manual remains prior art +for familiar field words and coercion expectations, but the `mpsc-log` contract +is the subset recorded in the design and this ADR. + +Duplicate writes to the same object path use last-wins semantics: + +1. Sidecar defaults seed the object. +2. CLI fields are processed in argument order. +3. Each CLI field replaces any existing value at its object path. +4. Explicit coercion flags affect the field they annotate before the value is + written. +5. The generated timestamp is inserted only when no `timestamp` field exists + after defaults and CLI fields. + +This means last-wins applies to duplicate top-level keys, duplicate nested +paths, and CLI overrides of sidecar defaults. It does not preserve repeated +textual JSON names. + +## Goals and non-goals + +- Goals: + - Make the compatibility boundary honest for users and implementers. + - Preserve the familiar shell-friendly field forms needed for logging. + - Keep the object-root `Record` map-based and its serialization + deterministic. + - Align duplicate-path behaviour with merge precedence. +- Non-goals: + - Preserve repeated textual JSON object names. + - Implement `jo` formatting, pretty-printing, array-root, or version options. + - Guarantee compatibility with every `jo` edge case. + - Define organization-wide schema governance for duplicate telemetry fields. + +## Migration plan + +1. Update product documentation to use `jo`-inspired or selected `jo` field + syntax rather than full compatibility language. +2. Implement parser support only for the accepted object-record forms listed in + the design. +3. Add tests for last-wins duplicate paths across sidecar defaults, CLI fields, + explicit coercion flags, and nested object paths. +4. Keep full `jo` compatibility in deferred scope unless a later ADR changes + the object-root product boundary. + +## Known risks and limitations + +- Users expecting byte-for-byte `jo` output can be surprised by duplicate keys + collapsing to one value. +- Last-wins can hide accidental repeated fields. The behaviour is predictable, + but the implementation should keep diagnostics clear for invalid paths and + unsupported forms. +- If future consumers require textual duplicate keys, the map-based record + builder would need a larger redesign. + +## Architectural rationale + +The accepted rule matches the product's actual abstraction. `mpsc-log` is a +structured journal writer, not a JSON text generator. A map-backed record gives +sidecar defaults, schema coercion, explicit flags, object paths, and timestamp +defaulting one deterministic merge model. Last-wins duplicate handling is the +least surprising behaviour inside that model, provided the documentation does +not overstate compatibility with `jo`. diff --git a/docs/contents.md b/docs/contents.md index 3a9800b..b2cab73 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -5,9 +5,18 @@ set. ## Project guides +- [Context](context.md) defines the project vocabulary for journals, + sidecars, rotation, and df12-build telemetry. +- [Design](mpsc-log-design.md) specifies the CLI, file formats, locking, + rotation, failure handling, and verification strategy for `mpsc-log`. +- [Terms of reference](terms-of-reference.md) defines the product problem, + users, scope boundaries, constraints, and open questions for `mpsc-log`. +- [Roadmap](roadmap.md) translates the design and terms of reference into + review-sized implementation tasks. - [User guide](users-guide.md) explains how to use the generated project and its public build and test commands. -- [Developer guide](developers-guide.md) explains the local workflow and +- [Developer guide](developers-guide.md) explains the planned implementation + architecture, module layering, adapter boundary, local workflow, and implementation tooling for contributors. - [Repository layout](repository-layout.md) explains the generated project's top-level files, directories, and ownership boundaries. @@ -33,3 +42,18 @@ set. - [Scripting standards](scripting-standards.md) explains the preferred Python scripting stack, command execution patterns, and test expectations for helper scripts. + +## Design artefacts + +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md) records how the + journal lock path is derived and which coordination/configuration suffixes + are reserved. +- [ADR 002: Testing strategy](adr-002-testing-strategy.md) records how the + repository's required testing prongs apply to the `mpsc-log` design. +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md) + records the selected `jo`-inspired field syntax and last-wins duplicate-path + semantics. +- [mpsc-log event schema](mpsc-log-event-schema.json) defines the initial JSON + Schema for journal records. +- [mpsc-log sidecar example](mpsc-log-sidecar.example.toml) shows the TOML + configuration shape used by the design. diff --git a/docs/context.md b/docs/context.md new file mode 100644 index 0000000..fda0085 --- /dev/null +++ b/docs/context.md @@ -0,0 +1,33 @@ +# Context + +[Context](context.md) defines the working vocabulary for `mpsc-log`. + +## Terms + +| Term | Definition | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Active log | The JSON Lines file named on the command line. Successful invocations append new records to this file unless rotation runs first. | +| Agent workflow | A workflow that launches one or more autonomous agents or helper processes that can invoke command-line tools. | +| Audit yield | The distribution of audit findings by severity, source, and remediation lane. | +| CodeRabbit attempt | One attempt to run CodeRabbit review, including its start time, result, wait time, retry status, and whether the review was deferred. | +| Corrupt final line | A newline-terminated final JSON Lines record that is not a valid JSON object. The default repair path reports this instead of truncating it. | +| Defect escape | A defect discovered after merge by later work, dogfooding, Continuous Integration (CI), or users. | +| Journal | A shared JSON Lines file used as an append-only record of workflow events. | +| Lock timeout | The maximum time an invocation waits for the journal lock before failing with a timeout diagnostic. | +| Object path | A `jo`-inspired field path that writes a value into a nested JSON object rather than into a literal top-level key. | +| Open Dynamic Workflows (ODW) | The workflow runtime used by `df12-build` to coordinate multi-agent roadmap execution. | +| Partial tail | Unterminated bytes after the final newline in the active log. Repair may truncate these bytes before appending the next record. | +| Record | One JSON object serialized as one JSON Lines value and terminated with `\n`. | +| Remediation lane | The route assigned to a review or audit follow-up, such as addendum, step task, later roadmap step, or dropped. | +| Review round | One pass through a review loop, including any blocking findings and subsequent fix attempt. | +| Rotated log | A previous active log generation retained under a numbered or scheduled filename, optionally compressed with gzip. | +| Scheduled break | A configured UTC time boundary, either hourly, daily, or weekly, that forces the current active log segment to rotate on the next invocation. | +| Sidecar configuration | The TOML configuration file derived from the active log path by replacing the filename extension with `.toml`. | +| Type coercion | Conversion of a CLI value to a JSON string, number, boolean, null, object, or array according to explicit flags, sidecar schema, or default `jo`-inspired inference. | +| Workflow sidecar directory | The caller-owned directory where a workflow stores run artefacts, including the `mpsc-log` journal chosen by that workflow. | + +## Naming + +- Use "journal" for the shared JSON Lines log written by `mpsc-log`. +- Use "sidecar configuration" for the `.toml` file next to the journal. +- Use "workflow sidecar directory" for the df12-build ODW artefact directory. diff --git a/docs/developers-guide.md b/docs/developers-guide.md index bbbe3f5..e2d2cd5 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -2,6 +2,72 @@ This guide explains the contributor workflow for the generated mpsc-log project. +## mpsc-log implementation architecture + +`mpsc-log` is planned as a small domain core surrounded by adapters. The domain +owns the record model and the write, rotation, and retention policy. Adapters +own every external format and effect. The authoritative specification is the +[design document](mpsc-log-design.md); this section summarizes the boundary it +defines so contributors can place new code correctly. + +`src/main.rs` owns process startup and `sysexits` mapping only. It constructs +the adapters, runs the domain, and translates the semantic error type into an +exit code. It carries no parsing, merging, rotation, or filesystem logic. + +### Planned modules + +| Module | Layer | Responsibility | +| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `args` | Adapter | Read process arguments and hand on the journal path and the raw selected `jo` field tail in order. | +| `fields` | Domain | Interpret field words, object paths, coercion flags, and file-value forms into domain `Value` results. | +| `config` | Adapter | Parse and validate the sidecar TOML at the input boundary, converting `[defaults]` and `[schema]` data into domain types. | +| `record` | Domain | Build the `Record`: merge defaults and CLI fields, apply coercion policy and object-path updates, and insert the generated timestamp. | +| `journal` | Domain | Plan tail repair, rotation, retention, and append, and drive them through the `JournalStore` port. | +| `errors` | Domain | Define the semantic error type. Exit-code mapping belongs to `src/main.rs`, not here. | +| `clock` | Domain port | Declare the `Clock` port; an infrastructure implementation supplies the real instant. | +| `fs` | Adapter | Implement `JournalStore` over the real filesystem, alongside the fault-injection test double. | + +### The adapter boundary + +The domain never names an external API. It works only with the domain `Record` +and `Value` types and with ports. Everything that reaches outside the process, +or that encodes an external format, is an adapter concern: + +- CLI argument reads. +- TOML parsing of the sidecar. +- JSON serialization of the record. +- Filesystem operations. +- Locking. +- Compression. +- Process exit mapping. + +The domain reaches time and persistence-related effects through two ports. +`Clock` supplies the invocation instant used for the generated `timestamp` +field. `JournalStore` performs every journal-directory effect: creating parent +directories, reading the sidecar, acquiring and releasing the journal lock, +inspecting and repairing the active tail, appending, truncating, renaming, +writing compression output, and syncing metadata. + +Because both are traits, tests substitute a fixed clock and a fault-injecting +store without mutating global process state. See +[reliable testing in Rust via dependency injection](reliable-testing-in-rust-via-dependency-injection.md) +for the injection patterns this repository expects. + +When adding code, put format and effect handling in an adapter and keep policy +in the domain. If a domain module needs a new external effect, add a port +rather than importing the concrete API. + +### Further reading + +- [Design](mpsc-log-design.md) specifies the CLI contract, record model, + ports, adapters, write protocol, and rotation protocol. +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md) records how the + journal lock path is derived and which suffixes are reserved. +- [ADR 002: Testing strategy](adr-002-testing-strategy.md) records how the + required testing prongs apply to this design. +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md) + records the selected field syntax and last-wins duplicate-path semantics. + ## Spelling policy Run `make spelling` to enforce en-GB-oxendict prose spelling. The generated diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md new file mode 100644 index 0000000..bc0684c --- /dev/null +++ b/docs/mpsc-log-design.md @@ -0,0 +1,713 @@ +# mpsc-log design + +- **Status:** Draft v0.1. +- **Audience:** Implementers, reviewers, and workflow operators. +- **Last substantive revision:** 2026-06-29. +- **Companion documents:** + - [Terms of reference](terms-of-reference.md). + - [Context](context.md). + - [Lock file naming ADR](adr-001-lock-file-naming.md). + - [Testing strategy ADR](adr-002-testing-strategy.md). + - [`jo` field syntax ADR](adr-003-jo-field-syntax-and-duplicate-keys.md). + - [Event schema](mpsc-log-event-schema.json). + - [Sidecar example](mpsc-log-sidecar.example.toml). + - [Users' guide](users-guide.md). + - [Developer guide](developers-guide.md). + +## 1. Context + +`mpsc-log` is a short-lived Rust command-line interface (CLI) for appending one +JSON object to a shared JSON Lines journal. It exists because multi-agent +workflows need structured telemetry without running a logging daemon or +rewriting the same `jo`, `flock`, append, timeout, and rotation glue in every +caller. + +The first integration target is the df12-build Open Dynamic Workflows (ODW) +workflow. Agents in that workflow will write journal records into a +caller-owned workflow sidecar directory. The records measure phase timing, +review rounds, task shape, CodeRabbit waits and HTTP 429s, audit yield, +remediation lane outcomes, and post-merge defect escape. + +## 2. Research baseline + +JSON Lines is the journal format because it requires UTF-8, one valid JSON +value per line, and `\n` line termination; it is also described as suitable for +log files and cooperating processes.[^jsonl] `mpsc-log` tightens the JSON Lines +contract by requiring every root value to be an object. + +The CLI field grammar is `jo`-inspired and uses selected `jo` field syntax +where that serves object-record logging. `jo` supports `key=value`, +`key@value`, type coercion flags, object paths, file-value prefixes, array +construction, and duplicate object keys.[^jo] `mpsc-log` rejects array roots +and builds an object-map `Record`, so duplicate paths resolve by last write +rather than producing repeated textual JSON object names. That compatibility +boundary is recorded in +[ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md). + +The default `timestamp` field uses RFC 3339 UTC. RFC 3339 defines an Internet +profile of ISO 8601 for timestamps, recommends UTC for interoperability, and +uses `date-time = full-date "T" full-time`.[^rfc3339] + +The sidecar configuration uses TOML v1.0.0. TOML v1.0.0 is UTF-8, maps +unambiguously to a hash table, supports dotted keys, and forbids defining the +same key multiple times.[^toml] + +Prior art informs the concurrency and rotation choices. `flock(1)` exposes +exclusive locks, non-blocking mode, timeout mode, and explicit timeout exit +status, while warning that NFS and CIFS may have limited lock support.[^flock] +`logrotate(8)` rotates, compresses, removes, and handles logs by size; its +`copytruncate` documentation names a small data-loss window, which this design +avoids by rotating under the same lock used for appends.[^logrotate] + +The Rust implementation baseline is conservative: + +| Need | Choice | Reason | +| ----------- | ------------------------------ | ------------------------------------------------------------------------------------------------- | +| CLI parsing | `clap` builder API | Handles help/version/error rendering while allowing raw selected `jo` field tails.[^clap] | +| JSON | `serde`, `serde_json` | Standard Rust serialization stack, used by the JSON output adapter rather than by domain modules. | +| TOML | `toml` crate, TOML v1.0 subset | Current crate supports TOML parsing; the product contract remains v1.0. | +| Time | `jiff` | Provides `Timestamp::now()` and RFC 3339-style instant formatting with nanosecond support.[^jiff] | +| Locking | `fs4` sync feature | Provides cross-platform file locks without requiring a daemon.[^fs4] | +| Gzip | `flate2` default backend | Supports gzip streams using a safe Rust default backend.[^flate2] | +| Error types | `thiserror` | Exposes semantic errors to the CLI boundary. | + +Table 1: Dependency baseline for implementation. + +The gzip choice is a conservative baseline, not a settled performance decision. +The roadmap requires a compression backend benchmark before gzip implementation +so `gzp` and `gzippy` can be evaluated against `flate2` for speed, +atomic-output integration, dependency risk, and portability.[^gzp] [^gzippy] + +## 3. Goals and non-goals + +### 3.1 Goals + +- Create missing parent directories for the journal path. +- Parse the first argument as the journal path and remaining arguments as + selected `jo`-inspired field words and coercion flags. +- Merge sidecar defaults, CLI fields, schema-guided coercions, and generated + timestamp values into one JSON object. +- Serialize exactly one compact JSON object plus `\n` per successful + invocation. +- Use one lock per journal to serialize repair, rotation, compression, and + append. +- Time out lock acquisition after five seconds by default. +- Rotate after 1 MiB by default, keep four plain generations, and gzip older + generations. +- Optionally rotate on UTC hourly, daily, or weekly boundaries while still + splitting within a period when the size threshold is reached. +- Provide stable exit codes and diagnostics for agents. + +### 3.2 Non-goals + +- `mpsc-log` does not provide query, dashboard, alerting, or retention + analysis. +- `mpsc-log` does not discover df12-build workflow sidecar directories. +- `mpsc-log` does not provide distributed locking across hosts. +- `mpsc-log` does not preserve textual duplicate JSON object keys. +- `mpsc-log` does not guarantee chronological record order by `timestamp`. +- `mpsc-log` is `jo`-inspired rather than `jo` compatible: it omits `jo` + formatting, pretty-printing, array-root, and version options; see section 5 + and [ADR 003](adr-003-jo-field-syntax-and-duplicate-keys.md). + +## 4. Architecture + +The design is a synchronous CLI with a small domain core surrounded by +adapters. The domain interprets field words, merges configuration, builds a +`Record`, and computes rotation actions. It reaches every external effect +through a port. Adapters own directory creation, sidecar reads, lock +acquisition, append, truncate repair, rename, gzip compression, and the +translation between external formats and domain types. + +```mermaid +flowchart LR + Agent[Agent or workflow] + + subgraph inbound [Input adapters] + Args[args: process arguments] + Cfg[config: sidecar TOML] + end + + subgraph core [Domain] + Fields[fields: field interpretation] + Rec[record: builds the Record] + Jnl[journal: plans repair and rotation] + Errs[errors: semantic errors] + end + + subgraph ports [Ports] + Clk[[Clock]] + Store[[JournalStore]] + end + + subgraph outbound [Output adapters] + Json[JSON serializer] + Fs[fs: filesystem] + Exit[main.rs: sysexits mapping] + end + + Agent --> Args + Args --> Fields + Cfg --> Rec + Fields --> Rec + Clk --> Rec + Rec --> Jnl + Jnl --> Json + Json --> Store + Jnl --> Store + Errs --> Exit + Store -. implemented by .-> Fs + Fs --> Lock[(Journal lock)] + Fs --> JournalFile[(JSONL journal)] + Fs --> Rotated[(Rotated logs)] +``` + +Figure 1: Runtime component topology. The domain reaches the outside world only +through the `Clock` and `JournalStore` ports; adapters own the process +arguments, the sidecar TOML, JSON serialization, the filesystem, and the +`sysexits` mapping. + +The lock file is the coordination boundary. The naming decision is recorded in +[ADR 001: Lock file naming](adr-001-lock-file-naming.md): `mpsc-log` appends +`.lock` to the complete journal filename in the same directory, reserves +`.lock` journal filenames, and rejects journal paths whose derived sidecar path +would equal the journal path. Every invocation creates parent directories +first, opens the lock file with create semantics, acquires an exclusive lock, +and only then reads configuration, repairs the journal tail, rotates, +compresses, and appends. + +The lock timeout is the one setting that cannot come from the locked +configuration read, because the lock must already be held to read configuration +authoritatively. The single acquisition attempt uses a five-second default, +which an unlocked advisory pre-read of the sidecar's `[locking] timeout_ms` may +override. That pre-read is advisory only: it selects how long this invocation +waits for the lock and never feeds repair, rotation, coercion, or defaults. A +missing, unreadable, or invalid pre-read falls back to five seconds without +failing the invocation. Once the lock is held, the authoritative configuration +read supplies one coherent view for everything else. There is no second +acquisition attempt. + +### 4.1 Domain record model + +The domain owns its record types so that no domain module depends on a +serialization crate. Two types carry every journal entry: + +- `Value` is the domain value tree, with null, boolean, integer, float, + string, array, and object variants. The object variant is a map from key to + `Value`. +- `Record` is the root of one entry. It is always an object map from key to + `Value`, so a record can never serialize as an array or a bare scalar. + +The domain defines this behaviour over those types: + +- Object paths. A path is the sequence of key segments produced by splitting + a field key on the configured delimiter. Writing a path creates any missing + intermediate objects. +- Nested objects. Writing `task.id` creates `task` as an object variant and + sets `id` within it. Writing through a segment whose existing value is not an + object replaces that value with a new object. +- Arrays. The `key[]` form appends to an array at that path, creating an + empty array first when the path is unset. +- Scalars. Null, boolean, integer, float, and string values are stored as the + matching `Value` variant rather than as text. +- Coercion results. Coercion is a domain policy mapping one raw field word + and its coercion source to one `Value`. Explicit `-s`, `-n`, and `-b` flags + win, then the matching `[schema]` entry, then default `jo`-inspired inference. +- Last-wins replacement. Writing a path that already holds a value replaces + that value. Ordering follows section 6, so sidecar defaults seed the record + and each later CLI write wins at its path. A `Record` therefore holds one + value per path and never repeats a key. + +### 4.2 Ports + +The domain reaches external effects only through ports. A port is a trait +declared by the domain in domain terms; an adapter implements it with a +concrete API. + +| Port | Responsibility | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Clock` | Supply the invocation instant used for the generated `timestamp` field, so tests can fix time without mutating process state. | +| `JournalStore` | Perform every journal-directory effect: create parent directories, read the sidecar, acquire and release the journal lock, inspect and repair the active tail, append, truncate, rename, write compression output, and sync metadata. | + +Table 2: Domain ports. + +No domain module names a filesystem API, a locking crate, a TOML parser, a JSON +serializer, or a `sysexits` code. The domain returns semantic errors, and the +process boundary maps them to exit codes as section 10 describes. + +### 4.3 Adapters + +Adapters sit on both sides of the domain: + +- The CLI adapter reads process arguments and hands the domain raw field + words in their original order. +- The TOML adapter parses the sidecar at the input boundary and converts + `[defaults]` and `[schema]` data into domain types, so the domain receives + `Value` defaults and coercion names rather than TOML values. +- The JSON adapter converts a `Record` into one compact JSON object at the + output boundary through `serde_json`, emitting object keys in a deterministic + order. +- The filesystem adapter implements `JournalStore` over the real filesystem. + A test adapter implements the same port for fault injection. +- The process adapter, `src/main.rs`, maps the semantic error type to + `sysexits` codes. + +## 5. CLI contract + +The first positional argument is always the journal path: + +```plaintext +mpsc-log [field-or-flag ...] +``` + +The field tail accepts this subset: + +| Syntax | Meaning | +| ---------------- | ---------------------------------------------------------- | +| `key=value` | Insert a value using schema coercion or default inference. | +| `key@value` | Insert a boolean using `jo` truthiness. | +| `key:=path` | Read JSON from `path` and insert it at `key`. | +| `key=@path` | Read `path` as UTF-8 text. | +| `key=:path` | Read `path` as JSON. | +| `key=%path` | Read `path` and base64 encode the bytes. | +| `-s key=value` | Force string coercion for the following word. | +| `-n key=value` | Force number coercion for the following word. | +| `-b key=value` | Force boolean coercion for the following word. | +| `-d.` or `-d .` | Set the object-path delimiter for later keys. | +| `key[]=value` | Append to an array at `key`. | +| `key[sub]=value` | Insert into an object at `key.sub`. | + +Unsupported `jo` options such as `-a`, `-p`, `-f`, `-D`, `-e`, `-v`, and `-V` +fail with `EX_USAGE`. The root is always an object. The field syntax is +`jo`-inspired rather than textually `jo` compatible: duplicate writes to the +same object path use last-wins semantics inside the record map. + +## 6. Sidecar configuration + +The sidecar path replaces the journal filename extension with `.toml`; if the +journal has no extension, the sidecar path appends `.toml`. For example, +`run.jsonl` uses `run.toml`, and `run` uses `run.toml`. This is a configuration +sharing rule, not a lock naming rule: accepted same-stem journals such as `run`, +`run.jsonl`, and `run.ndjson` have distinct lock files but share `run.toml`. +Callers needing independent sidecars must choose distinct stems or directories. + +The sidecar shape is defined by +[mpsc-log-sidecar.example.toml](mpsc-log-sidecar.example.toml). Configuration +has four tables: + +| Table | Responsibility | +| ------------ | --------------------------------------------------------------------------------------------------- | +| `[rotation]` | `schedule`, `max_bytes`, plain generation count, compressed generation count, and gzip policy. | +| `[locking]` | Lock timeout, read by an unlocked advisory pre-read, and partial-tail repair mode. | +| `[defaults]` | Default JSON fields inserted before CLI fields. | +| `[schema]` | Object paths mapped to coercion names: `string`, `integer`, `number`, `boolean`, `json`, or `null`. | + +Merge and coercion order is deterministic: + +| Step | Rule | Winner | +| ---- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Start with sidecar `[defaults]`, which the TOML adapter converts into domain `Value` equivalents. | Sidecar defaults seed the record. | +| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | +| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | +| 4 | Validate an override timestamp, if present. | A sidecar or CLI `timestamp` must match the canonical RFC 3339 UTC format in the event schema; an invalid override returns `EX_DATAERR` before append. | +| 5 | Insert the generated invocation timestamp only when no override exists. | The generated canonical UTC `timestamp` is added only when defaults and CLI fields did not produce one. | +| 6 | Serialize the resulting object. | The merged record is written as one compact JSON object. | + +The sidecar schema never overrides an explicit CLI coercion flag. Schema +entries affect only values supplied for the matching object path; they do not +change sidecar defaults or the generated timestamp. When default `jo`-inspired +inference is used, a field word that is valid JSON parses into the matching +`Value` variant, empty `key=` becomes a null `Value`, and other words remain +strings. + +## 7. Journal write protocol + +Each invocation follows one critical section: + +```mermaid +sequenceDiagram + participant A as Agent + participant M as mpsc-log + participant L as Lock file + participant J as Journal + participant R as Rotated logs + + A->>M: mpsc-log path fields... + M->>M: create parent directories + M->>L: open or create lock file + M->>L: acquire exclusive lock with pre-read or default timeout + M->>J: validate final record and repair partial tail + M->>R: rotate and compress if threshold or schedule requires it + M->>J: append record plus newline + M->>L: release lock + M-->>A: exit 0 +``` + +Figure 2: Critical-section protocol. + +The writer records the active file length before appending. If `write_all` +returns an error, the writer truncates the active file back to that length +before returning. If the process dies during a write, the next invocation first +classifies the active file tail while holding the journal lock: + +- Empty files need no repair. +- Files ending without `\n` have an unterminated final line. When + `repair_partial_tail = true`, the writer scans backward to the previous `\n`, + truncates the unterminated bytes, and appends the new record. When + `repair_partial_tail = false`, the invocation fails with `EX_DATAERR` and + leaves the file unchanged. +- Files ending with `\n` have no partial tail. The complete final line is the + bytes between the previous `\n` and the terminal `\n`. That line must be a + valid JSON object unless the file has no lines. If the complete final line is + invalid JSON, not an object, or empty, the invocation fails closed with + `EX_DATAERR` and leaves the file unchanged. + +A record is committed once its bytes and terminating newline are written and +flushed; before that point it is uncommitted. The writer restores the recorded +pre-append length on any in-process failure after bytes reach the file, +including flush and filesystem metadata failures, and all of these failures map +to `EX_IOERR`. A successful rollback leaves the record uncommitted, so a later +retry cannot duplicate it. + +A failed rollback truncate leaves the commit status unknown. The written bytes +stay on disk, and which repair rule applies depends on how far the write got: +an unterminated tail is removed by partial-tail repair, whereas a complete +newline-terminated record is preserved by the rule above and is therefore +committed. The invocation returns `EX_IOERR` without knowing which case +occurred, which is why repair classifies the tail rather than trusting the +previous writer. + +`mpsc-log` is therefore at-least-once whenever the writer cannot confirm the +outcome. A retry after `EX_IOERR` can duplicate a record that a failed rollback +truncate left complete, and a process killed between the write and the exit can +leave a complete record the caller never saw acknowledged. `mpsc-log` keeps no +unknown-commit reconciliation state, so callers needing exactly-once semantics +must carry their own idempotency key in the record and deduplicate when reading. + +The repair path does not quarantine files and does not truncate a +newline-terminated corrupt record by default. It also does not scan every +historical line on each append; it validates only the active file's complete +final line and any unterminated tail that can affect the next append. + +The append path uses `OpenOptions::append(true).create(true)`. The lock, not +append mode alone, provides cross-process serialization. The design treats NFS, +CIFS, and other filesystems with weak advisory locking as unsupported unless +later verification proves the target mount behaves correctly. + +## 8. Rotation protocol + +Rotation runs while holding the journal lock. The sidecar `schedule` value is +one of `none`, `hourly`, `daily`, or `weekly`; the default is `none`. There is +no `max_age` setting in the v1 configuration surface. + +When `schedule = "none"`, rotation is size-only. The active threshold +(`max_bytes`), the plain generation count `P` (`plain_generations`), and the +compressed generation count `C` (`compressed_generations`) are `[rotation]` +sidecar settings; the values below are the defaults and are overridable through +the sidecar: + +- active threshold (`max_bytes`): 1 MiB; +- plain generations `P` (`plain_generations`): four; +- compressed generations `C` (`compressed_generations`): 32; +- newest plain rotation: `.1`; +- oldest plain rotation before compression: `.P` (`.4` + by default); +- compressed rotations: `.(P+1).gz` through `.(P+C).gz` + (`.5.gz` through `.36.gz` by default). + +The rotation names above follow from the default counts; overriding the counts +shifts the highest plain and compressed suffixes accordingly. + +For `run.jsonl`, the active path is `run.jsonl`; the newest plain rotation is +`run.1.jsonl`; the first compressed rotation is `run.5.jsonl.gz`. + +`plain_generations` and `compressed_generations` are non-negative integers that +count retained archive generations; the active file is always separate. +Generation `1` is the newest archive and generation `P + C` is the oldest, so +the indices below derive from `P` and `C` rather than fixed numbers. Semantic +validation rejects negative counts with `EX_CONFIG`, and also rejects +`P + C == 0`: rotation must retain at least one generation. A configuration +that would leave rotation with nowhere to put the previous journal is a +configuration error rather than a supported mode, because discarding the +rotated journal cannot be made atomic and would contradict the guarantee that a +failed invocation leaves the previous journal readable. + +One zero count is valid because `P + C` is still at least one. When `P` is +zero, no plain archives exist and the active file is gzipped straight into +generation `1`. When `C` is zero, no archive is compressed and the oldest plain +generation is evicted rather than gzipped. + +Rotation is evaluated after tail repair and before the append. The invocation +compares the repaired active file length plus the length of the serialized +pending record, including its terminating newline, against `max_bytes`. It +rotates only when the active file is non-empty and that combined length exceeds +`max_bytes`. When the active file is empty, the pending record is appended +directly, so an oversized record never rotates an empty active file into an +archive. + +Size-only rotation is a prepare phase followed by a commit point. The prepare +phase only creates files and renames them within the journal directory, so +every source generation stays on disk and recoverable until the rotation +commits. Superseded sources are renamed aside to staging names rather than +unlinked; staging names and the manifest below are reserved names derived from +the journal filename, alongside the `.lock` name from ADR 001. + +Before touching any file, the invocation writes a rotation manifest recording +the planned layout transition. The manifest is written through +`atomic-write-file` in the journal directory and is removed at the commit +point, so its presence means a rotation was interrupted. + +Steps 1 to 5 prepare the new layout, oldest-to-newest; steps 6 and 7 commit it: + +1. Stage the eviction. If generation `P + C` exists, rename it aside to a + staging name. Staging names share the journal directory, so every rename + stays within one filesystem and is atomic. +2. Rename each compressed generation `i` from `P + C - 1` down to `P + 1` into + generation `i + 1`. +3. Gzip plain generation `P` into generation `P + 1`, then rename generation + `P` aside to a staging name rather than unlinking it. Skip this step when + `C` is zero (generation `P` is already the staged eviction) or when `P` is + zero (the active file is compressed in step 5 instead). +4. Rename each plain generation `i` from `P - 1` down to `1` into generation + `i + 1`. +5. Move the active file into generation `1`. When `P` is at least one this is a + rename, which is atomic on a supported local filesystem. When `P` is zero, + gzip the active file into generation `1` and then rename the active file + aside to a staging name. +6. Create a fresh active file by appending the pending record. +7. Unlink the staged files, then remove the manifest. + +Every prepare step is idempotent: it is applied only when its source exists and +its target does not, so replaying the manifest can neither duplicate nor skip a +generation. Because nothing is unlinked before step 7, the prepare phase is +also reversible by renaming each staged or shifted file back. + +Rotation therefore settles into either the pre-rotation or the post-rotation +layout, never a mixture of the two: + +- An error during prepare reverses the renames already applied, discards any + uncommitted gzip temporary, removes the manifest, and aborts before the + append, leaving the pre-rotation layout intact. +- An error during the append in step 6 leaves the rotated layout in place with + the previous contents readable in generation `1`. The invocation still + performs step 7 before reporting the append failure, because the rotation + itself completed. +- A process killed at any point leaves the manifest behind. The next + invocation acquires the journal lock, finds the manifest, and drives the + recorded plan to a settled state before evaluating its own rotation: it + completes the outstanding prepare steps and commits when the plan can still + be completed, and otherwise reverses the applied renames and removes the + manifest. Only then does it handle its own record. + +When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period +boundaries derived from the invocation timestamp: + +| Schedule | Period beginning | Base rotated filename | +| -------- | --------------------------------------- | ------------------------- | +| `hourly` | UTC hour beginning | `run.2026-06-29T14.jsonl` | +| `daily` | UTC day beginning | `run.2026-06-29.jsonl` | +| `weekly` | UTC ISO week beginning, Monday 00:00:00 | `run.2026-W27.jsonl` | + +Scheduled rotation is opportunistic rather than daemon-driven. If no invocation +occurs at the exact boundary, the next invocation rotates the previous active +segment before appending the new record. The rotated filename is still based on +the period that produced the records, not the time the rotation was observed. + +The scheduled protocol is: + +1. Determine the pending record period from the invocation timestamp. +2. Determine the active file period from the first complete record in the + active log, falling back to the pending period when the active log is empty. +3. If the active period is older than the pending record period, finalize the + previous period first. Scan that period's existing archives: use its + unsuffixed scheduled filename only when no size-split archive exists and the + unsuffixed filename is unused; otherwise use the next unused positive + numeric suffix. Rotate the active log into that collision-free filename. The + active path is then a fresh empty file for the pending period. +4. Evaluate `max_bytes` against the current active file after any + period-boundary rollover. If the active file already contains data and that + data plus the pending record would exceed `max_bytes`, rotate it into the + pending period's next size-split filename and continue with a fresh active + file. When the active file is empty, append the oversized pending record + directly rather than rotating an empty active file into an archive. +5. Append the pending record to the active log. + +Within a scheduled period, reaching `max_bytes` before the time boundary +creates an interim size split. Size splits add a numeric suffix within that +period: `run..1.jsonl`, `run..2.jsonl`, and so on. The final +scheduled archive uses `run..jsonl` only when that filename is unused +and the period has no size splits. Otherwise, it uses the next unused positive +numeric suffix, so it never overwrites an archive or collides with +`run..1.jsonl`. + +Retention and compression are period-based in scheduled mode. The newest +`plain_generations` completed periods remain plain, including all size-split +files inside those periods. Older retained periods are gzipped up to +`compressed_generations`, and periods beyond +`plain_generations + compressed_generations` are deleted. This keeps hourly, +daily, and weekly retention predictable even when one busy period produces many +size-split files. + +All gzip output is written through `atomic-write-file`, then committed. A +failed compression leaves the source generation in place and aborts the +invocation before appending the new record. + +## 9. df12-build journal records + +The normative JSON Schema lives in +[mpsc-log-event-schema.json](mpsc-log-event-schema.json). The schema requires +`timestamp`, conditionally requires the table's fields for each named event, +permits additional fields, and reserves structured namespaces for `run`, `task`, +`attempt`, `coderabbit`, `audit`, and `defect`. + +The df12-build integration should emit these event names first: + +| Event | Required fields | Purpose | +| -------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| `phase.started` | `run.id`, `attempt.phase`, `attempt.agent` | Start timing a workflow phase. | +| `phase.finished` | `run.id`, `attempt.phase`, `attempt.status`, `attempt.duration_ms` | Measure phase duration and outcome. | +| `review.round` | `task.id`, `attempt.phase`, `attempt.round`, `attempt.status` | Measure design, code, and expert review round distributions. | +| `task.finished` | `task.id`, `task.phase`, `task.work_items`, `task.changed_files`, `task.dependency_depth` | Correlate outcome with roadmap shape. | +| `coderabbit.attempt` | `coderabbit.status`, `coderabbit.wait_ms`, `coderabbit.retry`, `coderabbit.deferred` | Measure waits, HTTP 429s, deferrals, and retry success. | +| `audit.finding` | `audit.severity`, `audit.lane`, `audit.finding_count` | Measure audit yield and remediation routing. | +| `defect.escape` | `defect.source`, `defect.escaped_from`, `defect.severity` | Measure defects found after merge. | + +Example invocation: + +```sh +mpsc-log .odw/run-20260629/journal.jsonl \ + event=review.round \ + run.id=df12-20260629-001 \ + run.workflow=df12-build-odw \ + task.id=1.2.8 \ + attempt.phase=code-review \ + -n attempt.round=2 \ + attempt.status=changes-requested +``` + +## 10. Errors and exit codes + +The library exposes a semantic error enum. The binary maps it to `sysexits` +style process statuses: + +| Exit | Name | Condition | +| ---- | -------------- | ---------------------------------------------------------------------------------------------------- | +| 0 | `OK` | Record appended. | +| 64 | `EX_USAGE` | Invalid CLI syntax, unsupported `jo` option, non-object root, or duplicate invalid path. | +| 65 | `EX_DATAERR` | Invalid JSON, invalid TOML, invalid schema coercion, corrupt final line, or malformed sidecar value. | +| 73 | `EX_CANTCREAT` | Parent directory, lock file, journal, or rotated file cannot be created. | +| 74 | `EX_IOERR` | Write, flush, truncate, rename, compression, or filesystem metadata failure. | +| 75 | `EX_TEMPFAIL` | Lock timeout. | +| 78 | `EX_CONFIG` | Sidecar configuration is syntactically valid TOML but semantically invalid. | + +Diagnostics go to standard error as one line: + +```plaintext +mpsc-log: : : +``` + +The command writes nothing to standard output on success. + +## 11. Correctness properties and verification + +The testing strategy is recorded in +[ADR 002: Testing strategy](adr-002-testing-strategy.md). That ADR maps the +repository's required unit, behavioural, snapshot, end-to-end, property-based, +bounded-model, and proof prongs to the design surfaces below. + +The implementation must satisfy these properties: + +| Property | Verification method | +| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| Successful invocations produce exactly one valid JSON object line. | Behavioural tests and property tests over generated field words. | +| Concurrent successful invocations produce the same number of complete records as successes. | Stress test with many processes writing one journal. | +| Rotation preserves successful records across active, plain rotated, scheduled rotated, and compressed rotated files. | End-to-end test that forces size and scheduled rotation under concurrent writers and counts decoded records. | +| Detected write failures do not leave a partial final line. | Fault-injection adapter test that fails after partial writes and checks truncate repair. | +| Unterminated partial tails are repaired before the next append when repair is enabled. | Fixture with malformed trailing bytes followed by a successful append. | +| Newline-terminated corrupt final records fail closed. | Fixture ending in `\n` with an invalid final line returns `EX_DATAERR` without truncating. | +| CLI coercion follows the selected `jo` field syntax. | Parameterized examples cover the accepted `jo`-inspired behaviours and last-wins duplicate paths. | +| Sidecar/CLI precedence is deterministic. | Table-driven tests covering defaults, schema coercion, explicit flags, and `timestamp`. | + +The combination surface is selected `jo` syntax form × coercion source × object +path × sidecar default × rotation schedule × rotation state × lock contention. +The test suite must cover pairwise combinations across those axes, plus +targeted cases for size and scheduled rotation under contention and failed +filesystem operations. + +Network filesystems remain outside the correctness claim until a separate +platform verification matrix proves lock and rename behaviour for a named +filesystem and mount configuration. + +## 12. Module structure + +`src/main.rs` owns CLI startup and process exit mapping only. The library owns +the implementation, split into domain modules, ports, and adapters: + +| Module | Layer | Responsibility | +| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `args` | Adapter | Read process arguments and hand on the journal path and the raw selected `jo` field tail in order. | +| `fields` | Domain | Interpret field words, object paths, coercion flags, and file-value forms into domain `Value` results. | +| `config` | Adapter | Parse and validate sidecar TOML at the input boundary, converting `[defaults]` and `[schema]` data into domain types. | +| `record` | Domain | Build the `Record`: merge defaults and CLI fields, apply coercion policy and object-path updates, and insert the generated timestamp. | +| `journal` | Domain | Plan tail repair, rotation, retention, and append, and drive them through the `JournalStore` port. | +| `errors` | Domain | Semantic error type. Exit-code mapping belongs to the process boundary, not to this module. | +| `clock` | Domain port | Declare the `Clock` port; an infrastructure implementation supplies the real instant. | +| `fs` | Adapter | Implement `JournalStore` over the real filesystem, alongside the fault-injection test double. | + +Table 3: Module layers and responsibilities. + +Two boundary concerns are not domain modules. JSON output serialization is an +adapter step at the output boundary, converting the domain `Record` into one +compact JSON object through `serde_json` immediately before the bytes reach the +`JournalStore` port. Process exit mapping belongs to `src/main.rs`, which +translates the semantic error type from `errors` into a `sysexits` code. + +The first implementation should expose no stable library API beyond what the +binary needs. Public library exports remain internal support until a roadmap +item explicitly commits to a supported API. + +## 13. Deferred ADRs + +- Rotation naming, compression, and retention defaults. +- CLI-only product boundary versus public Rust library API. + +## Appendix A. References + +[^jsonl]: [JSON Lines](https://jsonlines.org/), accessed 2026-06-29. + +[^jo]: Jan-Piet Mens, + [`jo` manual](https://github.com/jpmens/jo/blob/master/jo.md), + accessed 2026-06-29. + +[^rfc3339]: G. Klyne and C. Newman, + [RFC 3339: Date and time on the Internet: timestamps](https://www.rfc-editor.org/rfc/rfc3339), + July 2002, accessed 2026-06-29. + +[^toml]: Tom Preston-Werner, Pradyun Gedam, et al., + [TOML v1.0.0](https://toml.io/en/v1.0.0), accessed 2026-06-29. + +[^flock]: Michael Kerrisk, + [flock(1) Linux manual page](https://man7.org/linux/man-pages/man1/flock.1.html), + util-linux manual, accessed 2026-06-29. + +[^logrotate]: Michael Kerrisk, + [logrotate(8) Linux manual page](https://man7.org/linux/man-pages/man8/logrotate.8.html), + accessed 2026-06-29. + +[^clap]: [`clap` crate documentation](https://docs.rs/clap/latest/clap/), + version 4.6.1, accessed 2026-06-29. + +[^jiff]: [`jiff` crate documentation](https://docs.rs/jiff/latest/jiff/), + version 0.2.29, accessed 2026-06-29. + +[^fs4]: [`fs4` crate documentation](https://docs.rs/fs4/latest/fs4/), + version 1.1.0, accessed 2026-06-29. + +[^flate2]: [`flate2` crate documentation](https://docs.rs/flate2/latest/flate2/), + version 1.1.9, accessed 2026-06-29. + +[^gzp]: [`gzp` crate documentation](https://docs.rs/gzp/latest/gzp/), + version 2.0.2, accessed 2026-06-29. + +[^gzippy]: [`gzippy` crate documentation](https://docs.rs/gzippy/latest/gzippy/), + version 0.8.0, accessed 2026-06-29. diff --git a/docs/mpsc-log-event-schema.json b/docs/mpsc-log-event-schema.json new file mode 100644 index 0000000..fd2385a --- /dev/null +++ b/docs/mpsc-log-event-schema.json @@ -0,0 +1,152 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/leynos/mpsc-log/docs/mpsc-log-event-schema.json", + "title": "mpsc-log journal record", + "type": "object", + "additionalProperties": true, + "required": ["timestamp"], + "properties": { + "timestamp": { + "type": "string", + "format": "date-time", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9]:([0-5][0-9]|60)(\\.[0-9]{1,9})?Z$", + "description": "Canonical RFC 3339 UTC invocation timestamp using a Z suffix." + }, + "schema_version": { + "type": "integer", + "minimum": 1 + }, + "event": { + "type": "string", + "minLength": 1 + }, + "run": { + "type": "object", + "additionalProperties": true, + "properties": { + "id": { "type": "string" }, + "workflow": { "type": "string" }, + "sidecar_dir": { "type": "string" } + } + }, + "task": { + "type": "object", + "additionalProperties": true, + "properties": { + "id": { "type": "string" }, + "phase": { "type": "string" }, + "work_items": { "type": "integer", "minimum": 0 }, + "changed_files": { "type": "integer", "minimum": 0 }, + "dependency_depth": { "type": "integer", "minimum": 0 } + } + }, + "attempt": { + "type": "object", + "additionalProperties": true, + "properties": { + "agent": { "type": "string" }, + "phase": { "type": "string" }, + "round": { "type": "integer", "minimum": 1 }, + "status": { "type": "string" }, + "duration_ms": { "type": "integer", "minimum": 0 } + } + }, + "coderabbit": { + "type": "object", + "additionalProperties": true, + "properties": { + "status": { "type": "string" }, + "http_status": { "type": "integer", "minimum": 100, "maximum": 599 }, + "wait_ms": { "type": "integer", "minimum": 0 }, + "retry": { "type": "boolean" }, + "deferred": { "type": "boolean" } + } + }, + "audit": { + "type": "object", + "additionalProperties": true, + "properties": { + "severity": { "type": "string" }, + "lane": { "type": "string" }, + "finding_count": { "type": "integer", "minimum": 0 } + } + }, + "defect": { + "type": "object", + "additionalProperties": true, + "properties": { + "source": { "type": "string" }, + "escaped_from": { "type": "string" }, + "severity": { "type": "string" } + } + } + }, + "allOf": [ + { + "if": { "properties": { "event": { "const": "phase.started" } }, "required": ["event"] }, + "then": { + "required": ["run", "attempt"], + "properties": { + "run": { "required": ["id"] }, + "attempt": { "required": ["phase", "agent"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "phase.finished" } }, "required": ["event"] }, + "then": { + "required": ["run", "attempt"], + "properties": { + "run": { "required": ["id"] }, + "attempt": { "required": ["phase", "status", "duration_ms"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "review.round" } }, "required": ["event"] }, + "then": { + "required": ["task", "attempt"], + "properties": { + "task": { "required": ["id"] }, + "attempt": { "required": ["phase", "round", "status"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "task.finished" } }, "required": ["event"] }, + "then": { + "required": ["task"], + "properties": { + "task": { "required": ["id", "phase", "work_items", "changed_files", "dependency_depth"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "coderabbit.attempt" } }, "required": ["event"] }, + "then": { + "required": ["coderabbit"], + "properties": { + "coderabbit": { "required": ["status", "wait_ms", "retry", "deferred"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "audit.finding" } }, "required": ["event"] }, + "then": { + "required": ["audit"], + "properties": { + "audit": { "required": ["severity", "lane", "finding_count"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "defect.escape" } }, "required": ["event"] }, + "then": { + "required": ["defect"], + "properties": { + "defect": { "required": ["source", "escaped_from", "severity"] } + } + } + } + ] +} diff --git a/docs/mpsc-log-sidecar.example.toml b/docs/mpsc-log-sidecar.example.toml new file mode 100644 index 0000000..067a16d --- /dev/null +++ b/docs/mpsc-log-sidecar.example.toml @@ -0,0 +1,30 @@ +[rotation] +schedule = "none" +max_bytes = 1048576 +plain_generations = 4 +compressed_generations = 32 +gzip_after_plain_generations = true + +[locking] +timeout_ms = 5000 +# Repairs only unterminated final bytes; newline-terminated corrupt records +# fail closed with EX_DATAERR. +repair_partial_tail = true + +[defaults] +schema_version = 1 +event = "workflow.event" + +[schema] +"run.id" = "string" +"run.workflow" = "string" +"task.work_items" = "integer" +"task.changed_files" = "integer" +"task.dependency_depth" = "integer" +"attempt.round" = "integer" +"attempt.duration_ms" = "integer" +"coderabbit.http_status" = "integer" +"coderabbit.wait_ms" = "integer" +"coderabbit.retry" = "boolean" +"coderabbit.deferred" = "boolean" +"audit.finding_count" = "integer" diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..4085555 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,580 @@ +# mpsc-log roadmap + +This roadmap translates the terms of reference and technical design into an +outcome-oriented delivery sequence for `mpsc-log`. It does not promise dates. +Each phase carries one testable idea at the GIST level; each step answers a +sequencing question; each task is a review-sized execution unit with explicit +dependencies and source citations. + +The primary source documents are: + +- [terms of reference](terms-of-reference.md); +- [technical design](mpsc-log-design.md); +- [context](context.md); +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md); +- [ADR 002: Testing strategy](adr-002-testing-strategy.md); +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md); +- [event schema](mpsc-log-event-schema.json); +- [sidecar example](mpsc-log-sidecar.example.toml). + +No RFCs exist yet, so the first phase records the remaining decisions that +would otherwise force rework. + +## 1. Foundational contracts and build spine + +Idea: if `mpsc-log` ratifies its v1 contracts, crate boundary, and validation +spine before feature work starts, later slices can converge on one small CLI +instead of repeatedly reopening syntax, locking, and retention decisions. + +This phase resolves the decisions that affect every subsequent pull request: +the selected `jo` field syntax, public API boundary, filesystem support policy, +rotation naming, and the repository shape needed to validate a CLI whose main +risks are concurrency and persistence. + +### 1.1. Ratify the v1 decisions that would otherwise cause rework + +This step answers what `mpsc-log` v1 will promise to callers and what it will +explicitly leave out. Its outcome informs crate layout, user documentation, +error handling, and future compatibility work. See mpsc-log-design.md §§3, 5, +8, 12-13 and terms-of-reference.md §§6, 8-10. + +- [x] 1.1.1. Record lock-file naming and reserved suffixes in an ADR. + - See mpsc-log-design.md §§4, 6-7, 11, 13 and + adr-001-lock-file-naming.md. + - Success: the ADR defines adjacent lock naming, `.lock` and self-sidecar + journal rejection, sidecar sharing by stem, and the local-filesystem + coordination boundary. +- [x] 1.1.2. Record the selected `jo` field syntax and duplicate-key + behaviour in an ADR. + - See mpsc-log-design.md §§2, 5, 13, terms-of-reference.md §§6, 9, and + adr-003-jo-field-syntax-and-duplicate-keys.md. + - Success: the ADR names selected `jo`-inspired forms, rejected options, + object-root enforcement, object-path handling, and last-wins duplicate-key + semantics. +- [ ] 1.1.3. Record the CLI-only v1 product boundary in an ADR. + - See mpsc-log-design.md §§3, 10, 12-13 and terms-of-reference.md §§6, 9. + - Success: the ADR states that `src/main.rs` owns process exit mapping and + that library exports are internal until a later roadmap item changes that. +- [ ] 1.1.4. Record the rotation naming, compression, and retention policy in + an ADR. + - See mpsc-log-design.md §§3, 6, 8, 11, 13 and + terms-of-reference.md §§6, 8-9. + - Success: the ADR covers `schedule = "none"`, scheduled UTC period names, + size-split suffixes, gzip-after-four behaviour, and the absence of + `max_age`. + +### 1.2. Establish the implementation skeleton and quality gates + +This step answers whether the generated Rust scaffold can support the module +boundaries, dependency choices, and validation strategy described by the +design. It informs all later slices because every feature will pass through the +same CLI, domain, filesystem, and test seams. See mpsc-log-design.md §§2, 4, +10-12 and docs/repository-layout.md. + +- [ ] 1.2.1. Replace the scaffold library with the module skeleton described + by the design. + - Requires steps 1.1.1-1.1.3. + - See mpsc-log-design.md §§4, 10, 12. + - [ ] Add `args`, `fields`, `config`, `record`, `journal`, `errors`, + `clock`, and `fs` modules with module-level documentation. + - [ ] Keep `src/main.rs` limited to startup and exit-code mapping. + - Success: the crate builds with empty but documented module boundaries and + no stable public API beyond the binary's current needs. +- [ ] 1.2.2. Add the implementation dependencies with documented ownership. + - Requires 1.2.1. + - See mpsc-log-design.md §2 and docs/developers-guide.md. + - [ ] Add the selected crates for CLI parsing, JSON, TOML, time, locking, + gzip, atomic writes, and semantic errors using caret requirements. + - [ ] Document each dependency's purpose in the appropriate design or + developer guide location. + - Success: `make check-fmt`, `make lint`, and `make test` pass after the + dependency update. +- [ ] 1.2.3. Build deterministic test seams for time and filesystem effects. + - Requires 1.2.1 and 1.2.2. + - See mpsc-log-design.md §§7, 10-12 and + docs/reliable-testing-in-rust-via-dependency-injection.md. See + adr-002-testing-strategy.md. + - [ ] Provide injectable clock and filesystem adapter boundaries for record + timestamps, fault injection, and deterministic rotation fixtures. + - Success: failures such as partial writes, metadata errors, and fixed + timestamps can be exercised without mutating global process state. + +## 2. Day-one structured journal entries + +Idea: if one `mpsc-log` invocation can parse selected `jo`-inspired fields, +apply sidecar defaults, emit an RFC 3339 UTC timestamp, and append one JSONL +object, the tool already replaces the unsafe `jo >> file.jsonl` baseline for +simple workflows. + +This phase delivers the narrowest useful end-to-end command. It intentionally +starts without the full concurrency and rotation surface so that parser, +configuration, record-shaping, and diagnostics can stabilize against real CLI +examples before the filesystem protocol grows more complex. + +### 2.1. Prove the CLI field contract can build one object record + +This step answers whether the selected `jo` field syntax can be implemented +without breaking the object-root logging contract. It informs sidecar coercion, +error classification, and the later combinatorial test matrix. See +mpsc-log-design.md §§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. + +- [ ] 2.1.1. Implement positional argument parsing for the journal path and + raw field tail. + - Requires steps 1.1-1.2. + - See mpsc-log-design.md §§5, 10, 12. + - Success: the parser preserves field words and coercion flags in order and + rejects missing journal paths with `EX_USAGE`. +- [ ] 2.1.2. Implement field-word parsing for the selected `jo` object forms. + - Requires 2.1.1. + - See mpsc-log-design.md §§2, 5 and terms-of-reference.md §§2, 6. + - [ ] Cover `key=value`, `key@value`, object paths, array appends, and + bracketed object insertion. + - [ ] Reject unsupported `jo` options and any non-object root outcome. + - Success: representative selected `jo`-inspired examples produce typed + intermediate values or stable usage errors. +- [ ] 2.1.3. Implement explicit and inferred type coercion for CLI values. + - Requires 2.1.2. + - See mpsc-log-design.md §§5-6 and context.md. + - Success: explicit `-s`, `-n`, and `-b` flags override inference, empty + assignments become `null`, valid JSON values parse as JSON, and other + values remain strings. +- [ ] 2.1.4. Implement file-value forms with clear data and I/O errors. + - Requires 2.1.2 and 2.1.3. + - See mpsc-log-design.md §§5, 10. + - Success: `key:=path`, `key=@path`, `key=:path`, and `key=%path` produce + the documented values or `EX_DATAERR`/`EX_IOERR` diagnostics. + +### 2.2. Deliver sidecar-backed record construction + +This step answers whether configuration, schema coercion, defaults, and the +generated timestamp can merge deterministically. Its outcome informs the +external JSONL contract and df12-build event examples. See mpsc-log-design.md +§§2, 6, 9, 11-12, mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. + +- [ ] 2.2.1. Implement sidecar path derivation by extension replacement, TOML + v1.0 parsing, and semantic validation for the `[rotation]`, `[locking]`, + `[defaults]`, and `[schema]` tables. + - Requires 2.1.1. + - See mpsc-log-design.md §§2, 6, 10, adr-001-lock-file-naming.md, and + mpsc-log-sidecar.example.toml. + - Success: derivation replaces the journal filename extension with `.toml`, + appending `.toml` when the journal has no extension, so `run`, `run.jsonl`, + and `run.ndjson` each derive `run.toml`; missing sidecars use defaults; + malformed TOML returns `EX_DATAERR`; and semantically invalid configuration + returns `EX_CONFIG`. + - Success: a `.toml` journal filename whose derived sidecar path equals the + journal path is rejected with `EX_USAGE`. Tests cover the shared-stem + derivations and reject the self-sidecar `.toml` journal case. + - Success: `[rotation]` generation counts are validated as non-negative and + as retaining at least one generation, so a negative count and + `plain_generations = 0` with `compressed_generations = 0` each return + `EX_CONFIG`. +- [ ] 2.2.2. Implement deterministic record merging and schema-guided + coercion. + - Requires 2.1.3 and 2.2.1. + - See mpsc-log-design.md §§6, 9, 11 and + mpsc-log-event-schema.json. + - Success: sidecar defaults seed the record; CLI fields win over defaults + and earlier duplicate paths; and CLI coercion uses explicit `-s`, `-n`, or + `-b` flags before schema entries and default `jo`-inspired inference. +- [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. + - Requires 1.2.3 and 2.2.2. + - See mpsc-log-design.md §§2, 6 and terms-of-reference.md §§2, 6, 8. + - Success: defaults- and CLI-supplied `timestamp` values are validated + against the documented RFC 3339 UTC contract before append; valid overrides + are preserved, invalid overrides fail with `EX_DATAERR`, and an + invocation-time RFC 3339 UTC `timestamp` is generated only when no + `timestamp` override exists. + +### 2.3. Append the first useful JSONL record + +This step answers whether the command can complete a non-concurrent append +workflow with stable user-visible behaviour. It informs the later lock and +repair protocol because this slice defines the line format and diagnostics. See +mpsc-log-design.md §§3, 7, 10-12 and terms-of-reference.md §§5-7. + +- [ ] 2.3.1. Implement compact JSON object serialization and newline append. + - Requires steps 2.1-2.2. + - See mpsc-log-design.md §§3, 7, 11-12. + - Success: each successful invocation appends exactly one compact JSON + object followed by `\n`. +- [ ] 2.3.2. Map semantic errors to process exit codes and diagnostics. + - Requires steps 2.1-2.2. + - See mpsc-log-design.md §10 and terms-of-reference.md §§6-7. + - Success: success writes nothing to stdout, failures write one stderr + diagnostic, and invalid input, data errors, I/O errors, and configuration + errors use the documented statuses. +- [ ] 2.3.3. Document the day-one CLI usage in the users' guide. + - Requires 2.3.1 and 2.3.2. + - See mpsc-log-design.md §§5-6, 9-10 and + terms-of-reference.md §§4-7. + - Success: a workflow author can replace a simple `jo >> file.jsonl` + example with an equivalent `mpsc-log` invocation and understand failures. + +## 3. Trustworthy shared logging under contention + +Idea: if the same CLI can create missing directories, serialize concurrent +writers with bounded lock waiting, and repair partial tails, multi-agent +workflows can treat journal writes as boring infrastructure rather than a +source of telemetry loss. + +This phase turns the single-writer command into the product promised by the +terms of reference. It focuses on the one risk that motivates the tool: several +independent agents calling the same executable at the same time. + +### 3.1. Prove first-write and lock acquisition are safe + +This step answers whether concurrent invocations can create the same journal +and coordination artefacts without truncating or colliding. It informs the +critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, 7, +10-11 and terms-of-reference.md §§6-8. + +- [ ] 3.1.1. Create missing parent directories and coordination artefacts + before locking. + - Requires phase 2. + - See mpsc-log-design.md §§3-4, 7, 10 and + terms-of-reference.md §§6-8. + - Success: simultaneous first writes to a missing directory tree either + create one usable journal or fail with a specific creation diagnostic. +- [ ] 3.1.2. Implement exclusive journal locking with a configurable timeout. + - Requires 3.1.1. + - See mpsc-log-design.md §§2, 4, 6-7, 10 and context.md. + - Success: a pure function derives the lock path by appending `.lock` to the + complete journal filename in the same directory, rejects caller-supplied + `.lock` journal filenames, and gives `run` and `run.jsonl` distinct locks. + Tests cover those documented names; contending processes on a supported + local filesystem serialize through the same lock file; and lock timeout + failures return `EX_TEMPFAIL`. Network-filesystem support (NFS and CIFS) + remains deferred until the named filesystem verification matrix in step + 6.2.1. + - Success: the single lock acquisition attempt uses a five-second default + timeout, which an unlocked advisory pre-read of `[locking] timeout_ms` + may override; an absent, unreadable, or invalid pre-read falls back to + five seconds without failing the invocation; and no second acquisition + attempt is made. +- [ ] 3.1.3. Read sidecar configuration inside the journal critical section. + - Requires 3.1.2. + - See mpsc-log-design.md §§4, 6-7 and terms-of-reference.md §8.2. + - Success: each invocation uses one coherent sidecar view for repair, + rotation, compression, and append. + - Success: the advisory pre-read is used only to choose the lock timeout + and never supplies repair, rotation, coercion, or default values, which + come solely from the authoritative read taken under the lock. + +### 3.2. Preserve complete records across write and crash-like failures + +This step answers whether interrupted or failed writes can leave the journal in +a state the next invocation can recover. It informs the fault-injection adapter +and validates the design's append-plus-repair claim. See mpsc-log-design.md +§§7, 10-12 and terms-of-reference.md §§7-8. + +- [ ] 3.2.1. Implement append rollback when a write fails after extending the + active file. + - Requires 3.1.2. + - See mpsc-log-design.md §§7, 10-11. + - Success: injected write failures truncate the active file back to its + recorded pre-append length before returning an error. + - Success: rollback also covers flush and filesystem metadata failures + after bytes reach the file, all of which map to `EX_IOERR`; a successful + rollback leaves the record uncommitted; and a failed rollback truncate + still returns `EX_IOERR` while leaving the commit status unknown for the + next invocation to classify. +- [ ] 3.2.2. Implement partial-tail repair before every append. + - Requires 3.2.1. + - See mpsc-log-design.md §§6-7, 11. + - Success: unterminated final records are removed before a new valid record + is appended when repair is enabled, while newline-terminated invalid final + records fail closed with `EX_DATAERR` and leave the file unchanged. +- [ ] 3.2.3. Build the fault-injection filesystem coverage for write, truncate, + rename, compression, and metadata failures. + - Requires 1.2.3, 3.2.1, and 3.2.2. + - See mpsc-log-design.md §§7, 10-12. + - Success: each documented filesystem failure class has a deterministic + assertion for journal state and exit-code mapping, including the invalid + newline-terminated final-line case. + - [ ] Fault-inject a flush failure and a filesystem metadata failure after + bytes have reached the active file, plus a failed rollback truncate. + - Success: a successful rollback leaves no committed record; a failed + rollback truncate leaves either an unterminated tail that repair removes + or a complete record that repair preserves; and a retry after `EX_IOERR` + may duplicate a preserved record. The tests assert that duplication as + the documented at-least-once outcome rather than treating it as a + defect, because `mpsc-log` keeps no unknown-commit reconciliation state. + +### 3.3. Demonstrate concurrent append correctness end to end + +This step answers whether many real processes can use `mpsc-log` at once and +produce one complete record per successful command. It informs release +readiness for the first agent-workflow adoption. See mpsc-log-design.md §11 and +terms-of-reference.md §§5, 7. + +- [ ] 3.3.1. Build a multi-process concurrent append stress harness. + - Requires steps 3.1-3.2. + - See mpsc-log-design.md §11 and terms-of-reference.md §7.1. + - Success: a high-contention run produces the same number of complete, + decodable JSON object lines as successful child processes. +- [ ] 3.3.2. Add pairwise CLI/configuration combination coverage. + - Requires phase 2 and 3.3.1. + - See mpsc-log-design.md §11 and adr-002-testing-strategy.md. + - Success: the suite covers `jo` syntax form, coercion source, object path, + sidecar default, rotation schedule, rotation state, and lock contention. + - Success: parameterized duplicate-path tests cover duplicate writes at the + same object path across top-level keys, nested object paths, a sidecar + default overridden by a CLI field, and explicit `-s`, `-n`, and `-b` + coercion flags, asserting last-wins: the later write replaces the earlier + value at that object path. + +## 4. Rotation and retention without record loss + +Idea: if rotation, compression, and retention run under the same lock as +append, operators can keep journal files bounded without handing correctness +back to cron jobs or external shell glue. + +This phase delivers bounded storage while preserving the phase 3 concurrency +claim. It treats size-only rotation and scheduled rotation as user-facing modes +that need separate naming and retention evidence. + +### 4.1. Deliver size-only rotation for the default policy + +This step answers whether the default `schedule = "none"` policy can rotate +under contention without losing records. It informs the gzip and scheduled +rotation work because both build on the same locked action planner. See +mpsc-log-design.md §§6, 8, 11 and mpsc-log-sidecar.example.toml. + +- [ ] 4.1.1. Implement size-threshold detection and numeric generation + planning. + - Requires phase 3 and 1.1.4. + - See mpsc-log-design.md §§6, 8, 11. + - Success: `run.jsonl` rotates to `run.1.jsonl` after the configured + threshold while preserving the pending record. +- [ ] 4.1.2. Implement oldest-to-newest size-only rename execution. + - Requires 4.1.1. + - See mpsc-log-design.md §8. + - Success: generations advance without overwriting retained files and + rotation failures leave the previous readable state intact. +- [ ] 4.1.3. Document size-only rotation and retention in the users' guide. + - Requires 4.1.2. + - See mpsc-log-design.md §§6, 8 and terms-of-reference.md §§6-7. + - Success: users can predict active, plain rotated, and compressed filenames + from a journal path and sidecar settings. + +### 4.2. Compress and retain rotated logs atomically + +This step answers whether older generations can be gzipped without introducing +the data-loss window the design rejects. It informs scheduled-mode retention +because both modes rely on atomic gzip output and deletion ordering. See +mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. + +- [ ] 4.2.1. Benchmark and select the gzip compression backend. + - Requires 4.1.2. + - See mpsc-log-design.md §§2, 8. + - [ ] Benchmark `flate2`, `gzp`, and `gzippy` on representative rotated + journal files before selecting the compression backend. + - Success: the chosen backend has measured throughput, atomic-output + integration, dependency, and portability rationale documented in the design + or rotation ADR. +- [ ] 4.2.2. Implement atomic gzip output for rotated generations. + - Requires 4.2.1. + - See mpsc-log-design.md §§2, 8. + - Success: failed compression leaves the source generation in place and + aborts before appending the pending record. +- [ ] 4.2.3. Implement retention deletion for plain and compressed + generations. + - Requires 4.2.2. + - See mpsc-log-design.md §§6, 8. + - Success: files beyond `plain_generations + compressed_generations` are + deleted only after newer retained files are safely in place. + - Success: the evicted oldest generation and every superseded source + generation are staged aside rather than unlinked, and are removed only at + the commit point after the append. + - [ ] Fault-inject a failure at each rotation commit point: a partial rename + sequence, an uncommitted and a committed gzip output, staged source + removal, a failed append, and a process killed mid-rotation. + - Success: after every injected fault the journal settles into either the + pre-rotation or the post-rotation layout and never a mixture, no + record-bearing generation is lost, and the next invocation under the + journal lock completes or reverses the recorded plan and clears the + manifest before handling its own record. + +### 4.3. Deliver UTC scheduled rotation with size splits + +This step answers whether hourly, daily, and weekly modes rotate at period +boundaries while still splitting busy periods by bytes. It informs the final +operator contract because scheduled rotation is where naming expectations are +most visible. See mpsc-log-design.md §§6, 8, 11, mpsc-log-sidecar.example.toml, +and context.md. + +- [ ] 4.3.1. Implement period calculation for `hourly`, `daily`, and + `weekly` schedules. + - Requires 1.2.3 and 4.1.1. + - See mpsc-log-design.md §§6, 8 and context.md. + - Success: UTC boundaries produce the documented hour, day, and ISO-week + beginning names without using a `max_age` configuration field. +- [ ] 4.3.2. Implement active-period detection and scheduled boundary + rotation. + - Requires 4.3.1. + - See mpsc-log-design.md §8. + - Success: the next invocation after a time break archives the previous + active segment under the period that produced its records. + - Success: a unit or behavioural test shows period-boundary rollover runs + before the `max_bytes` size check, so the size split is evaluated against + the fresh active file for the pending period. +- [ ] 4.3.3. Implement interim size-split suffixes inside scheduled periods. + - Requires 4.3.2. + - See mpsc-log-design.md §8. + - Success: busy periods produce ordered `.n` suffixes, and final scheduled + archives use the next suffix when a period already has size splits. + - Success: a unit or behavioural test covers collision-free suffix selection + when archives for the period already exist, so rotation never overwrites an + existing archive. + - Success: a unit or behavioural test covers appending an oversized record + directly to an empty active journal without creating an empty archive. +- [ ] 4.3.4. Implement scheduled-mode period retention and compression. + - Requires 4.2.3 and 4.3.3. + - See mpsc-log-design.md §8. + - Success: the newest completed periods remain plain as complete groups, + older retained periods are gzipped, and expired periods are deleted. + +### 4.4. Demonstrate rotation correctness under contention + +This step answers whether size-only and scheduled rotation preserve every +successful record when many processes append concurrently. It informs release +readiness because it exercises the highest-risk feature interactions. See +mpsc-log-design.md §11 and terms-of-reference.md §7. + +- [ ] 4.4.1. Build the concurrent rotation end-to-end suite. + - Requires steps 4.1-4.3. + - See mpsc-log-design.md §11 and adr-002-testing-strategy.md. + - Success: forced size and scheduled rotations under concurrent writers + preserve the decoded record count across active, plain, scheduled, and + compressed files. +- [ ] 4.4.2. Add fixture coverage for failed rotation and compression actions. + - Requires 4.4.1. + - See mpsc-log-design.md §§8, 10-11. + - Success: injected rename, delete, gzip, and commit failures leave a + readable previous state and emit the documented exit code. + +## 5. df12-build telemetry adoption + +Idea: if the finished CLI can emit the first df12-build journal taxonomy with +documented sidecar defaults and realistic invocation fixtures, the workflow can +start collecting real-run evidence without `mpsc-log` becoming an analysis or +dashboard product. + +This phase connects the generic journalling tool to its first concrete use +case. It packages examples and verification around phase timing, review rounds, +task shape, CodeRabbit waits, audit yield, remediation lanes, and escaped +defects while respecting the non-goals around discovery and analytics. + +### 5.1. Make the df12-build event contract executable + +This step answers whether the schema, sidecar defaults, and example event names +are specific enough for the ODW workflow to call. It informs workflow adoption +and future telemetry analysis outside this crate. See mpsc-log-design.md §9, +mpsc-log-event-schema.json, mpsc-log-sidecar.example.toml, and +terms-of-reference.md §§2, 5-7. + +- [ ] 5.1.1. Add fixture invocations for the initial df12-build event names. + - Requires phase 4. + - See mpsc-log-design.md §9 and terms-of-reference.md §§2, 5-7. + - Success: fixtures cover `phase.started`, `phase.finished`, + `review.round`, `task.finished`, `coderabbit.attempt`, `audit.finding`, + and `defect.escape`, including CodeRabbit wait/throttle variants, audit + yield by severity and lane, remediation-lane outcomes, and post-merge + defect-escape cases. +- [ ] 5.1.2. Validate fixture output against the JSON Schema contract. + - Requires 5.1.1. + - See mpsc-log-design.md §9 and mpsc-log-event-schema.json. + - Success: generated fixture records satisfy required fields, reserved + namespaces, integer ranges, boolean coercions, and RFC 3339 timestamps. +- [ ] 5.1.3. Provide a df12-build sidecar configuration example. + - Requires 5.1.2. + - See mpsc-log-design.md §§6, 9 and + mpsc-log-sidecar.example.toml. + - Success: the example sidecar coerces the telemetry fields needed for the + first workflow integration without imposing organization-wide schema + governance. + +### 5.2. Document operator-facing adoption without expanding scope + +This step answers whether workflow authors and operators can use the tool +correctly without reading implementation code. It informs v1 release readiness +and keeps deferred analysis work out of the core CLI. See terms-of-reference.md +§§3-7 and mpsc-log-design.md §§3, 9-10. + +- [ ] 5.2.1. Update the users' guide with complete operational examples. + - Requires 5.1.3. + - See mpsc-log-design.md §§5-10 and terms-of-reference.md §§4-7. + - Success: the guide covers simple append, sidecar defaults, coercion + errors, lock timeout, size rotation, scheduled rotation, and df12-build + event examples. +- [ ] 5.2.2. Add troubleshooting guidance for agent callers. + - Requires 5.2.1. + - See mpsc-log-design.md §§7, 10-11 and terms-of-reference.md §§7-8. + - Success: operators can distinguish invalid arguments, malformed sidecars, + timeout, directory creation failure, partial-tail repair, + newline-terminated final-line corruption, and rotation failure from command + output and exit status. +- [ ] 5.2.3. Add a release smoke script for the documented workflows. + - Requires 5.2.1 and 5.2.2. + - See mpsc-log-design.md §§5-11. + - Success: the smoke script exercises documented commands against temporary + journals and fails if examples drift from the binary. + +## 6. Deferred extensions after the core v1 promise + +Idea: if the core v1 promise is already trustworthy and boring to operate, the +project can evaluate broader extensions on their product value instead of +letting them destabilize the main release. + +This phase captures work the current ToR and design explicitly defer. These +items should not block v1 unless a later ADR changes the product boundary. + +### 6.1. Evaluate broader observability and analysis features + +This step keeps dashboards, queries, and automatic tuning out of the core CLI +while preserving them as possible downstream products. See +terms-of-reference.md §§6.2, 7.3 and mpsc-log-design.md §3.2. + +- [ ] 6.1.1. Decide whether query, dashboard, or recommendation tooling belongs + in a separate crate or repository. + - Requires phase 5. + - See terms-of-reference.md §§6.2, 7.3 and mpsc-log-design.md §3.2. + - Success: the decision does not add query or dashboard obligations to the + append-only CLI. +- [ ] 6.1.2. Decide whether df12-build analysis should consume journal output + as a downstream tool. + - Requires 6.1.1. + - See terms-of-reference.md §§2, 6.2, 7.3 and mpsc-log-design.md §9. + - Success: any tuning or reporting work has a separate owner and does not + change the v1 write contract. + +### 6.2. Evaluate expanded platform and compatibility promises + +This step keeps distributed coordination, full `jo` parity, public library +APIs, and `max_age` out of v1 while identifying the decisions required if they +become valuable later. See terms-of-reference.md §§6.2, 8-9 and +mpsc-log-design.md §§3.2, 11-13. + +- [ ] 6.2.1. Decide whether network-filesystem support should graduate from + caveat to tested platform promise. + - Requires phase 5. + - See mpsc-log-design.md §§7, 11 and terms-of-reference.md §§6.2, 8.2. + - Success: the project either publishes a named filesystem verification + matrix or leaves the unsupported-network-filesystem caveat intact. +- [ ] 6.2.2. Decide whether full `jo` compatibility has enough value to expand + the object-root subset. + - Requires phase 5. + - See mpsc-log-design.md §§2, 5, 13 and terms-of-reference.md §§6.2, 9. + - Success: unsupported `jo` behaviours stay rejected unless an ADR explains + the compatibility gain and object-root implications. +- [ ] 6.2.3. Decide whether to expose a supported Rust library API. + - Requires phase 5. + - See mpsc-log-design.md §§3.2, 12-13 and terms-of-reference.md §9. + - Success: public API compatibility remains out of v1 unless an ADR defines + supported types, versioning, and documentation obligations. +- [ ] 6.2.4. Decide whether retention by age should be added after v1. + - Requires phase 5. + - See mpsc-log-design.md §§6, 8 and terms-of-reference.md §8.1. + - Success: `max_age` remains absent unless a later design defines its + interaction with size, schedule, compression, and concurrent writers. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md new file mode 100644 index 0000000..ae8622d --- /dev/null +++ b/docs/terms-of-reference.md @@ -0,0 +1,378 @@ +# mpsc-log – terms of reference + +- **Status:** Draft v0.1. +- **Audience:** Product owner, engineering maintainers, agent-workflow + authors, and future design reviewers. +- **Last substantive revision:** 2026-06-29. +- **Companion documents:** + - [Documentation contents](contents.md). + - [Users' guide](users-guide.md). + - [Developer guide](developers-guide.md). + - [Repository layout](repository-layout.md). + - [Documentation style guide](documentation-style-guide.md). + - [Context](context.md). + - [Technical design](mpsc-log-design.md). + - [Event schema](mpsc-log-event-schema.json). + - [Sidecar example](mpsc-log-sidecar.example.toml). + +## 1. Background and motivation + +`mpsc-log` exists for multi-agent and scripted workflows that need several +independent processes to record structured events in one append-only journal. +The immediate problem is not creating JSON. Tools such as `jo` already turn +shell arguments into JSON objects.[^1] The gap is that agents also need a small +command-line interface (CLI) that appends each record as a JSON Lines (JSONL) +entry to a shared journal without overwriting existing data, losing concurrent +writes, or corrupting the journal during rotation. + +The motivating context is local automation where several agents may run at the +same time and cannot coordinate through a long-lived service. Each agent should +be able to invoke the tool once, pass a target path and record fields, and then +continue. The tool must make the file-system coordination boring: safe file +creation, file locking, write atomicity, timeout handling, and rotation belong +inside the tool rather than being reimplemented by each caller. + +The first concrete use case is a multi-agent Open Dynamic Workflows (ODW) +workflow for advancing `docs/roadmap.md` through planning, design review, +implementation, code review, expert review, integration, audit, and remediation +phases. Agents in that workflow need to append journal events to a file in the +workflow sidecar directory. The journal should capture enough real-run +telemetry to replace architectural guesses with measured behaviour. + +This project is being defined before its design document. The existing +repository is a generated Rust application scaffold, so these terms of +reference treat the user brief and `Cargo.toml` package description as the +authoritative product inputs. + +## 2. Domain + +The product sits in structured local logging for automated command runners. Its +records are JSON objects serialized as one JSON value per line. JSONL keeps the +log easy to append, stream, grep, split, and ingest into later tools without +requiring the whole file to be rewritten. + +The command accepts the journal path as its first argument. Later arguments +describe fields using selected `jo`-inspired key and value words, including +type coercion flags and object paths. The CLI is not textually compatible with +all `jo` output: the root value must always be an object, and duplicate writes +to the same object path use last-wins semantics. + +The default entry includes a `timestamp` field containing a Coordinated +Universal Time (UTC) timestamp captured when the command is invoked. The +timestamp uses RFC 3339, the Internet timestamp profile of ISO 8601.[^2] + +The tool also reads a sidecar TOML file whose path is derived from the journal +path: it replaces the journal's final extension with `.toml`, or appends +`.toml` when the journal has no extension. It defines rotation configuration, +type-coercion schema, and default field values. + +The term "sidecar" has two relevant meanings in the current domain. The ODW +workflow has a workflow sidecar directory where run artefacts belong. +`mpsc-log` also has a sidecar TOML configuration file next to the selected +journal file. The design must keep those concepts distinct: callers choose the +journal path, and `mpsc-log` derives only its configuration path from that +journal path. + +The first journaled telemetry set is operational evidence for the df12-build +workflow: + +- phase timings and design-review or code-review round counts; +- task outcomes grouped by roadmap shape, including task size, work-item count, + changed-file count, dependency depth, phase kind, review failures, + implementation failures, merge conflicts, and audit follow-ups; +- CodeRabbit wait times, HTTP 429 frequency, deferred reviews, and retry + success; +- audit yield by severity and remediation lane, including dropped findings, + addenda, rerouted step tasks, and new roadmap steps; +- post-merge defect escape, including defects later found by dogfooding, + continuous integration, users, or subsequent roadmap work. + +## 3. Market context + +The current alternatives each solve part of the problem: + +- `jo` builds JSON from shell arguments but writes to standard output and does + not provide locked append, timeout, file creation, or rotation semantics. +- Shell redirection with `>>` is convenient but leaves callers responsible for + record construction, validation, rotation, and coordination under contention. +- `flock` can serialize shell commands, but each caller must compose locking, + JSON construction, append mode, error handling, and rotation correctly. +- General logging frameworks target applications that own their logging stack; + they are a poor fit for short-lived independent agents that need a single + external command. +- System loggers and centralized observability products handle broader + collection and retention concerns, but they introduce services, deployment, + and integration work that is disproportionate for a local agent log. +- `logrotate` can rotate files, but it is scheduled external maintenance rather + than a per-write safety mechanism integrated with concurrent appends. + +The gap is a narrowly scoped CLI that combines selected `jo`-inspired record +construction with safe append and rotation behaviour for one local journal. + +## 4. Users and stakeholders + +| Type | Context | Cares about | Will dislike or ignore | Current alternative | +| ------------------------------------------ | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| Primary user: agent workflow author | Builds scripts, prompts, wrappers, or orchestration around multiple CLI-capable agents | One command that reliably records structured events under concurrency | Running a daemon, writing bespoke lock scripts, or debugging corrupted logs | `jo` plus shell redirection, ad hoc scripts, or no structured log | +| Primary user: autonomous agent | Invokes tools from a constrained shell environment during task execution | Simple argument contract, bounded waiting, machine-readable failure | Interactive prompts, hidden global state, or non-deterministic output formats | Direct file append or caller-specific helper | +| Primary user: df12-build workflow operator | Runs ODW workflows that plan, review, implement, merge, audit, and remediate roadmap work | Evidence for tuning parallelism, review caps, audit loops, and remediation lanes | Manual reconstruction from transcripts, branch history, or final summaries | Architectural guesses and scattered workflow output | +| Secondary user: maintainer | Implements, reviews, and releases the Rust CLI | Clear correctness requirements, testable failure modes, and stable docs | Broad observability scope or vague compatibility promises | Generated project scaffold and maintainer convention | +| Stakeholder: project sponsor | Wants agents to leave durable audit trails without coordination overhead | Fewer lost events, easier review of concurrent runs, low setup cost | Feature creep into a logging platform | Manual run notes or scattered per-agent logs | +| Non-user: observability platform operator | Runs centralized ingestion, querying, alerting, and retention infrastructure | Fleet-level logging, dashboards, and policy controls | A local-file-only CLI | Existing observability stack | + +Table 1: Stakeholder mapping for the initial product boundary. + +## 5. Job to be done + +When an agent workflow launches several independent command-line processes, the +workflow author wants each process to append a structured event to the same +local journal, so they can inspect the run later without reconstructing events +from scattered output. + +When a df12-build ODW run processes roadmap tasks through planning, review, +implementation, integration, and audit, the workflow operator wants agents to +append structured journal events into the workflow sidecar directory, so they +can tune workflow defaults from real-run telemetry rather than from design-time +intuition. + +The functional dimension is durable structured append under contention. The +emotional dimension is confidence that a missing or malformed log entry points +to a real failure rather than a race in the logging helper. The social +dimension is that maintainers can review an agent run from one readable +artefact rather than trusting a caller's informal summary. + +## 6. Scope + +### 6.1 Goals + +- Accept a journal path as the first CLI argument and interpret following + arguments as record fields. +- Support the selected `jo`-inspired key/value syntax needed for object + records, including type coercion flags and object paths. +- Reject any invocation that would produce a non-object root. +- Add a default `timestamp` field using an invocation-time UTC timestamp unless + configuration or CLI input explicitly overrides it according to the final + precedence rules. +- Append exactly one valid JSON object followed by a newline for each successful + invocation. +- Prevent concurrent calls from overwriting existing logs or colliding with + each other while creating, appending to, or rotating the file. +- Create missing parent directories for the journal path automatically when + filesystem permissions allow it. +- Use locking and safe writes with a five-second default timeout. +- Gracefully handle simultaneous attempts to create the journal file and + sidecar coordination artefacts. +- Rotate by default after the active log reaches 1 MiB. +- Optionally rotate on hourly, daily, or weekly UTC time boundaries, with + interim size splits if the active log reaches the size threshold before the + next boundary. +- Define local generation retention on the writer's filesystem, as follows. +- For size-only rotation, retain the newest rotated generations (four by + default, via `plain_generations`) as plain files, and gzip older retained + generations (per `compressed_generations`). +- For scheduled rotation, retain every size-split file in the newest + completed periods (four by default, via `plain_generations`) as plain files, + gzip only older retained periods (per `compressed_generations`), and never + compress files in the current period. +- Read a sidecar TOML file for rotation configuration, schema-guided type + coercion, and default field values. +- Surface failures through stable exit codes and diagnostics suitable for + non-interactive agents. +- Support journal records for the df12-build ODW workflow's phase timings, + review rounds, task-shape outcomes, CodeRabbit waits and 429s, audit yield, + remediation lane outcomes, and post-merge defect escape. + +### 6.2 Non-goals + +- Centralized log ingestion, search, dashboards, alerting, and centralized or + downstream retention policy are out of scope; users needing those should ship + the JSONL output into an observability system. Local rotation and compression + retention remain in scope, per the goals above. +- A long-running daemon or background service is out of scope; the product is a + short-lived CLI. +- Preserving chronological file order by timestamp is out of scope. The default + timestamp is captured at invocation time, so entries can appear out of order + when one caller waits behind another. +- Full `jo` feature parity and textual duplicate-key preservation are out of + scope where they conflict with the object-root logging contract. +- Cross-host distributed locking is out of scope unless a later design + explicitly accepts the complexity and platform constraints. +- Querying, filtering, formatting, or editing historical log entries is out of + scope; readers can use existing JSONL and shell tooling. +- Log schema governance for an organization is out of scope. The sidecar schema + exists to coerce one tool's input, not to become a central event taxonomy. +- Computing dashboards, recommendations, or automatic tuning for df12-build is + out of scope. `mpsc-log` records the evidence; analysis can happen in later + tooling. +- Discovering the ODW workflow sidecar directory is out of scope unless a later + integration contract adds it. Callers pass the concrete journal path. + +## 7. Success criteria + +### 7.1 User-facing success + +- A workflow author can replace an ad hoc `jo >> file.jsonl` call with + `mpsc-log` without losing the ability to express nested object fields and + basic type coercions. +- Stress tests with many concurrent invocations produce the same number of + valid JSONL records as successful command exits. +- Simultaneous first writes to a missing journal create exactly one usable log + and do not truncate, replace, or interleave records. +- A first write to a journal path in a missing directory tree creates the + required parent directories when permissions allow it. +- Rotation during concurrent writes leaves every successful entry in either the + active file or a rotated file. +- A df12-build run can record one journal entry per meaningful phase, review + round, implementation attempt, CodeRabbit attempt, integration result, audit + finding, remediation triage decision, and post-merge defect report. + +### 7.2 Operational success + +- Lock acquisition obeys the configured timeout and defaults to five seconds. +- Rotation defaults are predictable: 1 MiB active-file threshold and compression + after four rotations. +- A failed invocation leaves the previous log state readable and does not + produce partial JSON records. +- Diagnostics are useful to agents and humans: invalid arguments, timeout, + malformed sidecar configuration, final-line corruption, and file-system + failures are distinguishable. +- Journal writes add low enough overhead that agents can record telemetry at + phase and attempt boundaries without changing workflow scheduling decisions. + +### 7.3 Strategic success + +- The tool remains small enough for agents to call as a normal utility rather + than as an integration project. +- The terms of reference, design document, and user guide give future + contributors enough boundary information to reject logging-platform feature + requests cleanly. +- df12-build maintainers can compare observed phase durations, review-round + counts, CodeRabbit waits, audit yield, and defect escape against + `MAX_PARALLEL`, `MAX_DESIGN_ROUNDS`, and `MAX_REVIEW_ROUNDS`. + +## 8. Constraints and assumptions + +### 8.1 Hard constraints + +- The first CLI parameter is the journal path. +- Later parameters are key/value words using the accepted `jo`-inspired + syntax. +- The root record must be a JSON object. +- The default lock timeout is five seconds. +- The default rotation threshold is 1 MiB. +- The default scheduled rotation policy is `none`. +- Scheduled rotation modes are `hourly`, `daily`, and `weekly`; they use UTC + period boundaries and do not imply a `max_age` retention setting. +- Rotated logs are compressed after the configured plain generation count, + four by default. +- The sidecar configuration file is TOML and derives its path from the + journal path by replacing the filename extension with `.toml`. +- The default record includes `timestamp` unless the final precedence rules say + otherwise. +- Missing parent directories for the journal path are created automatically + when permissions allow it. +- The tool must tolerate concurrent writers and concurrent initial file + creation. +- The workflow sidecar directory is caller-owned. `mpsc-log` receives the + journal file path explicitly and must not assume a df12-build directory + layout. + +### 8.2 Assumptions + +- The initial target is a local filesystem with locking semantics that can + protect cooperating `mpsc-log` processes. If users rely on network + filesystems with weaker locking, the tool may need documented limitations or + a different coordination strategy. +- Callers can pass shell arguments without needing standard input as the primary + data channel. If ARG_MAX or large payloads become common, file-value support + and streaming input need sharper requirements. +- Agent callers can handle non-zero exit codes and diagnostic text. If a caller + cannot observe failures, reliable logging cannot be guaranteed from the CLI + alone. +- Directory creation can fail because of permissions, read-only filesystems, or + invalid paths. Those failures must leave any existing log readable and return + diagnostics that distinguish directory creation from record construction. +- Sidecar defaults are stable enough to read during each invocation. If callers + edit sidecar configuration while writes are running, the design must specify + whether configuration reads are locked with log writes. +- RFC 3339 UTC timestamps are acceptable for the default timestamp field. If a + later integration requires a different timestamp profile, that format becomes + an explicit compatibility requirement. +- The df12-build ODW workflow can call `mpsc-log` at phase and attempt + boundaries. If agents cannot call the CLI from those points, the first + integration will need a wrapper or workflow-level helper. +- A single workflow run can identify its events with stable run, task, phase, + agent, and attempt identifiers. If those identifiers are not available, later + analysis will be limited to aggregate timing and count data. + +### 8.3 Dependencies + +- The `jo` manual is prior art for selected argument syntax, type coercion + flags, and object-path expectations; `mpsc-log` compatibility is limited by + the object-root and last-wins decisions.[^1] +- RFC 3339 is the timestamp reference for default `timestamp` values.[^2] +- The future technical design must choose Rust crates or platform APIs for JSON + serialization, TOML parsing, file locking, gzip compression, time handling, + and atomic file operations. + +## 9. Open questions + +| Question | Why it matters | Criteria for resolution | Suggested path | +| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- | +| What does "gzipping after 4 rotations" mean exactly? | Retention, naming, and compression timing affect concurrency and user expectations. | The rotation policy specifies filenames, retention count, compression trigger, and whether recent rotations remain plain text. | Technical design. | +| What platforms are supported at v1? | File-locking and atomic rename semantics differ across Unix, Windows, and network filesystems. | The project declares supported platforms and test coverage. | ADR candidate. | +| What are the stable exit codes? | Agents need to distinguish retryable timeout from invalid input or corrupted configuration. | The user guide lists exit codes and diagnostics. | User-guide update during implementation. | +| Is a Rust library API part of the product? | A public library API expands compatibility and documentation obligations. | The roadmap states whether the crate is CLI-only or also exposes supported library functions. | Product decision. | +| What is the df12-build event taxonomy? | The first use case needs comparable records for phase timings, task shape, review outcomes, CodeRabbit attempts, audit findings, remediation lanes, and defect escape. | The design defines stable event names, required fields, optional fields, and schema versions. | Technical design with workflow fixture. | +| How are workflow events correlated? | Without run, task, phase, agent, attempt, branch, and commit identifiers, telemetry cannot explain review loops, merge conflicts, or escaped defects. | The workflow integration names required correlation fields and their source. | Integration design. | + +## 10. Handoff + +### 10.1 Context additions + +The [context](context.md) document should continue to define the shared +vocabulary for: + +- Agent workflow. +- Audit yield. +- CodeRabbit attempt. +- Defect escape. +- df12-build. +- Open Dynamic Workflows (ODW). +- JSON Lines (JSONL). +- Journal. +- Record. +- Remediation lane. +- Review round. +- Sidecar configuration. +- Workflow sidecar directory. +- Rotation. +- Active log. +- Rotated log. +- Type coercion. +- Object path. +- Lock timeout. +- Scheduled break. + +### 10.2 ADR candidates + +- Locking and atomic-write strategy across supported platforms. +- Rotation naming, retention, and compression policy. +- CLI-only product boundary versus supported library API. + +### 10.3 Downstream readiness + +This document is complete enough to start a technical design. The design should +not begin implementation until the timestamp standard, selected `jo` field +syntax, sidecar precedence, rotation policy, and platform support questions are +resolved or explicitly deferred. + +## Appendix A. References + +[^1]: Jan-Piet Mens, + [`jo` manual](https://github.com/jpmens/jo/blob/master/jo.md), + accessed 2026-06-29. + +[^2]: G. Klyne and C. Newman, [RFC 3339: Date and time on the Internet: + timestamps](https://www.rfc-editor.org/rfc/rfc3339), July 2002, accessed + 2026-06-29. diff --git a/typos.local.toml b/typos.local.toml index f491eb4..271fd52 100644 --- a/typos.local.toml +++ b/typos.local.toml @@ -7,7 +7,7 @@ stems = [] [words] # Official names: Azure Architecture Center and GitHub Flavored Markdown. -accepted = [] +accepted = ["flate2"] [words.corrections] @@ -15,6 +15,7 @@ accepted = [] ignore = [ "\\bAzure Architecture\\s+Center \\| Microsoft Learn\\b", "\\bGitHub Flavored\\s+Markdown\\b", + "\\bflate2\\b", # Inline code spans quote identifiers verbatim, so they are not en-GB # prose; the shared base previously supplied this exemption, so this # restores it locally. diff --git a/typos.toml b/typos.toml index 9868d35..89afddd 100644 --- a/typos.toml +++ b/typos.toml @@ -36,6 +36,7 @@ extend-ignore-re = [ "Cranelift and mold", "\\bAzure Architecture\\s+Center \\| Microsoft Learn\\b", "\\bGitHub Flavored\\s+Markdown\\b", + "\\bflate2\\b", "\\brust-analyzer\\b", "`[^`\\n]+`", "`mold`", @@ -846,6 +847,7 @@ extend-ignore-re = [ "finalizers" = "finalizers" "finalizes" = "finalizes" "finalizing" = "finalizing" +"flate2" = "flate2" "formalisable" = "formalizable" "formalisably" = "formalizably" "formalisation" = "formalization"