From 6cdcd724b14107b1d9ab9d36db0ee37ef9ecff3a Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 17:13:23 +0100 Subject: [PATCH 01/27] Define mpsc-log design and roadmap Add the initial terms of reference, technical design, glossary, event schema, sidecar example, and delivery roadmap for `mpsc-log`. Together these documents establish the CLI contract, JSONL record shape, sidecar configuration, concurrency model, rotation policy, validation strategy, and df12-build telemetry adoption path. Update the documentation index so reviewers can find the new artefacts. Fix the generated doctest import to use the Rust crate identifier so the workspace test gate passes. --- docs/contents.md | 16 + docs/context.md | 31 ++ docs/mpsc-log-design.md | 431 +++++++++++++++++++++++++ docs/mpsc-log-event-schema.json | 83 +++++ docs/mpsc-log-sidecar.example.toml | 28 ++ docs/roadmap.md | 489 +++++++++++++++++++++++++++++ docs/terms-of-reference.md | 370 ++++++++++++++++++++++ 7 files changed, 1448 insertions(+) create mode 100644 docs/context.md create mode 100644 docs/mpsc-log-design.md create mode 100644 docs/mpsc-log-event-schema.json create mode 100644 docs/mpsc-log-sidecar.example.toml create mode 100644 docs/roadmap.md create mode 100644 docs/terms-of-reference.md diff --git a/docs/contents.md b/docs/contents.md index 3a9800b..83e4c25 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -5,6 +5,14 @@ set. ## Project guides +- [Context](context.md) defines the project vocabulary for journals, + sidecars, rotation, and df12-build telemetry. +- [Design](mpsc-log-design.md) specifies the CLI, file formats, locking, + rotation, failure handling, and verification strategy for `mpsc-log`. +- [Terms of reference](terms-of-reference.md) defines the product problem, + users, scope boundaries, constraints, and open questions for `mpsc-log`. +- [Roadmap](roadmap.md) translates the design and terms of reference into + review-sized implementation tasks. - [User guide](users-guide.md) explains how to use the generated project and its public build and test commands. - [Developer guide](developers-guide.md) explains the local workflow and @@ -33,3 +41,11 @@ set. - [Scripting standards](scripting-standards.md) explains the preferred Python scripting stack, command execution patterns, and test expectations for helper scripts. + + +## Design artefacts + +- [mpsc-log event schema](mpsc-log-event-schema.json) defines the initial JSON + Schema for journal records. +- [mpsc-log sidecar example](mpsc-log-sidecar.example.toml) shows the TOML + configuration shape used by the design. diff --git a/docs/context.md b/docs/context.md new file mode 100644 index 0000000..1a69a0a --- /dev/null +++ b/docs/context.md @@ -0,0 +1,31 @@ +# Context + +[Context](context.md) defines the working vocabulary for `mpsc-log`. + +## Terms + +| Term | Definition | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Active log | The JSON Lines file named on the command line. Successful invocations append new records to this file unless rotation runs first. | +| Agent workflow | A workflow that launches one or more autonomous agents or helper processes that can invoke command-line tools. | +| Audit yield | The distribution of audit findings by severity, source, and remediation lane. | +| CodeRabbit attempt | One attempt to run CodeRabbit review, including its start time, result, wait time, retry status, and whether the review was deferred. | +| Defect escape | A defect discovered after merge by later work, dogfooding, Continuous Integration (CI), or users. | +| Journal | A shared JSON Lines file used as an append-only record of workflow events. | +| Lock timeout | The maximum time an invocation waits for the journal lock before failing with a timeout diagnostic. | +| Object path | A `jo`-style field path that writes a value into a nested JSON object rather than into a literal top-level key. | +| Open Dynamic Workflows (ODW) | The workflow runtime used by `df12-build` to coordinate multi-agent roadmap execution. | +| Record | One JSON object serialized as one JSON Lines value and terminated with `\n`. | +| Remediation lane | The route assigned to a review or audit follow-up, such as addendum, step task, later roadmap step, or dropped. | +| Review round | One pass through a review loop, including any blocking findings and subsequent fix attempt. | +| Rotated log | A previous active log generation retained under a numbered or scheduled filename, optionally compressed with gzip. | +| Scheduled break | A configured UTC time boundary, either hourly, daily, or weekly, that forces the current active log segment to rotate on the next invocation. | +| Sidecar configuration | The TOML configuration file derived from the active log path by replacing the filename extension with `.toml`. | +| Type coercion | Conversion of a CLI value to a JSON string, number, boolean, null, object, or array according to explicit flags, sidecar schema, or default `jo`-compatible inference. | +| Workflow sidecar directory | The caller-owned directory where a workflow stores run artefacts, including the `mpsc-log` journal chosen by that workflow. | + +## Naming + +- Use "journal" for the shared JSON Lines log written by `mpsc-log`. +- Use "sidecar configuration" for the `.toml` file next to the journal. +- Use "workflow sidecar directory" for the df12-build ODW artefact directory. diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md new file mode 100644 index 0000000..eb27e63 --- /dev/null +++ b/docs/mpsc-log-design.md @@ -0,0 +1,431 @@ +# mpsc-log design + +- **Status:** Draft v0.1. +- **Audience:** Implementers, reviewers, and workflow operators. +- **Last substantive revision:** 2026-06-29. +- **Companion documents:** [Terms of reference](terms-of-reference.md), + [context](context.md), [event schema](mpsc-log-event-schema.json), + [sidecar example](mpsc-log-sidecar.example.toml), + [users' guide](users-guide.md), and [developer guide](developers-guide.md). + +## 1. Context + +`mpsc-log` is a short-lived Rust command-line interface (CLI) for appending one +JSON object to a shared JSON Lines journal. It exists because multi-agent +workflows need structured telemetry without running a logging daemon or +rewriting the same `jo`, `flock`, append, timeout, and rotation glue in every +caller. + +The first integration target is the df12-build Open Dynamic Workflows (ODW) +workflow at `../df12-build.worktrees/codex-annex/workflows/df12-build-odw.js`. +Agents in that workflow will write journal records into a caller-owned workflow +sidecar directory. The records measure phase timing, review rounds, task shape, +CodeRabbit waits and HTTP 429s, audit yield, remediation lane outcomes, and +post-merge defect escape. + +## 2. Research baseline + +JSON Lines is the journal format because it requires UTF-8, one valid JSON +value per line, and `\n` line termination; it is also described as suitable for +log files and cooperating processes.[^jsonl] `mpsc-log` tightens the JSON Lines +contract by requiring every root value to be an object. + +The CLI field grammar follows `jo` where that serves object-record logging. +`jo` supports `key=value`, `key@value`, type coercion flags, object paths, +file-value prefixes, array construction, and duplicate object keys.[^jo] +`mpsc-log` rejects array roots and serializes through `serde_json`, so +duplicate keys resolve by last write rather than producing repeated JSON object +names. + +The default `timestamp` field uses RFC 3339 UTC. RFC 3339 defines an Internet +profile of ISO 8601 for timestamps, recommends UTC for interoperability, and +uses `date-time = full-date "T" full-time`.[^rfc3339] + +The sidecar configuration uses TOML v1.0.0. TOML v1.0.0 is UTF-8, maps +unambiguously to a hash table, supports dotted keys, and forbids defining the +same key multiple times.[^toml] + +Prior art informs the concurrency and rotation choices. `flock(1)` exposes +exclusive locks, non-blocking mode, timeout mode, and explicit timeout exit +status, while warning that NFS and CIFS may have limited lock support.[^flock] +`logrotate(8)` rotates, compresses, removes, and handles logs by size; its +`copytruncate` documentation names a small data-loss window, which this design +avoids by rotating under the same lock used for appends.[^logrotate] + +The Rust implementation baseline is conservative: + +| Need | Choice | Reason | +| ----------- | ------------------------------ | ------------------------------------------------------------------------------------------------- | +| CLI parsing | `clap` builder API | Handles help/version/error rendering while allowing raw `jo`-style field tails.[^clap] | +| JSON | `serde`, `serde_json` | Standard Rust serialization stack; object maps naturally enforce last-wins duplicate handling. | +| TOML | `toml` crate, TOML v1.0 subset | Current crate supports TOML parsing; the product contract remains v1.0. | +| Time | `jiff` | Provides `Timestamp::now()` and RFC 3339-style instant formatting with nanosecond support.[^jiff] | +| Locking | `fs4` sync feature | Provides cross-platform file locks without requiring a daemon.[^fs4] | +| Gzip | `flate2` default backend | Supports gzip streams using a safe Rust default backend.[^flate2] | +| Error types | `thiserror` | Exposes semantic errors to the CLI boundary. | + +Table 1: Dependency baseline for implementation. + +## 3. Goals and non-goals + +### 3.1 Goals + +- Create missing parent directories for the journal path. +- Parse the first argument as the journal path and remaining arguments as + `jo`-style field words and coercion flags. +- Merge sidecar defaults, CLI fields, schema-guided coercions, and generated + timestamp values into one JSON object. +- Serialize exactly one compact JSON object plus `\n` per successful + invocation. +- Use one lock per journal to serialize repair, rotation, compression, and + append. +- Time out lock acquisition after five seconds by default. +- Rotate after 1 MiB by default, keep four plain generations, and gzip older + generations. +- Optionally rotate on UTC hourly, daily, or weekly boundaries while still + splitting within a period when the size threshold is reached. +- Provide stable exit codes and diagnostics for agents. + +### 3.2 Non-goals + +- `mpsc-log` does not provide query, dashboard, alerting, or retention + analysis. +- `mpsc-log` does not discover df12-build workflow sidecar directories. +- `mpsc-log` does not provide distributed locking across hosts. +- `mpsc-log` does not preserve textual duplicate JSON object keys. +- `mpsc-log` does not guarantee chronological record order by `timestamp`. + +## 4. Architecture + +The design is a synchronous CLI with a small domain core and filesystem +adapter. The core parses fields, merges configuration, produces a +`serde_json::Map`, and computes rotation actions. The adapter owns directory +creation, sidecar reads, lock acquisition, append, truncate repair, rename, and +gzip compression. + +```mermaid +flowchart LR + Agent[Agent or workflow] --> CLI[mpsc-log CLI] + CLI --> Parser[Argument parser] + CLI --> Config[Sidecar loader] + Parser --> Builder[Record builder] + Config --> Builder + Builder --> Writer[Journal writer] + Writer --> Lock[(Journal lock)] + Writer --> Journal[(JSONL journal)] + Writer --> Rotated[(Rotated logs)] +``` + +Figure 1: Runtime component topology. + +The lock file is the coordination boundary. `mpsc-log` derives it from the +journal path as `.lock` in the same directory. Every +invocation creates parent directories first, opens the lock file with create +semantics, acquires an exclusive lock, and only then reads configuration, +repairs the journal tail, rotates, compresses, and appends. + +## 5. CLI contract + +The first positional argument is always the journal path: + +```plaintext +mpsc-log [field-or-flag ...] +``` + +The field tail accepts this subset: + +| Syntax | Meaning | +| ---------------- | ---------------------------------------------------------- | +| `key=value` | Insert a value using schema coercion or default inference. | +| `key@value` | Insert a boolean using `jo` truthiness. | +| `key:=path` | Read JSON from `path` and insert it at `key`. | +| `key=@path` | Read `path` as UTF-8 text. | +| `key=:path` | Read `path` as JSON. | +| `key=%path` | Read `path` and base64 encode the bytes. | +| `-s key=value` | Force string coercion for the following word. | +| `-n key=value` | Force number coercion for the following word. | +| `-b key=value` | Force boolean coercion for the following word. | +| `-d.` or `-d .` | Set the object-path delimiter for later keys. | +| `key[]=value` | Append to an array at `key`. | +| `key[sub]=value` | Insert into an object at `key.sub`. | + +Unsupported `jo` options such as `-a`, `-p`, `-f`, `-D`, `-e`, `-v`, and `-V` +fail with `EX_USAGE`. The root is always an object. + +## 6. Sidecar configuration + +The sidecar path replaces the journal filename extension with `.toml`; if the +journal has no extension, the sidecar path appends `.toml`. For example, +`run.jsonl` uses `run.toml`, and `run` uses `run.toml`. + +The sidecar shape is defined by +[mpsc-log-sidecar.example.toml](mpsc-log-sidecar.example.toml). Configuration +has four tables: + +| Table | Responsibility | +| ------------ | ---------------------------------------------------------------------------------------------- | +| `[rotation]` | `schedule`, `max_bytes`, plain generation count, compressed generation count, and gzip policy. | +| `[locking]` | Lock timeout and partial-tail repair mode. | +| `[defaults]` | Default JSON fields inserted before CLI fields. | +| `[schema]` | Object paths mapped to coercion names: `string`, `number`, `boolean`, `json`, or `null`. | + +Merge order is deterministic: + +1. Start with sidecar `[defaults]`. +2. Apply CLI fields in argument order. +3. Insert generated `timestamp` if no field named `timestamp` exists. +4. Serialize the resulting object. + +Explicit `-s`, `-n`, and `-b` flags override the sidecar schema for that word. +When no flag or schema entry exists, default `jo` inference applies: valid JSON +values parse as JSON, empty `key=` becomes `null`, and other values remain +strings. + +## 7. Journal write protocol + +Each invocation follows one critical section: + +```mermaid +sequenceDiagram + participant A as Agent + participant M as mpsc-log + participant L as Lock file + participant J as Journal + participant R as Rotated logs + + A->>M: mpsc-log path fields... + M->>M: create parent directories + M->>L: open or create lock file + M->>L: acquire exclusive lock with timeout + M->>J: repair trailing partial record + M->>R: rotate and compress if threshold or schedule requires it + M->>J: append record plus newline + M->>L: release lock + M-->>A: exit 0 +``` + +Figure 2: Critical-section protocol. + +The writer records the active file length before appending. If `write_all` +returns an error, the writer truncates the active file back to that length +before returning. If the process dies during a write, the next invocation's +tail repair scans backward to the final `\n`, validates the remaining final +line if present, and truncates any partial tail before appending. + +The append path uses `OpenOptions::append(true).create(true)`. The lock, not +append mode alone, provides cross-process serialization. The design treats NFS, +CIFS, and other filesystems with weak advisory locking as unsupported unless +later verification proves the target mount behaves correctly. + +## 8. Rotation protocol + +Rotation runs while holding the journal lock. The sidecar `schedule` value is +one of `none`, `hourly`, `daily`, or `weekly`; the default is `none`. There is +no `max_age` setting in the v1 configuration surface. + +When `schedule = "none"`, rotation is size-only: + +- active threshold: 1 MiB; +- plain generations: four; +- compressed generations: 32; +- newest plain rotation: `.1`; +- oldest plain rotation before compression: `.4`; +- compressed rotations: `.5.gz` through `.36.gz`. + +For `run.jsonl`, the active path is `run.jsonl`; the newest plain rotation is +`run.1.jsonl`; the first compressed rotation is `run.5.jsonl.gz`. + +The size-only rotation order is oldest-to-newest: + +1. Delete generation 36 if present. +2. Rename compressed generations upward. +3. Gzip generation 4 into generation 5 and remove generation 4 only after the + gzip output is complete. +4. Rename plain generations 3 to 4, 2 to 3, and 1 to 2. +5. Rename the active file to generation 1. +6. Create a fresh active file by appending the pending record. + +When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period +boundaries derived from the invocation timestamp: + +| Schedule | Period beginning | Base rotated filename | +| -------- | --------------------------------------- | ------------------------- | +| `hourly` | UTC hour beginning | `run.2026-06-29T14.jsonl` | +| `daily` | UTC day beginning | `run.2026-06-29.jsonl` | +| `weekly` | UTC ISO week beginning, Monday 00:00:00 | `run.2026-06-29.jsonl` | + +Scheduled rotation is opportunistic rather than daemon-driven. If no invocation +occurs at the exact boundary, the next invocation rotates the previous active +segment before appending the new record. The rotated filename is still based on +the period that produced the records, not the time the rotation was observed. + +The scheduled protocol is: + +1. Determine the pending record period from the invocation timestamp. +2. Determine the active file period from the first complete record in the + active log, falling back to the pending period when the active log is empty. +3. If the active period is older than the pending record period, rotate the + active log into the active period's scheduled filename. +4. If the active log plus the pending record would exceed `max_bytes`, rotate + the active log into the pending period's next size-split filename. +5. Append the pending record to the active log. + +Within a scheduled period, reaching `max_bytes` before the time boundary +creates an interim size split. Size splits add a numeric suffix within that +period: `run.2026-06-29.1.jsonl`, `run.2026-06-29.2.jsonl`, and so on. If a +period has no size splits, its final scheduled archive uses the unsuffixed base +name. If a period already has one or more size-split files, the final archive +at the next time boundary also uses the next numeric suffix so that every file +for that period has an ordered generation number. + +Retention and compression are period-based in scheduled mode. The newest four +completed periods remain plain, including all size-split files inside those +periods. Older retained periods are gzipped, and periods beyond +`plain_generations + compressed_generations` are deleted. This keeps hourly, +daily, and weekly retention predictable even when one busy period produces many +size-split files. + +All gzip output is written through `atomic-write-file`, then committed. A +failed compression leaves the source generation in place and aborts the +invocation before appending the new record. + +## 9. df12-build journal records + +The normative JSON Schema lives in +[mpsc-log-event-schema.json](mpsc-log-event-schema.json). The schema requires +`timestamp`, permits additional fields, and reserves structured namespaces for +`run`, `task`, `attempt`, `coderabbit`, `audit`, and `defect`. + +The df12-build integration should emit these event names first: + +| Event | Required fields | Purpose | +| -------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| `phase.started` | `run.id`, `attempt.phase`, `attempt.agent` | Start timing a workflow phase. | +| `phase.finished` | `run.id`, `attempt.phase`, `attempt.status`, `attempt.duration_ms` | Measure phase duration and outcome. | +| `review.round` | `task.id`, `attempt.phase`, `attempt.round`, `attempt.status` | Measure design, code, and expert review round distributions. | +| `task.finished` | `task.id`, `task.phase`, `task.work_items`, `task.changed_files`, `task.dependency_depth` | Correlate outcome with roadmap shape. | +| `coderabbit.attempt` | `coderabbit.status`, `coderabbit.wait_ms`, `coderabbit.retry`, `coderabbit.deferred` | Measure waits, HTTP 429s, deferrals, and retry success. | +| `audit.finding` | `audit.severity`, `audit.lane`, `audit.finding_count` | Measure audit yield and remediation routing. | +| `defect.escape` | `defect.source`, `defect.escaped_from`, `defect.severity` | Measure defects found after merge. | + +Example invocation: + +```sh +mpsc-log .odw/run-20260629/journal.jsonl \ + event=review.round \ + run.id=df12-20260629-001 \ + run.workflow=df12-build-odw \ + task.id=1.2.8 \ + attempt.phase=code-review \ + -n attempt.round=2 \ + attempt.status=changes-requested +``` + +## 10. Errors and exit codes + +The library exposes a semantic error enum. The binary maps it to `sysexits` +style process statuses: + +| Exit | Name | Condition | +| ---- | -------------- | ---------------------------------------------------------------------------------------- | +| 0 | `OK` | Record appended. | +| 64 | `EX_USAGE` | Invalid CLI syntax, unsupported `jo` option, non-object root, or duplicate invalid path. | +| 65 | `EX_DATAERR` | Invalid JSON, invalid TOML, invalid schema coercion, or malformed sidecar value. | +| 73 | `EX_CANTCREAT` | Parent directory, lock file, journal, or rotated file cannot be created. | +| 74 | `EX_IOERR` | Write, flush, truncate, rename, compression, or filesystem metadata failure. | +| 75 | `EX_TEMPFAIL` | Lock timeout. | +| 78 | `EX_CONFIG` | Sidecar configuration is syntactically valid TOML but semantically invalid. | + +Diagnostics go to standard error as one line: + +```plaintext +mpsc-log: : : +``` + +The command writes nothing to standard output on success. + +## 11. Correctness properties and verification + +The implementation must satisfy these properties: + +| Property | Verification method | +| -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| Successful invocations produce exactly one valid JSON object line. | Behavioural tests and property tests over generated field words. | +| Concurrent successful invocations produce the same number of complete records as successes. | Stress test with many processes writing one journal. | +| Rotation preserves successful records across active, plain rotated, scheduled rotated, and compressed rotated files. | End-to-end test that forces size and scheduled rotation under concurrent writers and counts decoded records. | +| Detected write failures do not leave a partial final line. | Fault-injection adapter test that fails after partial writes and checks truncate repair. | +| Crash-like partial tails are repaired before the next append. | Fixture with malformed trailing bytes followed by a successful append. | +| CLI coercion follows the accepted `jo` subset. | Parameterized examples copied from the accepted `jo` behaviours. | +| Sidecar/CLI precedence is deterministic. | Table-driven tests covering defaults, schema coercion, explicit flags, and `timestamp`. | + +The combination surface is `jo` syntax form × coercion source × object path × +sidecar default × rotation schedule × rotation state × lock contention. The +test suite must cover pairwise combinations across those axes, plus targeted +cases for size and scheduled rotation under contention and failed filesystem +operations. + +Network filesystems remain outside the correctness claim until a separate +platform verification matrix proves lock and rename behaviour for a named +filesystem and mount configuration. + +## 12. Module structure + +`src/main.rs` owns CLI startup and process exit mapping only. The library owns +the implementation: + +| Module | Responsibility | +| --------- | ------------------------------------------------------------------------------ | +| `args` | Parse the journal path and raw `jo` field tail. | +| `fields` | Parse field words, object paths, coercion flags, and file-value forms. | +| `config` | Load and validate sidecar TOML. | +| `record` | Merge defaults, CLI fields, schema coercions, and generated timestamp. | +| `journal` | Create directories, acquire locks, repair tails, rotate, compress, and append. | +| `errors` | Semantic error enum and exit-code mapping. | +| `clock` | Injectable timestamp source for deterministic tests. | +| `fs` | Filesystem adapter boundary for fault injection. | + +The first implementation should expose no stable library API beyond what the +binary needs. Public library exports remain internal support until a roadmap +item explicitly commits to a supported API. + +## 13. Deferred ADRs + +- Accepted `jo` subset and last-wins duplicate-key handling. +- Lock-file naming and filesystem support policy. +- Rotation naming, compression, and retention defaults. +- CLI-only product boundary versus public Rust library API. + +## Appendix A. References + +[^jsonl]: [JSON Lines](https://jsonlines.org/), accessed 2026-06-29. + +[^jo]: Jan-Piet Mens, + [`jo` manual](https://github.com/jpmens/jo/blob/master/jo.md), + accessed 2026-06-29. + +[^rfc3339]: G. Klyne and C. Newman, + [RFC 3339: Date and time on the Internet: timestamps](https://www.rfc-editor.org/rfc/rfc3339), + July 2002, accessed 2026-06-29. + +[^toml]: Tom Preston-Werner, Pradyun Gedam, et al., + [TOML v1.0.0](https://toml.io/en/v1.0.0), accessed 2026-06-29. + +[^flock]: Michael Kerrisk, + [flock(1) Linux manual page](https://man7.org/linux/man-pages/man1/flock.1.html), + util-linux manual, accessed 2026-06-29. + +[^logrotate]: Michael Kerrisk, + [logrotate(8) Linux manual page](https://man7.org/linux/man-pages/man8/logrotate.8.html), + accessed 2026-06-29. + +[^clap]: [`clap` crate documentation](https://docs.rs/clap/latest/clap/), + version 4.6.1, accessed 2026-06-29. + +[^jiff]: [`jiff` crate documentation](https://docs.rs/jiff/latest/jiff/), + version 0.2.29, accessed 2026-06-29. + +[^fs4]: [`fs4` crate documentation](https://docs.rs/fs4/latest/fs4/), + version 1.1.0, accessed 2026-06-29. + +[^flate2]: [`flate2` crate documentation](https://docs.rs/flate2/latest/flate2/), + version 1.1.9, accessed 2026-06-29. diff --git a/docs/mpsc-log-event-schema.json b/docs/mpsc-log-event-schema.json new file mode 100644 index 0000000..7c05071 --- /dev/null +++ b/docs/mpsc-log-event-schema.json @@ -0,0 +1,83 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/leynos/mpsc-log/docs/mpsc-log-event-schema.json", + "title": "mpsc-log journal record", + "type": "object", + "additionalProperties": true, + "required": ["timestamp"], + "properties": { + "timestamp": { + "type": "string", + "format": "date-time", + "description": "RFC 3339 UTC invocation timestamp." + }, + "schema_version": { + "type": "integer", + "minimum": 1 + }, + "event": { + "type": "string", + "minLength": 1 + }, + "run": { + "type": "object", + "additionalProperties": true, + "properties": { + "id": { "type": "string" }, + "workflow": { "type": "string" }, + "sidecar_dir": { "type": "string" } + } + }, + "task": { + "type": "object", + "additionalProperties": true, + "properties": { + "id": { "type": "string" }, + "phase": { "type": "string" }, + "work_items": { "type": "integer", "minimum": 0 }, + "changed_files": { "type": "integer", "minimum": 0 }, + "dependency_depth": { "type": "integer", "minimum": 0 } + } + }, + "attempt": { + "type": "object", + "additionalProperties": true, + "properties": { + "agent": { "type": "string" }, + "phase": { "type": "string" }, + "round": { "type": "integer", "minimum": 1 }, + "status": { "type": "string" }, + "duration_ms": { "type": "integer", "minimum": 0 } + } + }, + "coderabbit": { + "type": "object", + "additionalProperties": true, + "properties": { + "status": { "type": "string" }, + "http_status": { "type": "integer", "minimum": 100, "maximum": 599 }, + "wait_ms": { "type": "integer", "minimum": 0 }, + "retry": { "type": "boolean" }, + "deferred": { "type": "boolean" } + } + }, + "audit": { + "type": "object", + "additionalProperties": true, + "properties": { + "severity": { "type": "string" }, + "lane": { "type": "string" }, + "finding_count": { "type": "integer", "minimum": 0 } + } + }, + "defect": { + "type": "object", + "additionalProperties": true, + "properties": { + "source": { "type": "string" }, + "escaped_from": { "type": "string" }, + "severity": { "type": "string" } + } + } + } +} diff --git a/docs/mpsc-log-sidecar.example.toml b/docs/mpsc-log-sidecar.example.toml new file mode 100644 index 0000000..09936ce --- /dev/null +++ b/docs/mpsc-log-sidecar.example.toml @@ -0,0 +1,28 @@ +[rotation] +schedule = "none" +max_bytes = 1048576 +plain_generations = 4 +compressed_generations = 32 +gzip_after_plain_generations = true + +[locking] +timeout_ms = 5000 +repair_partial_tail = true + +[defaults] +schema_version = 1 +event = "workflow.event" + +[schema] +"run.id" = "string" +"run.workflow" = "string" +"task.work_items" = "number" +"task.changed_files" = "number" +"task.dependency_depth" = "number" +"attempt.round" = "number" +"attempt.duration_ms" = "number" +"coderabbit.http_status" = "number" +"coderabbit.wait_ms" = "number" +"coderabbit.retry" = "boolean" +"coderabbit.deferred" = "boolean" +"audit.finding_count" = "number" diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..c8f6d47 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,489 @@ +# mpsc-log roadmap + +This roadmap translates the terms of reference and technical design into an +outcome-oriented delivery sequence for `mpsc-log`. It does not promise dates. +Each phase carries one testable idea at the GIST level; each step answers a +sequencing question; each task is a review-sized execution unit with explicit +dependencies and source citations. + +The primary source documents are [terms of reference](terms-of-reference.md), +[technical design](mpsc-log-design.md), [context](context.md), +[event schema](mpsc-log-event-schema.json), and +[sidecar example](mpsc-log-sidecar.example.toml). No RFCs or ADRs exist yet, +so the first phase records the decisions that would otherwise force rework. + +## 1. Foundational contracts and build spine + +Idea: if `mpsc-log` ratifies its v1 contracts, crate boundary, and validation +spine before feature work starts, later slices can converge on one small CLI +instead of repeatedly reopening syntax, locking, and retention decisions. + +This phase resolves the decisions that affect every subsequent pull request: +the accepted `jo` subset, public API boundary, filesystem support policy, +rotation naming, and the repository shape needed to validate a CLI whose main +risks are concurrency and persistence. + +### 1.1. Ratify the v1 decisions that would otherwise cause rework + +This step answers what `mpsc-log` v1 will promise to callers and what it will +explicitly leave out. Its outcome informs crate layout, user documentation, +error handling, and future compatibility work. See +mpsc-log-design.md §§3, 5, 8, 12-13 and terms-of-reference.md §§6, 8-10. + +- [ ] 1.1.1. Record the accepted `jo` subset and duplicate-key behaviour in + an ADR. + - See mpsc-log-design.md §§2, 5, 13 and terms-of-reference.md §§6, 9. + - Success: the ADR names supported forms, rejected options, object-root + enforcement, object-path handling, and last-wins duplicate-key semantics. +- [ ] 1.1.2. Record the CLI-only v1 product boundary in an ADR. + - See mpsc-log-design.md §§3, 10, 12-13 and terms-of-reference.md §§6, 9. + - Success: the ADR states that `src/main.rs` owns process exit mapping and + that library exports are internal until a later roadmap item changes that. +- [ ] 1.1.3. Record the filesystem and locking support policy in an ADR. + - See mpsc-log-design.md §§2, 4, 7, 11, 13 and + terms-of-reference.md §§6-8. + - Success: the ADR defines the local-filesystem correctness claim, lock-file + naming, unsupported network-filesystem caveat, and timeout semantics. +- [ ] 1.1.4. Record the rotation naming, compression, and retention policy in + an ADR. + - See mpsc-log-design.md §§3, 6, 8, 11, 13 and + terms-of-reference.md §§6, 8-9. + - Success: the ADR covers `schedule = "none"`, scheduled UTC period names, + size-split suffixes, gzip-after-four behaviour, and the absence of + `max_age`. + +### 1.2. Establish the implementation skeleton and quality gates + +This step answers whether the generated Rust scaffold can support the module +boundaries, dependency choices, and validation strategy described by the +design. It informs all later slices because every feature will pass through the +same CLI, domain, filesystem, and test seams. See mpsc-log-design.md §§2, 4, +10-12 and docs/repository-layout.md. + +- [ ] 1.2.1. Replace the scaffold library with the module skeleton described + by the design. + - Requires steps 1.1.1-1.1.3. + - See mpsc-log-design.md §§4, 10, 12. + - [ ] Add `args`, `fields`, `config`, `record`, `journal`, `errors`, + `clock`, and `fs` modules with module-level documentation. + - [ ] Keep `src/main.rs` limited to startup and exit-code mapping. + - Success: the crate builds with empty but documented module boundaries and + no stable public API beyond the binary's current needs. +- [ ] 1.2.2. Add the implementation dependencies with documented ownership. + - Requires 1.2.1. + - See mpsc-log-design.md §2 and docs/developers-guide.md. + - [ ] Add the selected crates for CLI parsing, JSON, TOML, time, locking, + gzip, atomic writes, and semantic errors using caret requirements. + - [ ] Document each dependency's purpose in the appropriate design or + developer guide location. + - Success: `make check-fmt`, `make lint`, and `make test` pass after the + dependency update. +- [ ] 1.2.3. Build deterministic test seams for time and filesystem effects. + - Requires 1.2.1 and 1.2.2. + - See mpsc-log-design.md §§7, 10-12 and + docs/reliable-testing-in-rust-via-dependency-injection.md. + - [ ] Provide injectable clock and filesystem adapter boundaries for record + timestamps, fault injection, and deterministic rotation fixtures. + - Success: failures such as partial writes, metadata errors, and fixed + timestamps can be exercised without mutating global process state. + +## 2. Day-one structured journal entries + +Idea: if one `mpsc-log` invocation can parse `jo`-style fields, apply sidecar +defaults, emit an RFC 3339 UTC timestamp, and append one JSONL object, the tool +already replaces the unsafe `jo >> file.jsonl` baseline for simple workflows. + +This phase delivers the narrowest useful end-to-end command. It intentionally +starts without the full concurrency and rotation surface so that parser, +configuration, record-shaping, and diagnostics can stabilize against real CLI +examples before the filesystem protocol grows more complex. + +### 2.1. Prove the CLI field contract can build one object record + +This step answers whether the accepted `jo` subset can be implemented without +breaking the object-root logging contract. It informs sidecar coercion, error +classification, and the later combinatorial test matrix. See +mpsc-log-design.md §§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. + +- [ ] 2.1.1. Implement positional argument parsing for the journal path and + raw field tail. + - Requires steps 1.1-1.2. + - See mpsc-log-design.md §§5, 10, 12. + - Success: the parser preserves field words and coercion flags in order and + rejects missing journal paths with `EX_USAGE`. +- [ ] 2.1.2. Implement field-word parsing for the accepted `jo` object forms. + - Requires 2.1.1. + - See mpsc-log-design.md §§2, 5 and terms-of-reference.md §§2, 6. + - [ ] Cover `key=value`, `key@value`, object paths, array appends, and + bracketed object insertion. + - [ ] Reject unsupported `jo` options and any non-object root outcome. + - Success: representative `jo` examples produce typed intermediate values + or stable usage errors. +- [ ] 2.1.3. Implement explicit and inferred type coercion for CLI values. + - Requires 2.1.2. + - See mpsc-log-design.md §§5-6 and context.md. + - Success: explicit `-s`, `-n`, and `-b` flags override inference, empty + assignments become `null`, valid JSON values parse as JSON, and other + values remain strings. +- [ ] 2.1.4. Implement file-value forms with clear data and I/O errors. + - Requires 2.1.2 and 2.1.3. + - See mpsc-log-design.md §§5, 10. + - Success: `key:=path`, `key=@path`, `key=:path`, and `key=%path` produce + the documented values or `EX_DATAERR`/`EX_IOERR` diagnostics. + +### 2.2. Deliver sidecar-backed record construction + +This step answers whether configuration, schema coercion, defaults, and the +generated timestamp can merge deterministically. Its outcome informs the +external JSONL contract and df12-build event examples. See +mpsc-log-design.md §§2, 6, 9, 11-12, +mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. + +- [ ] 2.2.1. Implement sidecar path derivation, TOML loading, and semantic + validation. + - Requires 2.1.1. + - See mpsc-log-design.md §§2, 6, 10 and + mpsc-log-sidecar.example.toml. + - Success: missing sidecars use defaults, malformed TOML returns + `EX_DATAERR`, and semantically invalid configuration returns `EX_CONFIG`. +- [ ] 2.2.2. Implement deterministic record merging and schema-guided + coercion. + - Requires 2.1.3 and 2.2.1. + - See mpsc-log-design.md §§6, 9, 11 and + mpsc-log-event-schema.json. + - Success: sidecar defaults, CLI fields, explicit flags, schema entries, + and duplicate paths resolve according to one table-driven contract. +- [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. + - Requires 1.2.3 and 2.2.2. + - See mpsc-log-design.md §§2, 6 and terms-of-reference.md §§2, 6, 8. + - Success: records receive an invocation-time UTC timestamp unless a prior + merge step already produced `timestamp`. + +### 2.3. Append the first useful JSONL record + +This step answers whether the command can complete a non-concurrent append +workflow with stable user-visible behaviour. It informs the later lock and +repair protocol because this slice defines the line format and diagnostics. +See mpsc-log-design.md §§3, 7, 10-12 and terms-of-reference.md §§5-7. + +- [ ] 2.3.1. Implement compact JSON object serialization and newline append. + - Requires steps 2.1-2.2. + - See mpsc-log-design.md §§3, 7, 11-12. + - Success: each successful invocation appends exactly one compact JSON + object followed by `\n`. +- [ ] 2.3.2. Map semantic errors to process exit codes and diagnostics. + - Requires steps 2.1-2.2. + - See mpsc-log-design.md §10 and terms-of-reference.md §§6-7. + - Success: success writes nothing to stdout, failures write one stderr + diagnostic, and invalid input, data errors, I/O errors, and configuration + errors use the documented statuses. +- [ ] 2.3.3. Document the day-one CLI usage in the users' guide. + - Requires 2.3.1 and 2.3.2. + - See mpsc-log-design.md §§5-6, 9-10 and + terms-of-reference.md §§4-7. + - Success: a workflow author can replace a simple `jo >> file.jsonl` + example with an equivalent `mpsc-log` invocation and understand failures. + +## 3. Trustworthy shared logging under contention + +Idea: if the same CLI can create missing directories, serialize concurrent +writers with bounded lock waiting, and repair partial tails, multi-agent +workflows can treat journal writes as boring infrastructure rather than a +source of telemetry loss. + +This phase turns the single-writer command into the product promised by the +terms of reference. It focuses on the one risk that motivates the tool: +several independent agents calling the same executable at the same time. + +### 3.1. Prove first-write and lock acquisition are safe + +This step answers whether concurrent invocations can create the same journal +and coordination artefacts without truncating or colliding. It informs the +critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, +7, 10-11 and terms-of-reference.md §§6-8. + +- [ ] 3.1.1. Create missing parent directories and coordination artefacts + before locking. + - Requires phase 2. + - See mpsc-log-design.md §§3-4, 7, 10 and + terms-of-reference.md §§6-8. + - Success: simultaneous first writes to a missing directory tree either + create one usable journal or fail with a specific creation diagnostic. +- [ ] 3.1.2. Implement exclusive journal locking with a configurable timeout. + - Requires 3.1.1. + - See mpsc-log-design.md §§2, 4, 6-7, 10 and context.md. + - Success: contending processes serialize through the same lock file, and + lock timeout failures return `EX_TEMPFAIL`. +- [ ] 3.1.3. Read sidecar configuration inside the journal critical section. + - Requires 3.1.2. + - See mpsc-log-design.md §§4, 6-7 and terms-of-reference.md §8.2. + - Success: each invocation uses one coherent sidecar view for repair, + rotation, compression, and append. + +### 3.2. Preserve complete records across write and crash-like failures + +This step answers whether interrupted or failed writes can leave the journal in +a state the next invocation can recover. It informs the fault-injection adapter +and validates the design's append-plus-repair claim. See mpsc-log-design.md +§§7, 10-12 and terms-of-reference.md §§7-8. + +- [ ] 3.2.1. Implement append rollback when a write fails after extending the + active file. + - Requires 3.1.2. + - See mpsc-log-design.md §§7, 10-11. + - Success: injected write failures truncate the active file back to its + recorded pre-append length before returning an error. +- [ ] 3.2.2. Implement partial-tail repair before every append. + - Requires 3.2.1. + - See mpsc-log-design.md §§6-7, 11. + - Success: malformed trailing bytes and unterminated final records are + removed before a new valid record is appended. +- [ ] 3.2.3. Build the fault-injection filesystem coverage for write, truncate, + rename, compression, and metadata failures. + - Requires 1.2.3, 3.2.1, and 3.2.2. + - See mpsc-log-design.md §§7, 10-12. + - Success: each documented filesystem failure class has a deterministic + assertion for journal state and exit-code mapping. + +### 3.3. Demonstrate concurrent append correctness end to end + +This step answers whether many real processes can use `mpsc-log` at once and +produce one complete record per successful command. It informs release +readiness for the first agent-workflow adoption. See mpsc-log-design.md §11 +and terms-of-reference.md §§5, 7. + +- [ ] 3.3.1. Build a multi-process concurrent append stress harness. + - Requires steps 3.1-3.2. + - See mpsc-log-design.md §11 and terms-of-reference.md §7.1. + - Success: a high-contention run produces the same number of complete, + decodable JSON object lines as successful child processes. +- [ ] 3.3.2. Add pairwise CLI/configuration combination coverage. + - Requires phase 2 and 3.3.1. + - See mpsc-log-design.md §11. + - Success: the suite covers `jo` syntax form, coercion source, object path, + sidecar default, rotation schedule, rotation state, and lock contention. + +## 4. Rotation and retention without record loss + +Idea: if rotation, compression, and retention run under the same lock as +append, operators can keep journal files bounded without handing correctness +back to cron jobs or external shell glue. + +This phase delivers bounded storage while preserving the phase 3 concurrency +claim. It treats size-only rotation and scheduled rotation as user-facing modes +that need separate naming and retention evidence. + +### 4.1. Deliver size-only rotation for the default policy + +This step answers whether the default `schedule = "none"` policy can rotate +under contention without losing records. It informs the gzip and scheduled +rotation work because both build on the same locked action planner. See +mpsc-log-design.md §§6, 8, 11 and mpsc-log-sidecar.example.toml. + +- [ ] 4.1.1. Implement size-threshold detection and numeric generation + planning. + - Requires phase 3 and 1.1.4. + - See mpsc-log-design.md §§6, 8, 11. + - Success: `run.jsonl` rotates to `run.1.jsonl` after the configured + threshold while preserving the pending record. +- [ ] 4.1.2. Implement oldest-to-newest size-only rename execution. + - Requires 4.1.1. + - See mpsc-log-design.md §8. + - Success: generations advance without overwriting retained files and + rotation failures leave the previous readable state intact. +- [ ] 4.1.3. Document size-only rotation and retention in the users' guide. + - Requires 4.1.2. + - See mpsc-log-design.md §§6, 8 and terms-of-reference.md §§6-7. + - Success: users can predict active, plain rotated, and compressed filenames + from a journal path and sidecar settings. + +### 4.2. Compress and retain rotated logs atomically + +This step answers whether older generations can be gzipped without introducing +the data-loss window the design rejects. It informs scheduled-mode retention +because both modes rely on atomic gzip output and deletion ordering. See +mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. + +- [ ] 4.2.1. Implement atomic gzip output for rotated generations. + - Requires 4.1.2. + - See mpsc-log-design.md §§2, 8. + - Success: failed compression leaves the source generation in place and + aborts before appending the pending record. +- [ ] 4.2.2. Implement retention deletion for plain and compressed + generations. + - Requires 4.2.1. + - See mpsc-log-design.md §§6, 8. + - Success: files beyond `plain_generations + compressed_generations` are + deleted only after newer retained files are safely in place. + +### 4.3. Deliver UTC scheduled rotation with size splits + +This step answers whether hourly, daily, and weekly modes rotate at period +boundaries while still splitting busy periods by bytes. It informs the final +operator contract because scheduled rotation is where naming expectations are +most visible. See mpsc-log-design.md §§6, 8, 11, +mpsc-log-sidecar.example.toml, and context.md. + +- [ ] 4.3.1. Implement period calculation for `hourly`, `daily`, and + `weekly` schedules. + - Requires 1.2.3 and 4.1.1. + - See mpsc-log-design.md §§6, 8 and context.md. + - Success: UTC boundaries produce the documented hour, day, and ISO-week + beginning names without using a `max_age` configuration field. +- [ ] 4.3.2. Implement active-period detection and scheduled boundary + rotation. + - Requires 4.3.1. + - See mpsc-log-design.md §8. + - Success: the next invocation after a time break archives the previous + active segment under the period that produced its records. +- [ ] 4.3.3. Implement interim size-split suffixes inside scheduled periods. + - Requires 4.3.2. + - See mpsc-log-design.md §8. + - Success: busy periods produce ordered `.n` suffixes, and final scheduled + archives use the next suffix when a period already has size splits. +- [ ] 4.3.4. Implement scheduled-mode period retention and compression. + - Requires 4.2.2 and 4.3.3. + - See mpsc-log-design.md §8. + - Success: the newest completed periods remain plain as complete groups, + older retained periods are gzipped, and expired periods are deleted. + +### 4.4. Demonstrate rotation correctness under contention + +This step answers whether size-only and scheduled rotation preserve every +successful record when many processes append concurrently. It informs release +readiness because it exercises the highest-risk feature interactions. See +mpsc-log-design.md §11 and terms-of-reference.md §7. + +- [ ] 4.4.1. Build the concurrent rotation end-to-end suite. + - Requires steps 4.1-4.3. + - See mpsc-log-design.md §11. + - Success: forced size and scheduled rotations under concurrent writers + preserve the decoded record count across active, plain, scheduled, and + compressed files. +- [ ] 4.4.2. Add fixture coverage for failed rotation and compression actions. + - Requires 4.4.1. + - See mpsc-log-design.md §§8, 10-11. + - Success: injected rename, delete, gzip, and commit failures leave a + readable previous state and emit the documented exit code. + +## 5. df12-build telemetry adoption + +Idea: if the finished CLI can emit the first df12-build journal taxonomy with +documented sidecar defaults and realistic invocation fixtures, the workflow +can start collecting real-run evidence without `mpsc-log` becoming an analysis +or dashboard product. + +This phase connects the generic journalling tool to its first concrete use +case. It packages examples and verification around phase timing, review rounds, +task shape, CodeRabbit waits, audit yield, remediation lanes, and escaped +defects while respecting the non-goals around discovery and analytics. + +### 5.1. Make the df12-build event contract executable + +This step answers whether the schema, sidecar defaults, and example event names +are specific enough for the ODW workflow to call. It informs workflow adoption +and future telemetry analysis outside this crate. See mpsc-log-design.md §9, +mpsc-log-event-schema.json, mpsc-log-sidecar.example.toml, and +terms-of-reference.md §§2, 5-7. + +- [ ] 5.1.1. Add fixture invocations for the initial df12-build event names. + - Requires phase 4. + - See mpsc-log-design.md §9 and terms-of-reference.md §§2, 5-7. + - Success: fixtures cover `phase.started`, `phase.finished`, + `review.round`, `task.finished`, `coderabbit.attempt`, `audit.finding`, + and `defect.escape`. +- [ ] 5.1.2. Validate fixture output against the JSON Schema contract. + - Requires 5.1.1. + - See mpsc-log-design.md §9 and mpsc-log-event-schema.json. + - Success: generated fixture records satisfy required fields, reserved + namespaces, integer ranges, boolean coercions, and RFC 3339 timestamps. +- [ ] 5.1.3. Provide a df12-build sidecar configuration example. + - Requires 5.1.2. + - See mpsc-log-design.md §§6, 9 and + mpsc-log-sidecar.example.toml. + - Success: the example sidecar coerces the telemetry fields needed for the + first workflow integration without imposing organization-wide schema + governance. + +### 5.2. Document operator-facing adoption without expanding scope + +This step answers whether workflow authors and operators can use the tool +correctly without reading implementation code. It informs v1 release readiness +and keeps deferred analysis work out of the core CLI. See +terms-of-reference.md §§3-7 and mpsc-log-design.md §§3, 9-10. + +- [ ] 5.2.1. Update the users' guide with complete operational examples. + - Requires 5.1.3. + - See mpsc-log-design.md §§5-10 and terms-of-reference.md §§4-7. + - Success: the guide covers simple append, sidecar defaults, coercion + errors, lock timeout, size rotation, scheduled rotation, and df12-build + event examples. +- [ ] 5.2.2. Add troubleshooting guidance for agent callers. + - Requires 5.2.1. + - See mpsc-log-design.md §§7, 10-11 and terms-of-reference.md §§7-8. + - Success: operators can distinguish invalid arguments, malformed sidecars, + timeout, directory creation failure, partial-tail repair, and rotation + failure from command output and exit status. +- [ ] 5.2.3. Add a release smoke script for the documented workflows. + - Requires 5.2.1 and 5.2.2. + - See mpsc-log-design.md §§5-11. + - Success: the smoke script exercises documented commands against temporary + journals and fails if examples drift from the binary. + +## 6. Deferred extensions after the core v1 promise + +Idea: if the core v1 promise is already trustworthy and boring to operate, the +project can evaluate broader extensions on their product value instead of +letting them destabilize the main release. + +This phase captures work the current ToR and design explicitly defer. These +items should not block v1 unless a later ADR changes the product boundary. + +### 6.1. Evaluate broader observability and analysis features + +This step keeps dashboards, queries, and automatic tuning out of the core CLI +while preserving them as possible downstream products. See +terms-of-reference.md §§6.2, 7.3 and mpsc-log-design.md §3.2. + +- [ ] 6.1.1. Decide whether query, dashboard, or recommendation tooling belongs + in a separate crate or repository. + - Requires phase 5. + - See terms-of-reference.md §§6.2, 7.3 and mpsc-log-design.md §3.2. + - Success: the decision does not add query or dashboard obligations to the + append-only CLI. +- [ ] 6.1.2. Decide whether df12-build analysis should consume journal output + as a downstream tool. + - Requires 6.1.1. + - See terms-of-reference.md §§2, 6.2, 7.3 and mpsc-log-design.md §9. + - Success: any tuning or reporting work has a separate owner and does not + change the v1 write contract. + +### 6.2. Evaluate expanded platform and compatibility promises + +This step keeps distributed coordination, full `jo` parity, public library +APIs, and `max_age` out of v1 while identifying the decisions required if they +become valuable later. See terms-of-reference.md §§6.2, 8-9 and +mpsc-log-design.md §§3.2, 11-13. + +- [ ] 6.2.1. Decide whether network-filesystem support should graduate from + caveat to tested platform promise. + - Requires phase 5. + - See mpsc-log-design.md §§7, 11 and terms-of-reference.md §§6.2, 8.2. + - Success: the project either publishes a named filesystem verification + matrix or leaves the unsupported-network-filesystem caveat intact. +- [ ] 6.2.2. Decide whether full `jo` compatibility has enough value to expand + the object-root subset. + - Requires phase 5. + - See mpsc-log-design.md §§2, 5, 13 and terms-of-reference.md §§6.2, 9. + - Success: unsupported `jo` behaviours stay rejected unless an ADR explains + the compatibility gain and object-root implications. +- [ ] 6.2.3. Decide whether to expose a supported Rust library API. + - Requires phase 5. + - See mpsc-log-design.md §§3.2, 12-13 and terms-of-reference.md §9. + - Success: public API compatibility remains out of v1 unless an ADR defines + supported types, versioning, and documentation obligations. +- [ ] 6.2.4. Decide whether retention by age should be added after v1. + - Requires phase 5. + - See mpsc-log-design.md §§6, 8 and terms-of-reference.md §8.1. + - Success: `max_age` remains absent unless a later design defines its + interaction with size, schedule, compression, and concurrent writers. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md new file mode 100644 index 0000000..2fad8be --- /dev/null +++ b/docs/terms-of-reference.md @@ -0,0 +1,370 @@ +# mpsc-log – terms of reference + +- **Status:** Draft v0.1. +- **Audience:** Product owner, engineering maintainers, agent-workflow + authors, and future design reviewers. +- **Last substantive revision:** 2026-06-29. +- **Companion documents:** [Documentation contents](contents.md), + [users' guide](users-guide.md), [developer guide](developers-guide.md), + [repository layout](repository-layout.md), + [documentation style guide](documentation-style-guide.md), + [context](context.md), [technical design](mpsc-log-design.md), + [event schema](mpsc-log-event-schema.json), + and [sidecar example](mpsc-log-sidecar.example.toml). + +## 1. Background and motivation + +`mpsc-log` exists for multi-agent and scripted workflows that need several +independent processes to record structured events in one append-only log file. +The immediate problem is not creating JSON. Tools such as `jo` already turn +shell arguments into JSON objects.[^1] The gap is that agents also need a small +command-line interface (CLI) that appends each record to a shared JSON Lines +(JSONL) file without overwriting existing data, losing concurrent writes, or +corrupting the file during rotation. + +The motivating context is local automation where several agents may run at the +same time and cannot coordinate through a long-lived service. Each agent should +be able to invoke the tool once, pass a target path and record fields, and then +continue. The tool must make the file-system coordination boring: safe file +creation, file locking, write atomicity, timeout handling, and rotation belong +inside the tool rather than being reimplemented by each caller. + +The first concrete use case is +`../df12-build.worktrees/codex-annex/workflows/df12-build-odw.js`, a +multi-agent Open Dynamic Workflows (ODW) workflow for advancing +`docs/roadmap.md` through planning, design review, implementation, code review, +expert review, integration, audit, and remediation. Agents in that workflow +need to append journal events to a file in the workflow sidecar directory. The +journal should capture enough real-run telemetry to replace architectural +guesses with measured behaviour. + +This project is being defined before its design document. The existing +repository is a generated Rust application scaffold, so this terms of reference +treats the user brief and `Cargo.toml` package description as the authoritative +product inputs. + +## 2. Domain + +The product sits in structured local logging for automated command runners. Its +records are JSON objects serialized as one JSON value per line. JSONL keeps the +log easy to append, stream, grep, split, and ingest into later tools without +requiring the whole file to be rewritten. + +The command accepts the log path as its first argument. Later arguments +describe fields using `jo`-style key and value words, including type coercion +flags and object paths. Unlike `jo`, the root value must always be an object: +array roots are out of scope because each log entry needs named fields and +configurable defaults. + +The default entry includes a `timestamp` field containing a Coordinated +Universal Time (UTC) timestamp captured when the command is invoked. The +timestamp uses RFC 3339, the Internet timestamp profile of ISO 8601.[^2] + +The tool also reads a sidecar TOML file next to the log file. The sidecar has +the same base filename as the log file and a `.toml` extension. It defines +rotation configuration, type-coercion schema, and default field values. + +The term "sidecar" has two relevant meanings in the current domain. The ODW +workflow has a workflow sidecar directory where run artefacts belong. +`mpsc-log` also has a sidecar TOML configuration file next to the selected +journal file. The design must keep those concepts distinct: callers choose the +journal path, and `mpsc-log` derives only its configuration path from that +journal path. + +The first journaled telemetry set is operational evidence for the df12-build +workflow: + +- phase timings and design-review or code-review round counts; +- task outcomes grouped by roadmap shape, including task size, work-item count, + changed-file count, dependency depth, phase kind, review failures, + implementation failures, merge conflicts, and audit follow-ups; +- CodeRabbit wait times, HTTP 429 frequency, deferred reviews, and retry + success; +- audit yield by severity and remediation lane, including dropped findings, + addenda, rerouted step tasks, and new roadmap steps; +- post-merge defect escape, including defects later found by dogfooding, + continuous integration, users, or subsequent roadmap work. + +## 3. Market context + +The current alternatives each solve part of the problem: + +- `jo` builds JSON from shell arguments but writes to standard output and does + not provide locked append, timeout, file creation, or rotation semantics. +- Shell redirection with `>>` is convenient but leaves callers responsible for + record construction, validation, rotation, and coordination under contention. +- `flock` can serialize shell commands, but each caller must compose locking, + JSON construction, append mode, error handling, and rotation correctly. +- General logging frameworks target applications that own their logging stack; + they are a poor fit for short-lived independent agents that need a single + external command. +- System loggers and centralized observability products handle broader + collection and retention concerns, but they introduce services, deployment, + and integration work that is disproportionate for a local agent log. +- `logrotate` can rotate files, but it is scheduled external maintenance rather + than a per-write safety mechanism integrated with concurrent appends. + +The gap is a narrowly scoped CLI that combines `jo`-style record construction +with safe append and rotation behaviour for one local JSONL file. + +## 4. Users and stakeholders + +| Type | Context | Cares about | Will dislike or ignore | Current alternative | +| ------------------------------------------ | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| Primary user: agent workflow author | Builds scripts, prompts, wrappers, or orchestration around multiple CLI-capable agents | One command that reliably records structured events under concurrency | Running a daemon, writing bespoke lock scripts, or debugging corrupted logs | `jo` plus shell redirection, ad hoc scripts, or no structured log | +| Primary user: autonomous agent | Invokes tools from a constrained shell environment during task execution | Simple argument contract, bounded waiting, machine-readable failure | Interactive prompts, hidden global state, or non-deterministic output formats | Direct file append or caller-specific helper | +| Primary user: df12-build workflow operator | Runs ODW workflows that plan, review, implement, merge, audit, and remediate roadmap work | Evidence for tuning parallelism, review caps, audit loops, and remediation lanes | Manual reconstruction from transcripts, branch history, or final summaries | Architectural guesses and scattered workflow output | +| Secondary user: maintainer | Implements, reviews, and releases the Rust CLI | Clear correctness requirements, testable failure modes, and stable docs | Broad observability scope or vague compatibility promises | Generated project scaffold and maintainer convention | +| Stakeholder: project sponsor | Wants agents to leave durable audit trails without coordination overhead | Fewer lost events, easier review of concurrent runs, low setup cost | Feature creep into a logging platform | Manual run notes or scattered per-agent logs | +| Non-user: observability platform operator | Runs centralized ingestion, querying, alerting, and retention infrastructure | Fleet-level logging, dashboards, and policy controls | A local-file-only CLI | Existing observability stack | + +Table 1: Stakeholder mapping for the initial product boundary. + +## 5. Job to be done + +When an agent workflow launches several independent command-line processes, the +workflow author wants each process to append a structured event to the same +local log file, so they can inspect the run later without reconstructing events +from scattered output. + +When a df12-build ODW run processes roadmap tasks through planning, review, +implementation, integration, and audit, the workflow operator wants agents to +append structured journal events into the workflow sidecar directory, so they +can tune workflow defaults from real-run telemetry rather than from design-time +intuition. + +The functional dimension is durable structured append under contention. The +emotional dimension is confidence that a missing or malformed log entry points +to a real failure rather than a race in the logging helper. The social +dimension is that maintainers can review an agent run from one readable +artefact rather than trusting a caller's informal summary. + +## 6. Scope + +### 6.1 Goals + +- Accept a log-file path as the first CLI argument and interpret following + arguments as record fields. +- Support the `jo`-style key/value syntax needed for object records, including + type coercion flags and object paths. +- Reject any invocation that would produce a non-object root. +- Add a default `timestamp` field using an invocation-time UTC timestamp unless + configuration or CLI input explicitly overrides it according to the final + precedence rules. +- Append exactly one valid JSON object followed by a newline for each successful + invocation. +- Prevent concurrent calls from overwriting existing logs or colliding with + each other while creating, appending to, or rotating the file. +- Create missing parent directories for the log-file path automatically when + filesystem permissions allow it. +- Use locking and safe writes with a five-second default timeout. +- Gracefully handle simultaneous attempts to create the log file and sidecar + coordination artefacts. +- Rotate by default after the active log reaches 1 MiB. +- Optionally rotate on hourly, daily, or weekly UTC time boundaries, with + interim size splits if the active log reaches the size threshold before the + next boundary. +- Compress rotated logs after the fourth rotation, subject to the final naming + and retention policy. +- Read a sidecar TOML file for rotation configuration, schema-guided type + coercion, and default field values. +- Surface failures through stable exit codes and diagnostics suitable for + non-interactive agents. +- Support journal records for the df12-build ODW workflow's phase timings, + review rounds, task-shape outcomes, CodeRabbit waits and 429s, audit yield, + remediation lane outcomes, and post-merge defect escape. + +### 6.2 Non-goals + +- Centralized log ingestion, search, dashboards, alerting, and retention policy + are out of scope; users needing those should ship the JSONL output into an + observability system. +- A long-running daemon or background service is out of scope; the product is a + short-lived CLI. +- Preserving chronological file order by timestamp is out of scope. The default + timestamp is captured at invocation time, so entries can appear out of order + when one caller waits behind another. +- Full `jo` feature parity is out of scope where it conflicts with the + object-root logging contract. +- Cross-host distributed locking is out of scope unless a later design + explicitly accepts the complexity and platform constraints. +- Querying, filtering, formatting, or editing historical log entries is out of + scope; readers can use existing JSONL and shell tooling. +- Log schema governance for an organization is out of scope. The sidecar schema + exists to coerce one tool's input, not to become a central event taxonomy. +- Computing dashboards, recommendations, or automatic tuning for df12-build is + out of scope. `mpsc-log` records the evidence; analysis can happen in later + tooling. +- Discovering the ODW workflow sidecar directory is out of scope unless a later + integration contract adds it. Callers pass the concrete journal path. + +## 7. Success criteria + +### 7.1 User-facing success + +- A workflow author can replace an ad hoc `jo >> file.jsonl` call with + `mpsc-log` without losing the ability to express nested object fields and + basic type coercions. +- Stress tests with many concurrent invocations produce the same number of + valid JSONL records as successful command exits. +- Simultaneous first writes to a missing log file create exactly one usable log + and do not truncate, replace, or interleave records. +- A first write to a log path in a missing directory tree creates the required + parent directories when permissions allow it. +- Rotation during concurrent writes leaves every successful entry in either the + active file or a rotated file. +- A df12-build run can record one journal entry per meaningful phase, review + round, implementation attempt, CodeRabbit attempt, integration result, audit + finding, remediation triage decision, and post-merge defect report. + +### 7.2 Operational success + +- Lock acquisition obeys the configured timeout and defaults to five seconds. +- Rotation defaults are predictable: 1 MiB active-file threshold and compression + after four rotations. +- A failed invocation leaves the previous log state readable and does not + produce partial JSON records. +- Diagnostics are useful to agents and humans: invalid arguments, timeout, + malformed sidecar configuration, and file-system failures are distinguishable. +- Journal writes add low enough overhead that agents can record telemetry at + phase and attempt boundaries without changing workflow scheduling decisions. + +### 7.3 Strategic success + +- The tool remains small enough for agents to call as a normal utility rather + than as an integration project. +- The terms of reference, design document, and user guide give future + contributors enough boundary information to reject logging-platform feature + requests cleanly. +- df12-build maintainers can compare observed phase durations, review-round + counts, CodeRabbit waits, audit yield, and defect escape against + `MAX_PARALLEL`, `MAX_DESIGN_ROUNDS`, and `MAX_REVIEW_ROUNDS`. + +## 8. Constraints and assumptions + +### 8.1 Hard constraints + +- The first CLI parameter is the log-file path. +- Later parameters are key/value words using the accepted `jo`-style syntax. +- The root record must be a JSON object. +- The default lock timeout is five seconds. +- The default rotation threshold is 1 MiB. +- The default scheduled rotation policy is `none`. +- Scheduled rotation modes are `hourly`, `daily`, and `weekly`; they use UTC + period boundaries and do not imply a `max_age` retention setting. +- Rotated logs are compressed after four rotations. +- The sidecar configuration file is TOML and derives its path from the log file + path by replacing the filename extension with `.toml`. +- The default record includes `timestamp` unless the final precedence rules say + otherwise. +- Missing parent directories for the log-file path are created automatically + when permissions allow it. +- The tool must tolerate concurrent writers and concurrent initial file + creation. +- The workflow sidecar directory is caller-owned. `mpsc-log` receives the + journal file path explicitly and must not assume a df12-build directory + layout. + +### 8.2 Assumptions + +- The initial target is a local filesystem with locking semantics that can + protect cooperating `mpsc-log` processes. If users rely on network + filesystems with weaker locking, the tool may need documented limitations or + a different coordination strategy. +- Callers can pass shell arguments without needing standard input as the primary + data channel. If ARG_MAX or large payloads become common, file-value support + and streaming input need sharper requirements. +- Agent callers can handle non-zero exit codes and diagnostic text. If a caller + cannot observe failures, reliable logging cannot be guaranteed from the CLI + alone. +- Directory creation can fail because of permissions, read-only filesystems, or + invalid paths. Those failures must leave any existing log readable and return + diagnostics that distinguish directory creation from record construction. +- Sidecar defaults are stable enough to read during each invocation. If callers + edit sidecar configuration while writes are running, the design must specify + whether configuration reads are locked with log writes. +- RFC 3339 UTC timestamps are acceptable for the default timestamp field. If a + later integration requires a different timestamp profile, that format becomes + an explicit compatibility requirement. +- The df12-build ODW workflow can call `mpsc-log` at phase and attempt + boundaries. If agents cannot call the CLI from those points, the first + integration will need a wrapper or workflow-level helper. +- A single workflow run can identify its events with stable run, task, phase, + agent, and attempt identifiers. If those identifiers are not available, later + analysis will be limited to aggregate timing and count data. + +### 8.3 Dependencies + +- The `jo` manual is the behavioural reference for compatible argument syntax, + type coercion flags, and object-path expectations.[^1] +- RFC 3339 is the timestamp reference for default `timestamp` values.[^2] +- The future technical design must choose Rust crates or platform APIs for JSON + serialization, TOML parsing, file locking, gzip compression, time handling, + and atomic file operations. + +## 9. Open questions + +| Question | Why it matters | Criteria for resolution | Suggested path | +| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- | +| Which exact subset of `jo` syntax is in scope? | `jo` includes arrays, file-value operators, duplicate-key behaviour, object paths, and coercion flags that may conflict with an object-root log contract. | The design records accepted, rejected, and modified syntax with examples. | Design spike against `jo` examples. | +| What are the sidecar precedence rules? | Defaults, schema coercion, CLI values, and `timestamp` overrides can conflict. | A precedence table defines every conflict outcome. | Technical design. | +| How should duplicate keys resolve? | JSON permits duplicate object names textually, but consumers often collapse them. | The product chooses reject, last-wins, first-wins, or compatibility behaviour. | ADR candidate. | +| What does "gzipping after 4 rotations" mean exactly? | Retention, naming, and compression timing affect concurrency and user expectations. | The rotation policy specifies filenames, retention count, compression trigger, and whether recent rotations remain plain text. | Technical design. | +| What lock protects rotation and sidecar reads? | Append-only locking may not be enough when a process rotates while others are opening or creating files. | The design names the lock artefact and the critical sections it protects. | Technical design and stress tests. | +| What platforms are supported at v1? | File-locking and atomic rename semantics differ across Unix, Windows, and network filesystems. | The project declares supported platforms and test coverage. | ADR candidate. | +| What are the stable exit codes? | Agents need to distinguish retryable timeout from invalid input or corrupted configuration. | The user guide lists exit codes and diagnostics. | User-guide update during implementation. | +| Is a Rust library API part of the product? | A public library API expands compatibility and documentation obligations. | The roadmap states whether the crate is CLI-only or also exposes supported library functions. | Product decision. | +| What is the df12-build event taxonomy? | The first use case needs comparable records for phase timings, task shape, review outcomes, CodeRabbit attempts, audit findings, remediation lanes, and defect escape. | The design defines stable event names, required fields, optional fields, and schema versions. | Technical design with workflow fixture. | +| How are workflow events correlated? | Without run, task, phase, agent, attempt, branch, and commit identifiers, telemetry cannot explain review loops, merge conflicts, or escaped defects. | The workflow integration names required correlation fields and their source. | Integration design. | + +## 10. Handoff + +### 10.1 Context additions + +The [context](context.md) document should continue to define the shared +vocabulary for: + +- Agent workflow. +- Audit yield. +- CodeRabbit attempt. +- Defect escape. +- df12-build. +- Open Dynamic Workflows (ODW). +- JSON Lines (JSONL). +- Journal. +- Record. +- Remediation lane. +- Review round. +- Sidecar configuration. +- Workflow sidecar directory. +- Rotation. +- Active log. +- Rotated log. +- Type coercion. +- Object path. +- Lock timeout. +- Scheduled break. + +### 10.2 ADR candidates + +- Accepted `jo` compatibility subset and deviations. +- Locking and atomic-write strategy across supported platforms. +- Rotation naming, retention, and compression policy. +- CLI-only product boundary versus supported library API. + +### 10.3 Downstream readiness + +This document is complete enough to start a technical design. The design should +not begin implementation until the timestamp standard, `jo` compatibility +subset, sidecar precedence, rotation policy, and platform support questions are +resolved or explicitly deferred. + +## Appendix A. References + +[^1]: Jan-Piet Mens, + [`jo` manual](https://github.com/jpmens/jo/blob/master/jo.md), + accessed 2026-06-29. + +[^2]: G. Klyne and C. Newman, [RFC 3339: Date and time on the Internet: + timestamps](https://www.rfc-editor.org/rfc/rfc3339), July 2002, accessed + 2026-06-29. From 5774d4b4eada4fbe8502310cbde65a0b4f19245c Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 17:52:43 +0100 Subject: [PATCH 02/27] Record testing strategy ADR Add ADR 001 to define how the repository's required testing prongs apply to `mpsc-log`'s CLI, record-building, filesystem, concurrency, repair, rotation, and telemetry contracts. Reference the ADR from the technical design, documentation index, and roadmap so implementation tasks inherit the testing strategy rather than treating it as an isolated testing phase. --- docs/adr-001-testing-strategy.md | 215 +++++++++++++++++++++++++++++++ docs/contents.md | 2 + docs/mpsc-log-design.md | 6 + docs/roadmap.md | 15 ++- 4 files changed, 232 insertions(+), 6 deletions(-) create mode 100644 docs/adr-001-testing-strategy.md diff --git a/docs/adr-001-testing-strategy.md b/docs/adr-001-testing-strategy.md new file mode 100644 index 0000000..f90e372 --- /dev/null +++ b/docs/adr-001-testing-strategy.md @@ -0,0 +1,215 @@ +# Architectural decision record (ADR) 001: Testing strategy + +## Status + +Accepted on 2026-06-29. The project will apply the testing prongs mandated in +`AGENTS.md` as a layered strategy tied to the `mpsc-log` design's contracts, +not as a separate testing phase. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` is a short-lived Rust CLI that appends one JSON object to a shared +JSON Lines journal while handling sidecar configuration, type coercion, +locking, repair, rotation, compression, and bounded lock waiting. The design's +highest-risk behaviours are externally observable: concurrent writers must not +lose records, rotation must preserve every successful entry, partial tails must +be repaired, and diagnostics must let non-interactive agents distinguish usage, +data, configuration, I/O, and timeout failures. + +`AGENTS.md` mandates several testing prongs for this repository: unit tests, +behavioural tests, snapshots, end-to-end tests, property tests, bounded model +checking, and Verus proofs where contractual business logic warrants exhaustive +proof. The project needs one decision record explaining how those prongs apply +to the proposed design so implementers do not either under-test concurrency and +persistence or create standalone testing theatre that does not improve +confidence. + +## Decision drivers + +- The CLI's main promise is durable append under contention, so process-level + and filesystem-level evidence matter more than line coverage alone. +- The design already separates pure domain logic from clock and filesystem + adapters, which should make deterministic and fault-injection tests practical. +- `AGENTS.md` requires `rstest`, `rstest-bdd` where applicable, `insta` where + output variants matter, end-to-end tests for observable workflows, and + property-based or formal methods for invariants over ranges of states. +- Tests must remain review-sized with development tasks. Unit and behavioural + tests belong with their implementation work; only cross-feature end-to-end, + combinatorial, or formal hardening suites should stand alone. +- Direct environment mutation in tests is forbidden unless guarded through + shared helpers as a last resort. The current design should avoid relying on + mutable process environment state. + +## Requirements + +### Functional requirements + +- Verify that each successful invocation appends exactly one complete JSON + object line. +- Verify that concurrent successful invocations produce the same number of + complete records as successful process exits. +- Verify that size-only rotation, scheduled rotation, gzip compression, and + retention preserve successful records. +- Verify that malformed tails and injected write failures do not leave partial + final records after the next invocation. +- Verify the accepted `jo` subset, object-root enforcement, sidecar precedence, + schema-guided coercion, explicit coercion flags, default timestamp insertion, + and stable exit-code mapping. +- Verify df12-build fixture records against the JSON Schema and sidecar + coercion contract. + +### Technical requirements + +- Use `rstest` fixtures and parameterized cases for unit and integration tests. +- Use `rstest-bdd` for externally observable CLI scenarios where the + Given/When/Then shape clarifies user-facing behaviour. +- Use `insta` snapshots only for stable, reviewer-useful output boundaries, + pairing snapshots with semantic assertions and normalizing nondeterministic + fields. +- Use process-level end-to-end tests for CLI workflows, concurrent appends, + rotation, and df12-build fixture invocations. +- Use `proptest` for generated field words, object-path merges, sidecar/CLI + precedence, timestamp override cases, tail-repair inputs, and rotation-plan + invariants. +- Use `kani` for bounded state-machine checks when a pure planner has a compact + state space, such as generation shifting, scheduled period grouping, or + append/repair transitions. +- Use Verus only for introduced lemmas or contractual business logic whose + safety property is important enough to justify proof maintenance. +- Keep filesystem, time, and failure injection behind explicit adapters rather + than mutating global state. + +## Options considered + +### Option A: Unit and CLI smoke tests only + +This option would test parsers, record construction, and a few happy-path CLI +commands. It is fast and easy to maintain, but it does not prove the +concurrency, repair, or rotation promises that justify the tool. + +### Option B: Apply every testing tool to every feature + +This option would require unit, behavioural, snapshot, end-to-end, +property-based, model-checking, and proof coverage for every change. It is +maximally strict, but it would slow review, produce weak tests for features +that do not benefit from a given prong, and make formal tools feel ceremonial. + +### Option C: Risk-based layered strategy + +This option maps each testing prong to the design surface it can validate best. +Pure logic gets unit and property tests. User-facing workflows get behavioural +and end-to-end tests. Stable text or file-shape contracts get narrow snapshots. +Concurrency, rotation, and repair get stress, fault-injection, and model-based +coverage. Formal proof is reserved for compact contractual invariants. + +| Topic | Option A | Option B | Option C | +| ----- | -------- | -------- | -------- | +| Concurrency confidence | Weak | Strong but costly | Strong where required | +| Review size | Small | Often too large | Review-sized by risk | +| Formal methods | Absent | Over-applied | Applied to compact invariants | +| Alignment with `AGENTS.md` | Incomplete | Literal but inefficient | Complete and targeted | +| Maintenance cost | Low initially | High | Proportional to risk | + +_Table 1: Comparison of testing strategy options._ + +## Decision outcome / proposed direction + +Adopt Option C, the risk-based layered testing strategy. + +Every implementation task must include the unit, behavioural, property, or +snapshot coverage needed for the behaviour it introduces. Dedicated roadmap +tasks remain appropriate for cross-feature evidence that exceeds one pull +request, such as multi-process stress testing, pairwise CLI/configuration +coverage, concurrent rotation end-to-end tests, and broader formal hardening of +shared planners. + +The testing prongs apply as follows: + +- Unit tests with `rstest` cover pure domain logic in `args`, `fields`, + `config`, `record`, `errors`, `clock`, and rotation-planning helpers. +- Behavioural tests with `rstest-bdd` cover user-visible CLI behaviours: + successful append, unsupported syntax, sidecar precedence, diagnostics, + timeout, repair, and rotation scenarios. +- Snapshot tests with `insta` cover stable outputs that reviewers benefit from + seeing as artefacts: help text, one-line diagnostics, canonical fixture + records after nondeterministic normalization, and rotation filename matrices. +- End-to-end tests cover real binary invocations against temporary directories, + including parent-directory creation, simultaneous first writes, concurrent + appends, forced size rotation, scheduled rotation, compression, and + df12-build fixture commands. +- Property tests with `proptest` cover input and state ranges for field + parsing, object-path insertion, type coercion, merge precedence, partial-tail + repair, and rotation-plan invariants. +- Bounded model checks with `kani` cover compact pure planners where exhaustive + small-state exploration is more useful than randomized testing, especially + generation shifting and scheduled-period retention. +- Verus proofs are reserved for explicit lemmas or contractual business logic + introduced during implementation. They are not required for ordinary glue + code, parser plumbing, or tests that can be more clearly expressed with + `rstest`, `proptest`, or `kani`. + +## Goals and non-goals + +- Goals: + - Tie each required testing prong to a concrete `mpsc-log` design surface. + - Keep tests deterministic through clock and filesystem injection. + - Make concurrency, rotation, repair, and failure-mode evidence release + blockers. + - Keep implementation tasks review-sized by embedding ordinary tests in the + task that introduces the behaviour. +- Non-goals: + - Mandate every testing tool for every implementation task. + - Treat coverage percentage as a substitute for contract evidence. + - Prove network-filesystem correctness before the design accepts that + platform promise. + - Add `max_age` retention tests while `max_age` remains out of scope. + +## Migration plan + +1. Add test support before the first feature slice lands. + - Create shared `rstest` fixtures for temporary directories, fixed clocks, + filesystem adapters, command invocation, and decoded JSONL records. + - Add helper modules under `tests/` or `src/` behind `#[cfg(test)]` where + they preserve ownership boundaries. +2. Add domain tests with each implementation module. + - Pair parser, sidecar, record, error, and rotation-planner work with + `rstest` cases and `proptest` strategies for their accepted input spaces. +3. Add behavioural and end-to-end suites once the CLI can append a record. + - Use `rstest-bdd` for user-facing scenarios and process-level tests for + real binary execution. +4. Add concurrency, repair, and rotation hardening as the filesystem adapter + lands. + - Exercise multi-process contention, lock timeout, partial-tail repair, + injected I/O failures, size rotation, scheduled rotation, gzip, and + retention. +5. Add model checking or proofs only when the implementation introduces a pure + invariant-bearing planner or lemma. + - Prefer `kani` for bounded state machines and Verus for explicit lemmas + whose proof is clearer than a large generated test matrix. + +## Known risks and limitations + +- Multi-process stress tests can be timing-sensitive. They must assert durable + outcomes, not exact scheduling order. +- Snapshot tests can become brittle if they capture broad objects or raw + timestamps. They must stay narrow and normalize nondeterministic fields. +- Formal tools add maintenance cost. They should protect compact invariants, + not duplicate straightforward example tests. +- The initial correctness claim remains limited to local filesystems until a + separate platform verification matrix names and validates other filesystem + behaviours. + +## Architectural rationale + +The chosen strategy follows the design's architecture. Pure record-building and +rotation-planning code can be tested exhaustively and deterministically. +Filesystem effects pass through adapters, allowing fault injection without +global state mutation. User-facing CLI behaviour is verified at the process +boundary because agents experience the tool through exit codes, diagnostics, +and files on disk. The result keeps the repository's mandated testing prongs +connected to real product risk instead of distributing them mechanically across +the codebase. diff --git a/docs/contents.md b/docs/contents.md index 83e4c25..5b0f180 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -45,6 +45,8 @@ set. ## Design artefacts +- [ADR 001: Testing strategy](adr-001-testing-strategy.md) records how the + repository's required testing prongs apply to the `mpsc-log` design. - [mpsc-log event schema](mpsc-log-event-schema.json) defines the initial JSON Schema for journal records. - [mpsc-log sidecar example](mpsc-log-sidecar.example.toml) shows the TOML diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index eb27e63..a9cef73 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -6,6 +6,7 @@ - **Companion documents:** [Terms of reference](terms-of-reference.md), [context](context.md), [event schema](mpsc-log-event-schema.json), [sidecar example](mpsc-log-sidecar.example.toml), + [testing strategy ADR](adr-001-testing-strategy.md), [users' guide](users-guide.md), and [developer guide](developers-guide.md). ## 1. Context @@ -346,6 +347,11 @@ The command writes nothing to standard output on success. ## 11. Correctness properties and verification +The testing strategy is recorded in +[ADR 001: Testing strategy](adr-001-testing-strategy.md). That ADR maps the +repository's required unit, behavioural, snapshot, end-to-end, +property-based, bounded-model, and proof prongs to the design surfaces below. + The implementation must satisfy these properties: | Property | Verification method | diff --git a/docs/roadmap.md b/docs/roadmap.md index c8f6d47..59bcb79 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -8,9 +8,11 @@ dependencies and source citations. The primary source documents are [terms of reference](terms-of-reference.md), [technical design](mpsc-log-design.md), [context](context.md), -[event schema](mpsc-log-event-schema.json), and -[sidecar example](mpsc-log-sidecar.example.toml). No RFCs or ADRs exist yet, -so the first phase records the decisions that would otherwise force rework. +[event schema](mpsc-log-event-schema.json), +[sidecar example](mpsc-log-sidecar.example.toml), and +[ADR 001: Testing strategy](adr-001-testing-strategy.md). No RFCs exist yet, +so the first phase records the remaining decisions that would otherwise force +rework. ## 1. Foundational contracts and build spine @@ -81,7 +83,8 @@ same CLI, domain, filesystem, and test seams. See mpsc-log-design.md §§2, 4, - [ ] 1.2.3. Build deterministic test seams for time and filesystem effects. - Requires 1.2.1 and 1.2.2. - See mpsc-log-design.md §§7, 10-12 and - docs/reliable-testing-in-rust-via-dependency-injection.md. + docs/reliable-testing-in-rust-via-dependency-injection.md. See + adr-001-testing-strategy.md. - [ ] Provide injectable clock and filesystem adapter boundaries for record timestamps, fault injection, and deterministic rotation fixtures. - Success: failures such as partial writes, metadata errors, and fixed @@ -259,7 +262,7 @@ and terms-of-reference.md §§5, 7. decodable JSON object lines as successful child processes. - [ ] 3.3.2. Add pairwise CLI/configuration combination coverage. - Requires phase 2 and 3.3.1. - - See mpsc-log-design.md §11. + - See mpsc-log-design.md §11 and adr-001-testing-strategy.md. - Success: the suite covers `jo` syntax form, coercion source, object path, sidecar default, rotation schedule, rotation state, and lock contention. @@ -356,7 +359,7 @@ mpsc-log-design.md §11 and terms-of-reference.md §7. - [ ] 4.4.1. Build the concurrent rotation end-to-end suite. - Requires steps 4.1-4.3. - - See mpsc-log-design.md §11. + - See mpsc-log-design.md §11 and adr-001-testing-strategy.md. - Success: forced size and scheduled rotations under concurrent writers preserve the decoded record count across active, plain, scheduled, and compressed files. From 5df6d02c556bed254655d0998bc7502e0771ef27 Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 17:56:30 +0100 Subject: [PATCH 03/27] Apply repository formatting Run `make fmt` across the repository and keep the resulting Markdown wrapping changes. Reshape the long companion-document link lists so the formatter target can complete without reintroducing markdownlint line-length failures. --- docs/adr-001-testing-strategy.md | 14 ++++---- docs/mpsc-log-design.md | 17 +++++---- docs/roadmap.md | 61 +++++++++++++++++--------------- docs/terms-of-reference.md | 17 +++++---- 4 files changed, 59 insertions(+), 50 deletions(-) diff --git a/docs/adr-001-testing-strategy.md b/docs/adr-001-testing-strategy.md index f90e372..45593c0 100644 --- a/docs/adr-001-testing-strategy.md +++ b/docs/adr-001-testing-strategy.md @@ -106,13 +106,13 @@ and end-to-end tests. Stable text or file-shape contracts get narrow snapshots. Concurrency, rotation, and repair get stress, fault-injection, and model-based coverage. Formal proof is reserved for compact contractual invariants. -| Topic | Option A | Option B | Option C | -| ----- | -------- | -------- | -------- | -| Concurrency confidence | Weak | Strong but costly | Strong where required | -| Review size | Small | Often too large | Review-sized by risk | -| Formal methods | Absent | Over-applied | Applied to compact invariants | -| Alignment with `AGENTS.md` | Incomplete | Literal but inefficient | Complete and targeted | -| Maintenance cost | Low initially | High | Proportional to risk | +| Topic | Option A | Option B | Option C | +| -------------------------- | ------------- | ----------------------- | ----------------------------- | +| Concurrency confidence | Weak | Strong but costly | Strong where required | +| Review size | Small | Often too large | Review-sized by risk | +| Formal methods | Absent | Over-applied | Applied to compact invariants | +| Alignment with `AGENTS.md` | Incomplete | Literal but inefficient | Complete and targeted | +| Maintenance cost | Low initially | High | Proportional to risk | _Table 1: Comparison of testing strategy options._ diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index a9cef73..6306fd5 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -3,11 +3,14 @@ - **Status:** Draft v0.1. - **Audience:** Implementers, reviewers, and workflow operators. - **Last substantive revision:** 2026-06-29. -- **Companion documents:** [Terms of reference](terms-of-reference.md), - [context](context.md), [event schema](mpsc-log-event-schema.json), - [sidecar example](mpsc-log-sidecar.example.toml), - [testing strategy ADR](adr-001-testing-strategy.md), - [users' guide](users-guide.md), and [developer guide](developers-guide.md). +- **Companion documents:** + - [Terms of reference](terms-of-reference.md). + - [Context](context.md). + - [Event schema](mpsc-log-event-schema.json). + - [Sidecar example](mpsc-log-sidecar.example.toml). + - [Testing strategy ADR](adr-001-testing-strategy.md). + - [Users' guide](users-guide.md). + - [Developer guide](developers-guide.md). ## 1. Context @@ -349,8 +352,8 @@ The command writes nothing to standard output on success. The testing strategy is recorded in [ADR 001: Testing strategy](adr-001-testing-strategy.md). That ADR maps the -repository's required unit, behavioural, snapshot, end-to-end, -property-based, bounded-model, and proof prongs to the design surfaces below. +repository's required unit, behavioural, snapshot, end-to-end, property-based, +bounded-model, and proof prongs to the design surfaces below. The implementation must satisfy these properties: diff --git a/docs/roadmap.md b/docs/roadmap.md index 59bcb79..245364f 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -6,13 +6,17 @@ Each phase carries one testable idea at the GIST level; each step answers a sequencing question; each task is a review-sized execution unit with explicit dependencies and source citations. -The primary source documents are [terms of reference](terms-of-reference.md), -[technical design](mpsc-log-design.md), [context](context.md), -[event schema](mpsc-log-event-schema.json), -[sidecar example](mpsc-log-sidecar.example.toml), and -[ADR 001: Testing strategy](adr-001-testing-strategy.md). No RFCs exist yet, -so the first phase records the remaining decisions that would otherwise force -rework. +The primary source documents are: + +- [terms of reference](terms-of-reference.md); +- [technical design](mpsc-log-design.md); +- [context](context.md); +- [event schema](mpsc-log-event-schema.json); +- [sidecar example](mpsc-log-sidecar.example.toml); and +- [ADR 001: Testing strategy](adr-001-testing-strategy.md). + +No RFCs exist yet, so the first phase records the remaining decisions that +would otherwise force rework. ## 1. Foundational contracts and build spine @@ -29,8 +33,8 @@ risks are concurrency and persistence. This step answers what `mpsc-log` v1 will promise to callers and what it will explicitly leave out. Its outcome informs crate layout, user documentation, -error handling, and future compatibility work. See -mpsc-log-design.md §§3, 5, 8, 12-13 and terms-of-reference.md §§6, 8-10. +error handling, and future compatibility work. See mpsc-log-design.md §§3, 5, +8, 12-13 and terms-of-reference.md §§6, 8-10. - [ ] 1.1.1. Record the accepted `jo` subset and duplicate-key behaviour in an ADR. @@ -105,8 +109,8 @@ examples before the filesystem protocol grows more complex. This step answers whether the accepted `jo` subset can be implemented without breaking the object-root logging contract. It informs sidecar coercion, error -classification, and the later combinatorial test matrix. See -mpsc-log-design.md §§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. +classification, and the later combinatorial test matrix. See mpsc-log-design.md +§§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. - [ ] 2.1.1. Implement positional argument parsing for the journal path and raw field tail. @@ -138,9 +142,8 @@ mpsc-log-design.md §§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. This step answers whether configuration, schema coercion, defaults, and the generated timestamp can merge deterministically. Its outcome informs the -external JSONL contract and df12-build event examples. See -mpsc-log-design.md §§2, 6, 9, 11-12, -mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. +external JSONL contract and df12-build event examples. See mpsc-log-design.md +§§2, 6, 9, 11-12, mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. - [ ] 2.2.1. Implement sidecar path derivation, TOML loading, and semantic validation. @@ -166,8 +169,8 @@ mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. This step answers whether the command can complete a non-concurrent append workflow with stable user-visible behaviour. It informs the later lock and -repair protocol because this slice defines the line format and diagnostics. -See mpsc-log-design.md §§3, 7, 10-12 and terms-of-reference.md §§5-7. +repair protocol because this slice defines the line format and diagnostics. See +mpsc-log-design.md §§3, 7, 10-12 and terms-of-reference.md §§5-7. - [ ] 2.3.1. Implement compact JSON object serialization and newline append. - Requires steps 2.1-2.2. @@ -195,15 +198,15 @@ workflows can treat journal writes as boring infrastructure rather than a source of telemetry loss. This phase turns the single-writer command into the product promised by the -terms of reference. It focuses on the one risk that motivates the tool: -several independent agents calling the same executable at the same time. +terms of reference. It focuses on the one risk that motivates the tool: several +independent agents calling the same executable at the same time. ### 3.1. Prove first-write and lock acquisition are safe This step answers whether concurrent invocations can create the same journal and coordination artefacts without truncating or colliding. It informs the -critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, -7, 10-11 and terms-of-reference.md §§6-8. +critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, 7, +10-11 and terms-of-reference.md §§6-8. - [ ] 3.1.1. Create missing parent directories and coordination artefacts before locking. @@ -252,8 +255,8 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md This step answers whether many real processes can use `mpsc-log` at once and produce one complete record per successful command. It informs release -readiness for the first agent-workflow adoption. See mpsc-log-design.md §11 -and terms-of-reference.md §§5, 7. +readiness for the first agent-workflow adoption. See mpsc-log-design.md §11 and +terms-of-reference.md §§5, 7. - [ ] 3.3.1. Build a multi-process concurrent append stress harness. - Requires steps 3.1-3.2. @@ -324,8 +327,8 @@ mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. This step answers whether hourly, daily, and weekly modes rotate at period boundaries while still splitting busy periods by bytes. It informs the final operator contract because scheduled rotation is where naming expectations are -most visible. See mpsc-log-design.md §§6, 8, 11, -mpsc-log-sidecar.example.toml, and context.md. +most visible. See mpsc-log-design.md §§6, 8, 11, mpsc-log-sidecar.example.toml, +and context.md. - [ ] 4.3.1. Implement period calculation for `hourly`, `daily`, and `weekly` schedules. @@ -372,9 +375,9 @@ mpsc-log-design.md §11 and terms-of-reference.md §7. ## 5. df12-build telemetry adoption Idea: if the finished CLI can emit the first df12-build journal taxonomy with -documented sidecar defaults and realistic invocation fixtures, the workflow -can start collecting real-run evidence without `mpsc-log` becoming an analysis -or dashboard product. +documented sidecar defaults and realistic invocation fixtures, the workflow can +start collecting real-run evidence without `mpsc-log` becoming an analysis or +dashboard product. This phase connects the generic journalling tool to its first concrete use case. It packages examples and verification around phase timing, review rounds, @@ -412,8 +415,8 @@ terms-of-reference.md §§2, 5-7. This step answers whether workflow authors and operators can use the tool correctly without reading implementation code. It informs v1 release readiness -and keeps deferred analysis work out of the core CLI. See -terms-of-reference.md §§3-7 and mpsc-log-design.md §§3, 9-10. +and keeps deferred analysis work out of the core CLI. See terms-of-reference.md +§§3-7 and mpsc-log-design.md §§3, 9-10. - [ ] 5.2.1. Update the users' guide with complete operational examples. - Requires 5.1.3. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index 2fad8be..ec09680 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -4,13 +4,16 @@ - **Audience:** Product owner, engineering maintainers, agent-workflow authors, and future design reviewers. - **Last substantive revision:** 2026-06-29. -- **Companion documents:** [Documentation contents](contents.md), - [users' guide](users-guide.md), [developer guide](developers-guide.md), - [repository layout](repository-layout.md), - [documentation style guide](documentation-style-guide.md), - [context](context.md), [technical design](mpsc-log-design.md), - [event schema](mpsc-log-event-schema.json), - and [sidecar example](mpsc-log-sidecar.example.toml). +- **Companion documents:** + - [Documentation contents](contents.md). + - [Users' guide](users-guide.md). + - [Developer guide](developers-guide.md). + - [Repository layout](repository-layout.md). + - [Documentation style guide](documentation-style-guide.md). + - [Context](context.md). + - [Technical design](mpsc-log-design.md). + - [Event schema](mpsc-log-event-schema.json). + - [Sidecar example](mpsc-log-sidecar.example.toml). ## 1. Background and motivation From a9cc137dc46ad3ebce350d8983fbbe85526c4ca7 Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 18:00:19 +0100 Subject: [PATCH 04/27] Clarify design review contracts Resolve the still-valid review findings in the design documentation. Spell out record merge and coercion precedence, restrict schema timestamps to canonical UTC form, align sidecar schema examples with integer event fields, and tighten roadmap success criteria for merge precedence and df12-build fixture coverage. --- docs/mpsc-log-design.md | 40 ++++++++++++++++-------------- docs/mpsc-log-event-schema.json | 3 ++- docs/mpsc-log-sidecar.example.toml | 16 ++++++------ docs/roadmap.md | 11 +++++--- 4 files changed, 40 insertions(+), 30 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 6306fd5..6fbfc7c 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -166,24 +166,28 @@ The sidecar shape is defined by [mpsc-log-sidecar.example.toml](mpsc-log-sidecar.example.toml). Configuration has four tables: -| Table | Responsibility | -| ------------ | ---------------------------------------------------------------------------------------------- | -| `[rotation]` | `schedule`, `max_bytes`, plain generation count, compressed generation count, and gzip policy. | -| `[locking]` | Lock timeout and partial-tail repair mode. | -| `[defaults]` | Default JSON fields inserted before CLI fields. | -| `[schema]` | Object paths mapped to coercion names: `string`, `number`, `boolean`, `json`, or `null`. | - -Merge order is deterministic: - -1. Start with sidecar `[defaults]`. -2. Apply CLI fields in argument order. -3. Insert generated `timestamp` if no field named `timestamp` exists. -4. Serialize the resulting object. - -Explicit `-s`, `-n`, and `-b` flags override the sidecar schema for that word. -When no flag or schema entry exists, default `jo` inference applies: valid JSON -values parse as JSON, empty `key=` becomes `null`, and other values remain -strings. +| Table | Responsibility | +| ------------ | --------------------------------------------------------------------------------------------------- | +| `[rotation]` | `schedule`, `max_bytes`, plain generation count, compressed generation count, and gzip policy. | +| `[locking]` | Lock timeout and partial-tail repair mode. | +| `[defaults]` | Default JSON fields inserted before CLI fields. | +| `[schema]` | Object paths mapped to coercion names: `string`, `integer`, `number`, `boolean`, `json`, or `null`. | + +Merge and coercion order is deterministic: + +| Step | Rule | Winner | +| ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | +| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | +| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo` inference wins. | +| 4 | Insert the generated invocation timestamp. | The generated canonical UTC `timestamp` is added only when no `timestamp` field exists after defaults and CLI fields. | +| 5 | Serialize the resulting object. | The merged record is written as one compact JSON object. | + +The sidecar schema never overrides an explicit CLI coercion flag. Schema +entries affect only values supplied for the matching object path; they do not +change sidecar defaults or the generated timestamp. When default `jo` inference +is used, valid JSON values parse as JSON, empty `key=` becomes `null`, and +other values remain strings. ## 7. Journal write protocol diff --git a/docs/mpsc-log-event-schema.json b/docs/mpsc-log-event-schema.json index 7c05071..58c0034 100644 --- a/docs/mpsc-log-event-schema.json +++ b/docs/mpsc-log-event-schema.json @@ -9,7 +9,8 @@ "timestamp": { "type": "string", "format": "date-time", - "description": "RFC 3339 UTC invocation timestamp." + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):[0-5][0-9]:([0-5][0-9]|60)(\\.[0-9]{1,9})?Z$", + "description": "Canonical RFC 3339 UTC invocation timestamp using a Z suffix." }, "schema_version": { "type": "integer", diff --git a/docs/mpsc-log-sidecar.example.toml b/docs/mpsc-log-sidecar.example.toml index 09936ce..7c41c1f 100644 --- a/docs/mpsc-log-sidecar.example.toml +++ b/docs/mpsc-log-sidecar.example.toml @@ -16,13 +16,13 @@ event = "workflow.event" [schema] "run.id" = "string" "run.workflow" = "string" -"task.work_items" = "number" -"task.changed_files" = "number" -"task.dependency_depth" = "number" -"attempt.round" = "number" -"attempt.duration_ms" = "number" -"coderabbit.http_status" = "number" -"coderabbit.wait_ms" = "number" +"task.work_items" = "integer" +"task.changed_files" = "integer" +"task.dependency_depth" = "integer" +"attempt.round" = "integer" +"attempt.duration_ms" = "integer" +"coderabbit.http_status" = "integer" +"coderabbit.wait_ms" = "integer" "coderabbit.retry" = "boolean" "coderabbit.deferred" = "boolean" -"audit.finding_count" = "number" +"audit.finding_count" = "integer" diff --git a/docs/roadmap.md b/docs/roadmap.md index 245364f..fbce7cd 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -157,8 +157,11 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md - Requires 2.1.3 and 2.2.1. - See mpsc-log-design.md §§6, 9, 11 and mpsc-log-event-schema.json. - - Success: sidecar defaults, CLI fields, explicit flags, schema entries, - and duplicate paths resolve according to one table-driven contract. + - Success: sidecar defaults seed the record; CLI fields win over defaults + and earlier duplicate paths; CLI coercion uses explicit `-s`, `-n`, or + `-b` flags before schema entries and default `jo` inference; and the + generated `timestamp` is inserted only if no timestamp exists after + defaults and CLI fields. - [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. - Requires 1.2.3 and 2.2.2. - See mpsc-log-design.md §§2, 6 and terms-of-reference.md §§2, 6, 8. @@ -397,7 +400,9 @@ terms-of-reference.md §§2, 5-7. - See mpsc-log-design.md §9 and terms-of-reference.md §§2, 5-7. - Success: fixtures cover `phase.started`, `phase.finished`, `review.round`, `task.finished`, `coderabbit.attempt`, `audit.finding`, - and `defect.escape`. + and `defect.escape`, including CodeRabbit wait/throttle variants, audit + yield by severity and lane, remediation-lane outcomes, and post-merge + defect-escape cases. - [ ] 5.1.2. Validate fixture output against the JSON Schema contract. - Requires 5.1.1. - See mpsc-log-design.md §9 and mpsc-log-event-schema.json. From dc02c9463d271b167d567ec711c2195cb2229a64 Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 18:11:21 +0100 Subject: [PATCH 05/27] Promote lock naming ADR Record lock file naming as ADR 001 because concurrency and rotation safety depend on every writer deriving the same coordination artefact. Rename the testing strategy ADR to ADR 002, update the design and roadmap references, and mark the lock naming roadmap task as completed. --- docs/adr-001-lock-file-naming.md | 201 ++++++++++++++++++ ...trategy.md => adr-002-testing-strategy.md} | 2 +- docs/contents.md | 5 +- docs/mpsc-log-design.md | 24 ++- docs/roadmap.md | 26 +-- 5 files changed, 235 insertions(+), 23 deletions(-) create mode 100644 docs/adr-001-lock-file-naming.md rename docs/{adr-001-testing-strategy.md => adr-002-testing-strategy.md} (99%) diff --git a/docs/adr-001-lock-file-naming.md b/docs/adr-001-lock-file-naming.md new file mode 100644 index 0000000..2933da1 --- /dev/null +++ b/docs/adr-001-lock-file-naming.md @@ -0,0 +1,201 @@ +# Architectural decision record (ADR) 001: Lock file naming + +## Status + +Accepted on 2026-06-29. `mpsc-log` reserves coordination filenames and derives +one journal lock by appending `.lock` to the complete journal filename in the +same directory. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` serializes repair, sidecar reads, rotation, compression, and append +through one advisory lock per journal. Every safety claim in the design depends +on all cooperating invocations choosing the same lock for the same journal and +choosing different locks for different journals. + +The obvious rule, `.lock` in the same directory, is simple +but needs collision boundaries. Names such as `run`, `run.jsonl`, +`run.jsonl.lock`, and sidecar paths derived by extension replacement can +otherwise create hidden ambiguity: + +- `run` should not silently collide with `run.jsonl`; +- `run.jsonl.lock` is naturally the lock path for `run.jsonl`, but it could + also be supplied as a journal path unless reserved; +- `run.toml` can be both a journal path and the extension-replaced sidecar path + for `run` unless `.toml` journal paths are rejected; +- journals that share a stem, such as `run` and `run.jsonl`, share the sidecar + `run.toml` under the design's extension-replacement rule. + +The project needs this decision before implementing filesystem safety, because +lock naming, sidecar derivation, and reserved suffixes define what the writer +can safely create under contention. + +## Decision drivers + +- Lock naming must be deterministic from the caller-provided journal path. +- The lock path must not use extension replacement; otherwise `run` and + `run.jsonl` can become ambiguous. +- Coordination artefacts must be reserved so callers cannot accidentally append + JSONL records into another journal's lock file. +- The sidecar derivation rule is already part of the design and should remain + easy to explain. +- The rule must work before the journal exists and while several first writers + create parent directories concurrently. + +## Requirements + +### Functional requirements + +- Derive one lock path for each accepted journal path. +- Ensure accepted journal paths cannot name another journal's lock file. +- Ensure accepted journal paths cannot equal their own sidecar path. +- Keep `run` and `run.jsonl` lock paths distinct. +- Keep the sidecar extension-replacement rule explicit when multiple journal + names share one stem. + +### Technical requirements + +- Create parent directories before opening the lock file. +- Open the lock file with create semantics and acquire the exclusive lock before + reading sidecar configuration, repairing tails, rotating, compressing, or + appending. +- Treat `.lock` journal paths as invalid because `.lock` is reserved for + coordination artefacts. +- Treat `.toml` journal paths as invalid when the derived sidecar path would be + the journal path itself. +- Keep lock naming local to the journal directory; do not introduce a global + lock directory or daemon. + +## Options considered + +### Option A: Append `.lock` to the full journal filename and reserve suffixes + +For an accepted journal path, derive the lock path by appending `.lock` to the +complete filename in the same directory. Reject journal filenames ending in +`.lock`, and reject journal filenames whose derived sidecar path equals the +journal path. + +Examples: + +| Journal path | Lock path | Sidecar path | Accepted | +| ---------------- | ----------------- | ------------ | ------------------------------- | +| `run` | `run.lock` | `run.toml` | Yes | +| `run.jsonl` | `run.jsonl.lock` | `run.toml` | Yes | +| `run.ndjson` | `run.ndjson.lock` | `run.toml` | Yes | +| `run.jsonl.lock` | None | None | No, `.lock` is reserved | +| `run.toml` | None | None | No, sidecar would equal journal | + +_Table 1: Filename outcomes for the accepted naming policy._ + +This option keeps lock derivation simple, avoids lock collisions between +extension variants, and makes reserved coordination/configuration names +explicit. + +### Option B: Replace the journal extension with `.lock` + +This option would make `run.jsonl` use `run.lock`. It is short, but it collides +with the lock path for `run` and repeats the ambiguity already present in +extension-replaced sidecar names. + +### Option C: Use a hidden lock filename + +This option would use a name such as `.run.jsonl.lock` in the same directory. +It avoids some visible filename clutter, but it does not remove the need for +reserved names, makes operator inspection harder, and adds platform-specific +hidden-file expectations without improving the safety claim. + +### Option D: Use a central lock directory + +This option would hash canonical journal paths into a shared lock directory. It +can avoid adjacent artefacts, but it introduces global state, path canonicalize +questions, lifecycle cleanup, and permission behaviour that are +disproportionate for a local CLI. + +| Topic | Option A | Option B | Option C | Option D | +| ---------------------------------- | -------- | -------- | -------- | -------- | +| Same-stem collision risk | Low | High | Low | Low | +| Operator inspectability | High | High | Lower | Lower | +| Requires global state | No | No | No | Yes | +| Handles `run.jsonl.lock` ambiguity | Yes | No | Partly | Partly | +| Complexity | Low | Low | Medium | High | + +_Table 2: Comparison of lock naming options._ + +## Decision outcome / proposed direction + +Adopt Option A. + +The writer derives lock paths by appending `.lock` to the complete accepted +journal filename in the same directory: + +```plaintext +/.lock +``` + +The operation appends to the complete filename; it does not replace an +extension. The lock for `run` is `run.lock`. The lock for `run.jsonl` is +`run.jsonl.lock`. + +The suffix `.lock` is reserved for lock files. A caller-supplied journal path +whose final filename ends in `.lock` fails before append because it can be +mistaken for another journal's coordination artefact. A caller-supplied journal +path whose derived sidecar path equals the journal path also fails; +practically, that reserves `.toml` journal filenames under the current sidecar +rule. + +Sidecar configuration still uses extension replacement: `run.jsonl` and +`run.ndjson` both use `run.toml`, while `run` also uses `run.toml`. That shared +sidecar is intentional under the current design. Callers needing independent +configuration must choose distinct stems or directories, such as +`run-jsonl.jsonl` and `run-raw.ndjson`. + +## Goals and non-goals + +- Goals: + - Make lock naming deterministic and collision-resistant for accepted journal + paths. + - Reserve coordination and configuration artefact names before implementation. + - Preserve the simple adjacent lock-file model. + - Keep sidecar sharing by stem explicit rather than accidental. +- Non-goals: + - Provide distributed locking across hosts. + - Canonicalize paths across symlinks or mount aliases beyond normal + filesystem API behaviour. + - Replace sidecar extension derivation with a different configuration scheme. + - Permit arbitrary journal filenames when they collide with coordination + artefacts. + +## Migration plan + +1. Add journal-path validation before opening the lock file. + - Reject final filenames ending in `.lock`. + - Reject paths whose derived sidecar path equals the journal path. +2. Implement lock-path derivation as a pure function and test the examples in + Table 1. +3. Use the derived lock path for the complete critical section: sidecar read, + tail repair, rotation, compression, and append. +4. Document reserved suffixes and sidecar sharing in the users' guide when the + CLI implementation lands. + +## Known risks and limitations + +- The rule does not prevent two accepted journal paths with the same stem from + sharing a sidecar. That is documented as intentional configuration sharing. +- The rule does not solve symlink aliasing or path canonicalization across + mount points. Those remain outside the local-filesystem correctness claim. +- Operators may still choose confusing stems, such as `run.locked.jsonl`; those + are accepted because they do not use the reserved final `.lock` suffix. + +## Architectural rationale + +The accepted rule keeps the lock path adjacent to the data it protects and +removes hidden ambiguity around lock artefacts. Appending `.lock` to the full +filename keeps `run` and `run.jsonl` separate, while reserving `.lock` journal +paths prevents a caller from treating another journal's lock as data. The +decision is deliberately conservative because every later concurrency, repair, +rotation, and timeout guarantee assumes all cooperating invocations agree on +the same coordination boundary. diff --git a/docs/adr-001-testing-strategy.md b/docs/adr-002-testing-strategy.md similarity index 99% rename from docs/adr-001-testing-strategy.md rename to docs/adr-002-testing-strategy.md index 45593c0..421fa60 100644 --- a/docs/adr-001-testing-strategy.md +++ b/docs/adr-002-testing-strategy.md @@ -1,4 +1,4 @@ -# Architectural decision record (ADR) 001: Testing strategy +# Architectural decision record (ADR) 002: Testing strategy ## Status diff --git a/docs/contents.md b/docs/contents.md index 5b0f180..df8b735 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -45,7 +45,10 @@ set. ## Design artefacts -- [ADR 001: Testing strategy](adr-001-testing-strategy.md) records how the +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md) records how the + journal lock path is derived and which coordination/configuration suffixes + are reserved. +- [ADR 002: Testing strategy](adr-002-testing-strategy.md) records how the repository's required testing prongs apply to the `mpsc-log` design. - [mpsc-log event schema](mpsc-log-event-schema.json) defines the initial JSON Schema for journal records. diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 6fbfc7c..c9a57b3 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -6,9 +6,10 @@ - **Companion documents:** - [Terms of reference](terms-of-reference.md). - [Context](context.md). + - [Lock file naming ADR](adr-001-lock-file-naming.md). + - [Testing strategy ADR](adr-002-testing-strategy.md). - [Event schema](mpsc-log-event-schema.json). - [Sidecar example](mpsc-log-sidecar.example.toml). - - [Testing strategy ADR](adr-001-testing-strategy.md). - [Users' guide](users-guide.md). - [Developer guide](developers-guide.md). @@ -122,11 +123,14 @@ flowchart LR Figure 1: Runtime component topology. -The lock file is the coordination boundary. `mpsc-log` derives it from the -journal path as `.lock` in the same directory. Every -invocation creates parent directories first, opens the lock file with create -semantics, acquires an exclusive lock, and only then reads configuration, -repairs the journal tail, rotates, compresses, and appends. +The lock file is the coordination boundary. The naming decision is recorded in +[ADR 001: Lock file naming](adr-001-lock-file-naming.md): `mpsc-log` appends +`.lock` to the complete journal filename in the same directory, reserves +`.lock` journal filenames, and rejects journal paths whose derived sidecar path +would equal the journal path. Every invocation creates parent directories +first, opens the lock file with create semantics, acquires an exclusive lock, +and only then reads configuration, repairs the journal tail, rotates, +compresses, and appends. ## 5. CLI contract @@ -160,7 +164,10 @@ fail with `EX_USAGE`. The root is always an object. The sidecar path replaces the journal filename extension with `.toml`; if the journal has no extension, the sidecar path appends `.toml`. For example, -`run.jsonl` uses `run.toml`, and `run` uses `run.toml`. +`run.jsonl` uses `run.toml`, and `run` uses `run.toml`. This is a configuration +sharing rule, not a lock naming rule: accepted same-stem journals such as `run`, +`run.jsonl`, and `run.ndjson` have distinct lock files but share `run.toml`. +Callers needing independent sidecars must choose distinct stems or directories. The sidecar shape is defined by [mpsc-log-sidecar.example.toml](mpsc-log-sidecar.example.toml). Configuration @@ -355,7 +362,7 @@ The command writes nothing to standard output on success. ## 11. Correctness properties and verification The testing strategy is recorded in -[ADR 001: Testing strategy](adr-001-testing-strategy.md). That ADR maps the +[ADR 002: Testing strategy](adr-002-testing-strategy.md). That ADR maps the repository's required unit, behavioural, snapshot, end-to-end, property-based, bounded-model, and proof prongs to the design surfaces below. @@ -404,7 +411,6 @@ item explicitly commits to a supported API. ## 13. Deferred ADRs - Accepted `jo` subset and last-wins duplicate-key handling. -- Lock-file naming and filesystem support policy. - Rotation naming, compression, and retention defaults. - CLI-only product boundary versus public Rust library API. diff --git a/docs/roadmap.md b/docs/roadmap.md index fbce7cd..13f32d1 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -11,9 +11,10 @@ The primary source documents are: - [terms of reference](terms-of-reference.md); - [technical design](mpsc-log-design.md); - [context](context.md); +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md); +- [ADR 002: Testing strategy](adr-002-testing-strategy.md); - [event schema](mpsc-log-event-schema.json); -- [sidecar example](mpsc-log-sidecar.example.toml); and -- [ADR 001: Testing strategy](adr-001-testing-strategy.md). +- [sidecar example](mpsc-log-sidecar.example.toml). No RFCs exist yet, so the first phase records the remaining decisions that would otherwise force rework. @@ -36,20 +37,21 @@ explicitly leave out. Its outcome informs crate layout, user documentation, error handling, and future compatibility work. See mpsc-log-design.md §§3, 5, 8, 12-13 and terms-of-reference.md §§6, 8-10. -- [ ] 1.1.1. Record the accepted `jo` subset and duplicate-key behaviour in +- [x] 1.1.1. Record lock-file naming and reserved suffixes in an ADR. + - See mpsc-log-design.md §§4, 6-7, 11, 13 and + adr-001-lock-file-naming.md. + - Success: the ADR defines adjacent lock naming, `.lock` and self-sidecar + journal rejection, sidecar sharing by stem, and the local-filesystem + coordination boundary. +- [ ] 1.1.2. Record the accepted `jo` subset and duplicate-key behaviour in an ADR. - See mpsc-log-design.md §§2, 5, 13 and terms-of-reference.md §§6, 9. - Success: the ADR names supported forms, rejected options, object-root enforcement, object-path handling, and last-wins duplicate-key semantics. -- [ ] 1.1.2. Record the CLI-only v1 product boundary in an ADR. +- [ ] 1.1.3. Record the CLI-only v1 product boundary in an ADR. - See mpsc-log-design.md §§3, 10, 12-13 and terms-of-reference.md §§6, 9. - Success: the ADR states that `src/main.rs` owns process exit mapping and that library exports are internal until a later roadmap item changes that. -- [ ] 1.1.3. Record the filesystem and locking support policy in an ADR. - - See mpsc-log-design.md §§2, 4, 7, 11, 13 and - terms-of-reference.md §§6-8. - - Success: the ADR defines the local-filesystem correctness claim, lock-file - naming, unsupported network-filesystem caveat, and timeout semantics. - [ ] 1.1.4. Record the rotation naming, compression, and retention policy in an ADR. - See mpsc-log-design.md §§3, 6, 8, 11, 13 and @@ -88,7 +90,7 @@ same CLI, domain, filesystem, and test seams. See mpsc-log-design.md §§2, 4, - Requires 1.2.1 and 1.2.2. - See mpsc-log-design.md §§7, 10-12 and docs/reliable-testing-in-rust-via-dependency-injection.md. See - adr-001-testing-strategy.md. + adr-002-testing-strategy.md. - [ ] Provide injectable clock and filesystem adapter boundaries for record timestamps, fault injection, and deterministic rotation fixtures. - Success: failures such as partial writes, metadata errors, and fixed @@ -268,7 +270,7 @@ terms-of-reference.md §§5, 7. decodable JSON object lines as successful child processes. - [ ] 3.3.2. Add pairwise CLI/configuration combination coverage. - Requires phase 2 and 3.3.1. - - See mpsc-log-design.md §11 and adr-001-testing-strategy.md. + - See mpsc-log-design.md §11 and adr-002-testing-strategy.md. - Success: the suite covers `jo` syntax form, coercion source, object path, sidecar default, rotation schedule, rotation state, and lock contention. @@ -365,7 +367,7 @@ mpsc-log-design.md §11 and terms-of-reference.md §7. - [ ] 4.4.1. Build the concurrent rotation end-to-end suite. - Requires steps 4.1-4.3. - - See mpsc-log-design.md §11 and adr-001-testing-strategy.md. + - See mpsc-log-design.md §11 and adr-002-testing-strategy.md. - Success: forced size and scheduled rotations under concurrent writers preserve the decoded record count across active, plain, scheduled, and compressed files. From 5b0450fbd5f5db19d6c7b9cec9526847189979ca Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 18:26:04 +0100 Subject: [PATCH 06/27] Track compression backend exploration Add a roadmap task to benchmark flate2, gzp, and gzippy before selecting the gzip backend for rotated journal compression. Clarify that flate2 remains the conservative design baseline until measured throughput, atomic-output integration, dependency risk, and portability justify a faster backend. --- docs/mpsc-log-design.md | 11 +++++++++++ docs/roadmap.md | 16 ++++++++++++---- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index c9a57b3..0ec4b5a 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -71,6 +71,11 @@ The Rust implementation baseline is conservative: Table 1: Dependency baseline for implementation. +The gzip choice is a conservative baseline, not a settled performance decision. +The roadmap requires a compression backend benchmark before gzip implementation +so `gzp` and `gzippy` can be evaluated against `flate2` for speed, +atomic-output integration, dependency risk, and portability.[^gzp] [^gzippy] + ## 3. Goals and non-goals ### 3.1 Goals @@ -448,3 +453,9 @@ item explicitly commits to a supported API. [^flate2]: [`flate2` crate documentation](https://docs.rs/flate2/latest/flate2/), version 1.1.9, accessed 2026-06-29. + +[^gzp]: [`gzp` crate documentation](https://docs.rs/gzp/latest/gzp/), + version 2.0.2, accessed 2026-06-29. + +[^gzippy]: [`gzippy` crate documentation](https://docs.rs/gzippy/latest/gzippy/), + version 0.8.0, accessed 2026-06-29. diff --git a/docs/roadmap.md b/docs/roadmap.md index 13f32d1..f4ee4e3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -315,14 +315,22 @@ the data-loss window the design rejects. It informs scheduled-mode retention because both modes rely on atomic gzip output and deletion ordering. See mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. -- [ ] 4.2.1. Implement atomic gzip output for rotated generations. +- [ ] 4.2.1. Benchmark and select the gzip compression backend. - Requires 4.1.2. - See mpsc-log-design.md §§2, 8. + - [ ] Benchmark `flate2`, `gzp`, and `gzippy` on representative rotated + journal files before selecting the compression backend. + - Success: the chosen backend has measured throughput, atomic-output + integration, dependency, and portability rationale documented in the design + or rotation ADR. +- [ ] 4.2.2. Implement atomic gzip output for rotated generations. + - Requires 4.2.1. + - See mpsc-log-design.md §§2, 8. - Success: failed compression leaves the source generation in place and aborts before appending the pending record. -- [ ] 4.2.2. Implement retention deletion for plain and compressed +- [ ] 4.2.3. Implement retention deletion for plain and compressed generations. - - Requires 4.2.1. + - Requires 4.2.2. - See mpsc-log-design.md §§6, 8. - Success: files beyond `plain_generations + compressed_generations` are deleted only after newer retained files are safely in place. @@ -353,7 +361,7 @@ and context.md. - Success: busy periods produce ordered `.n` suffixes, and final scheduled archives use the next suffix when a period already has size splits. - [ ] 4.3.4. Implement scheduled-mode period retention and compression. - - Requires 4.2.2 and 4.3.3. + - Requires 4.2.3 and 4.3.3. - See mpsc-log-design.md §8. - Success: the newest completed periods remain plain as complete groups, older retained periods are gzipped, and expired periods are deleted. From f71d64fcc24e73b466ad1cf73b20eb8496fae249 Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 18:34:05 +0100 Subject: [PATCH 07/27] Ratify selected jo field syntax Add ADR 003 to state that `mpsc-log` is `jo`-inspired rather than textually compatible with `jo` output. Document last-wins duplicate-path semantics, update the design and terms of reference to stop implying full `jo` compatibility, and mark the roadmap syntax decision as complete. --- docs/adr-002-testing-strategy.md | 6 +- ...-003-jo-field-syntax-and-duplicate-keys.md | 176 ++++++++++++++++++ docs/contents.md | 3 + docs/context.md | 38 ++-- docs/mpsc-log-design.md | 58 +++--- docs/roadmap.md | 40 ++-- docs/terms-of-reference.md | 37 ++-- 7 files changed, 273 insertions(+), 85 deletions(-) create mode 100644 docs/adr-003-jo-field-syntax-and-duplicate-keys.md diff --git a/docs/adr-002-testing-strategy.md b/docs/adr-002-testing-strategy.md index 421fa60..1ae4773 100644 --- a/docs/adr-002-testing-strategy.md +++ b/docs/adr-002-testing-strategy.md @@ -56,9 +56,9 @@ confidence. retention preserve successful records. - Verify that malformed tails and injected write failures do not leave partial final records after the next invocation. -- Verify the accepted `jo` subset, object-root enforcement, sidecar precedence, - schema-guided coercion, explicit coercion flags, default timestamp insertion, - and stable exit-code mapping. +- Verify the selected `jo` field syntax, object-root enforcement, sidecar + precedence, schema-guided coercion, explicit coercion flags, default + timestamp insertion, and stable exit-code mapping. - Verify df12-build fixture records against the JSON Schema and sidecar coercion contract. diff --git a/docs/adr-003-jo-field-syntax-and-duplicate-keys.md b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md new file mode 100644 index 0000000..027b8b0 --- /dev/null +++ b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md @@ -0,0 +1,176 @@ +# Architectural decision record (ADR) 003: `jo` field syntax and duplicate keys + +## Status + +Accepted on 2026-06-29. `mpsc-log` uses selected `jo`-inspired field syntax for +object records, but does not promise textual JSON compatibility with `jo`. +Duplicate writes to the same object path resolve with last-wins semantics. + +## Date + +2026-06-29. + +## Context and problem statement + +`mpsc-log` should feel familiar to workflow authors who already use `jo` to +build JSON from shell arguments. The product goal is not to be a drop-in `jo` +replacement. It is a reliable multi-process JSON Lines writer whose root record +is always an object and whose output is produced through `serde_json::Map`. + +That distinction matters for duplicate object keys. Textual JSON can contain +repeated object names, and `jo` can produce those names in its output. A JSON +object map cannot preserve repeated textual keys. Once `mpsc-log` chooses a map +as its internal representation, duplicate writes must either be rejected or +collapsed into one value. + +The project needs an explicit compatibility statement before implementing the +argument parser, merge rules, sidecar schema interaction, and tests. + +## Decision drivers + +- The CLI should preserve the ergonomic field words that make `jo` useful for + shell callers. +- The root record must remain a JSON object so defaults, telemetry fields, and + object-path writes have predictable names. +- Consumers of JSON Lines logs usually parse records into maps, so repeated + textual keys would be fragile even if the writer could emit them. +- The implementation should use structured JSON serialization rather than + manually assembling JSON text. +- Duplicate-path handling must match sidecar merge precedence and CLI + left-to-right processing. + +## Requirements + +### Functional requirements + +- Accept selected `jo`-inspired field words for object records. +- Support explicit coercion flags, schema-guided coercion, object paths, and + file-value forms that preserve an object root. +- Reject `jo` features that would produce a non-object root or require textual + output compatibility. +- Define duplicate writes to the same object path deterministically. +- Document that `mpsc-log` is `jo`-inspired, not fully `jo` compatible. + +### Technical requirements + +- Store records in `serde_json::Map` before serialization. +- Process CLI field words in argument order. +- Let later writes to the same object path replace earlier writes. +- Pair the duplicate-key behaviour with parameterized tests covering top-level + keys, nested object paths, sidecar defaults, and explicit coercion flags. + +## Options considered + +### Option A: Selected `jo`-inspired syntax with last-wins duplicate paths + +This option accepts the object-record syntax needed by `mpsc-log` and resolves +duplicate writes by replacing the earlier value at the same object path. + +Examples: + +| Arguments | Result | +| ---------------------------------------------- | ------------------- | +| `status=started status=done` | `{"status":"done"}` | +| `task.id=1 task.id=2` with `-d .` | `{"task":{"id":2}}` | +| sidecar `status = "queued"`, CLI `status=done` | `{"status":"done"}` | + +_Table 1: Last-wins examples for duplicate paths._ + +This option matches the existing merge model: sidecar defaults seed the record, +CLI fields are processed left to right, and each later CLI write wins at its +path. + +### Option B: Reject duplicate paths + +This option would fail when a CLI field writes a path already set by a sidecar +default or earlier CLI field. It is strict, but it makes deliberate overrides +awkward and complicates the existing precedence model. + +### Option C: Preserve textual duplicate JSON object names + +This option would attempt closer textual compatibility with `jo` by emitting +duplicate JSON object names. It conflicts with `serde_json::Map`, makes schema +coercion and object-path updates harder to reason about, and produces records +that many consumers collapse differently. + +### Option D: Full `jo` compatibility + +This option would expand the CLI toward all `jo` behaviours, including array +roots and formatting options. It conflicts with the object-root logging +contract and would pull the tool away from its append-safe journal purpose. + +| Topic | Option A | Option B | Option C | Option D | +| -------------------------- | -------- | -------- | -------- | -------- | +| Object-root fit | High | High | Medium | Low | +| Familiar shell syntax | High | Medium | High | High | +| Textual `jo` compatibility | Partial | Partial | Higher | High | +| Structured JSON safety | High | High | Low | Medium | +| Implementation complexity | Low | Medium | High | High | + +_Table 2: Comparison of field syntax and duplicate-key options._ + +## Decision outcome / proposed direction + +Adopt Option A. + +`mpsc-log` describes its CLI as `jo`-inspired or as selected `jo` field syntax. +It does not claim full `jo` compatibility. The `jo` manual remains prior art +for familiar field words and coercion expectations, but the `mpsc-log` contract +is the subset recorded in the design and this ADR. + +Duplicate writes to the same object path use last-wins semantics: + +1. Sidecar defaults seed the object. +2. CLI fields are processed in argument order. +3. Each CLI field replaces any existing value at its object path. +4. Explicit coercion flags affect the field they annotate before the value is + written. +5. The generated timestamp is inserted only when no `timestamp` field exists + after defaults and CLI fields. + +This means last-wins applies to duplicate top-level keys, duplicate nested +paths, and CLI overrides of sidecar defaults. It does not preserve repeated +textual JSON names. + +## Goals and non-goals + +- Goals: + - Make the compatibility boundary honest for users and implementers. + - Preserve the familiar shell-friendly field forms needed for logging. + - Keep object-root JSON serialization map-based and deterministic. + - Align duplicate-path behaviour with merge precedence. +- Non-goals: + - Preserve repeated textual JSON object names. + - Implement `jo` formatting, pretty-printing, array-root, or version options. + - Guarantee compatibility with every `jo` edge case. + - Define organization-wide schema governance for duplicate telemetry fields. + +## Migration plan + +1. Update product documentation to use `jo`-inspired or selected `jo` field + syntax rather than full compatibility language. +2. Implement parser support only for the accepted object-record forms listed in + the design. +3. Add tests for last-wins duplicate paths across sidecar defaults, CLI fields, + explicit coercion flags, and nested object paths. +4. Keep full `jo` compatibility in deferred scope unless a later ADR changes + the object-root product boundary. + +## Known risks and limitations + +- Users expecting byte-for-byte `jo` output can be surprised by duplicate keys + collapsing to one value. +- Last-wins can hide accidental repeated fields. The behaviour is predictable, + but the implementation should keep diagnostics clear for invalid paths and + unsupported forms. +- If future consumers require textual duplicate keys, the map-based record + builder would need a larger redesign. + +## Architectural rationale + +The accepted rule matches the product's actual abstraction. `mpsc-log` is a +structured journal writer, not a JSON text generator. A map-backed record gives +sidecar defaults, schema coercion, explicit flags, object paths, and timestamp +defaulting one deterministic merge model. Last-wins duplicate handling is the +least surprising behaviour inside that model, provided the documentation does +not overstate compatibility with `jo`. diff --git a/docs/contents.md b/docs/contents.md index df8b735..d204350 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -50,6 +50,9 @@ set. are reserved. - [ADR 002: Testing strategy](adr-002-testing-strategy.md) records how the repository's required testing prongs apply to the `mpsc-log` design. +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md) + records the selected `jo`-inspired field syntax and last-wins duplicate-path + semantics. - [mpsc-log event schema](mpsc-log-event-schema.json) defines the initial JSON Schema for journal records. - [mpsc-log sidecar example](mpsc-log-sidecar.example.toml) shows the TOML diff --git a/docs/context.md b/docs/context.md index 1a69a0a..4abb92e 100644 --- a/docs/context.md +++ b/docs/context.md @@ -4,25 +4,25 @@ ## Terms -| Term | Definition | -| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Active log | The JSON Lines file named on the command line. Successful invocations append new records to this file unless rotation runs first. | -| Agent workflow | A workflow that launches one or more autonomous agents or helper processes that can invoke command-line tools. | -| Audit yield | The distribution of audit findings by severity, source, and remediation lane. | -| CodeRabbit attempt | One attempt to run CodeRabbit review, including its start time, result, wait time, retry status, and whether the review was deferred. | -| Defect escape | A defect discovered after merge by later work, dogfooding, Continuous Integration (CI), or users. | -| Journal | A shared JSON Lines file used as an append-only record of workflow events. | -| Lock timeout | The maximum time an invocation waits for the journal lock before failing with a timeout diagnostic. | -| Object path | A `jo`-style field path that writes a value into a nested JSON object rather than into a literal top-level key. | -| Open Dynamic Workflows (ODW) | The workflow runtime used by `df12-build` to coordinate multi-agent roadmap execution. | -| Record | One JSON object serialized as one JSON Lines value and terminated with `\n`. | -| Remediation lane | The route assigned to a review or audit follow-up, such as addendum, step task, later roadmap step, or dropped. | -| Review round | One pass through a review loop, including any blocking findings and subsequent fix attempt. | -| Rotated log | A previous active log generation retained under a numbered or scheduled filename, optionally compressed with gzip. | -| Scheduled break | A configured UTC time boundary, either hourly, daily, or weekly, that forces the current active log segment to rotate on the next invocation. | -| Sidecar configuration | The TOML configuration file derived from the active log path by replacing the filename extension with `.toml`. | -| Type coercion | Conversion of a CLI value to a JSON string, number, boolean, null, object, or array according to explicit flags, sidecar schema, or default `jo`-compatible inference. | -| Workflow sidecar directory | The caller-owned directory where a workflow stores run artefacts, including the `mpsc-log` journal chosen by that workflow. | +| Term | Definition | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Active log | The JSON Lines file named on the command line. Successful invocations append new records to this file unless rotation runs first. | +| Agent workflow | A workflow that launches one or more autonomous agents or helper processes that can invoke command-line tools. | +| Audit yield | The distribution of audit findings by severity, source, and remediation lane. | +| CodeRabbit attempt | One attempt to run CodeRabbit review, including its start time, result, wait time, retry status, and whether the review was deferred. | +| Defect escape | A defect discovered after merge by later work, dogfooding, Continuous Integration (CI), or users. | +| Journal | A shared JSON Lines file used as an append-only record of workflow events. | +| Lock timeout | The maximum time an invocation waits for the journal lock before failing with a timeout diagnostic. | +| Object path | A `jo`-inspired field path that writes a value into a nested JSON object rather than into a literal top-level key. | +| Open Dynamic Workflows (ODW) | The workflow runtime used by `df12-build` to coordinate multi-agent roadmap execution. | +| Record | One JSON object serialized as one JSON Lines value and terminated with `\n`. | +| Remediation lane | The route assigned to a review or audit follow-up, such as addendum, step task, later roadmap step, or dropped. | +| Review round | One pass through a review loop, including any blocking findings and subsequent fix attempt. | +| Rotated log | A previous active log generation retained under a numbered or scheduled filename, optionally compressed with gzip. | +| Scheduled break | A configured UTC time boundary, either hourly, daily, or weekly, that forces the current active log segment to rotate on the next invocation. | +| Sidecar configuration | The TOML configuration file derived from the active log path by replacing the filename extension with `.toml`. | +| Type coercion | Conversion of a CLI value to a JSON string, number, boolean, null, object, or array according to explicit flags, sidecar schema, or default `jo`-inspired inference. | +| Workflow sidecar directory | The caller-owned directory where a workflow stores run artefacts, including the `mpsc-log` journal chosen by that workflow. | ## Naming diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 0ec4b5a..9f2ebf6 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -8,6 +8,7 @@ - [Context](context.md). - [Lock file naming ADR](adr-001-lock-file-naming.md). - [Testing strategy ADR](adr-002-testing-strategy.md). + - [`jo` field syntax ADR](adr-003-jo-field-syntax-and-duplicate-keys.md). - [Event schema](mpsc-log-event-schema.json). - [Sidecar example](mpsc-log-sidecar.example.toml). - [Users' guide](users-guide.md). @@ -35,12 +36,14 @@ value per line, and `\n` line termination; it is also described as suitable for log files and cooperating processes.[^jsonl] `mpsc-log` tightens the JSON Lines contract by requiring every root value to be an object. -The CLI field grammar follows `jo` where that serves object-record logging. -`jo` supports `key=value`, `key@value`, type coercion flags, object paths, -file-value prefixes, array construction, and duplicate object keys.[^jo] -`mpsc-log` rejects array roots and serializes through `serde_json`, so -duplicate keys resolve by last write rather than producing repeated JSON object -names. +The CLI field grammar is `jo`-inspired and uses selected `jo` field syntax +where that serves object-record logging. `jo` supports `key=value`, +`key@value`, type coercion flags, object paths, file-value prefixes, array +construction, and duplicate object keys.[^jo] `mpsc-log` rejects array roots +and serializes through `serde_json`, so duplicate paths resolve by last write +rather than producing repeated textual JSON object names. That compatibility +boundary is recorded in +[ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md). The default `timestamp` field uses RFC 3339 UTC. RFC 3339 defines an Internet profile of ISO 8601 for timestamps, recommends UTC for interoperability, and @@ -61,7 +64,7 @@ The Rust implementation baseline is conservative: | Need | Choice | Reason | | ----------- | ------------------------------ | ------------------------------------------------------------------------------------------------- | -| CLI parsing | `clap` builder API | Handles help/version/error rendering while allowing raw `jo`-style field tails.[^clap] | +| CLI parsing | `clap` builder API | Handles help/version/error rendering while allowing raw selected `jo` field tails.[^clap] | | JSON | `serde`, `serde_json` | Standard Rust serialization stack; object maps naturally enforce last-wins duplicate handling. | | TOML | `toml` crate, TOML v1.0 subset | Current crate supports TOML parsing; the product contract remains v1.0. | | Time | `jiff` | Provides `Timestamp::now()` and RFC 3339-style instant formatting with nanosecond support.[^jiff] | @@ -82,7 +85,7 @@ atomic-output integration, dependency risk, and portability.[^gzp] [^gzippy] - Create missing parent directories for the journal path. - Parse the first argument as the journal path and remaining arguments as - `jo`-style field words and coercion flags. + selected `jo`-inspired field words and coercion flags. - Merge sidecar defaults, CLI fields, schema-guided coercions, and generated timestamp values into one JSON object. - Serialize exactly one compact JSON object plus `\n` per successful @@ -163,7 +166,9 @@ The field tail accepts this subset: | `key[sub]=value` | Insert into an object at `key.sub`. | Unsupported `jo` options such as `-a`, `-p`, `-f`, `-D`, `-e`, `-v`, and `-V` -fail with `EX_USAGE`. The root is always an object. +fail with `EX_USAGE`. The root is always an object. The field syntax is +`jo`-inspired rather than textually `jo` compatible: duplicate writes to the +same object path use last-wins semantics inside the record map. ## 6. Sidecar configuration @@ -187,19 +192,19 @@ has four tables: Merge and coercion order is deterministic: -| Step | Rule | Winner | -| ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | -| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | -| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo` inference wins. | -| 4 | Insert the generated invocation timestamp. | The generated canonical UTC `timestamp` is added only when no `timestamp` field exists after defaults and CLI fields. | -| 5 | Serialize the resulting object. | The merged record is written as one compact JSON object. | +| Step | Rule | Winner | +| ---- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | +| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | +| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | +| 4 | Insert the generated invocation timestamp. | The generated canonical UTC `timestamp` is added only when no `timestamp` field exists after defaults and CLI fields. | +| 5 | Serialize the resulting object. | The merged record is written as one compact JSON object. | The sidecar schema never overrides an explicit CLI coercion flag. Schema entries affect only values supplied for the matching object path; they do not -change sidecar defaults or the generated timestamp. When default `jo` inference -is used, valid JSON values parse as JSON, empty `key=` becomes `null`, and -other values remain strings. +change sidecar defaults or the generated timestamp. When default `jo`-inspired +inference is used, valid JSON values parse as JSON, empty `key=` becomes +`null`, and other values remain strings. ## 7. Journal write protocol @@ -380,14 +385,14 @@ The implementation must satisfy these properties: | Rotation preserves successful records across active, plain rotated, scheduled rotated, and compressed rotated files. | End-to-end test that forces size and scheduled rotation under concurrent writers and counts decoded records. | | Detected write failures do not leave a partial final line. | Fault-injection adapter test that fails after partial writes and checks truncate repair. | | Crash-like partial tails are repaired before the next append. | Fixture with malformed trailing bytes followed by a successful append. | -| CLI coercion follows the accepted `jo` subset. | Parameterized examples copied from the accepted `jo` behaviours. | +| CLI coercion follows the selected `jo` field syntax. | Parameterized examples cover the accepted `jo`-inspired behaviours and last-wins duplicate paths. | | Sidecar/CLI precedence is deterministic. | Table-driven tests covering defaults, schema coercion, explicit flags, and `timestamp`. | -The combination surface is `jo` syntax form × coercion source × object path × -sidecar default × rotation schedule × rotation state × lock contention. The -test suite must cover pairwise combinations across those axes, plus targeted -cases for size and scheduled rotation under contention and failed filesystem -operations. +The combination surface is selected `jo` syntax form × coercion source × object +path × sidecar default × rotation schedule × rotation state × lock contention. +The test suite must cover pairwise combinations across those axes, plus +targeted cases for size and scheduled rotation under contention and failed +filesystem operations. Network filesystems remain outside the correctness claim until a separate platform verification matrix proves lock and rename behaviour for a named @@ -400,7 +405,7 @@ the implementation: | Module | Responsibility | | --------- | ------------------------------------------------------------------------------ | -| `args` | Parse the journal path and raw `jo` field tail. | +| `args` | Parse the journal path and raw selected `jo` field tail. | | `fields` | Parse field words, object paths, coercion flags, and file-value forms. | | `config` | Load and validate sidecar TOML. | | `record` | Merge defaults, CLI fields, schema coercions, and generated timestamp. | @@ -415,7 +420,6 @@ item explicitly commits to a supported API. ## 13. Deferred ADRs -- Accepted `jo` subset and last-wins duplicate-key handling. - Rotation naming, compression, and retention defaults. - CLI-only product boundary versus public Rust library API. diff --git a/docs/roadmap.md b/docs/roadmap.md index f4ee4e3..f436215 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -13,6 +13,7 @@ The primary source documents are: - [context](context.md); - [ADR 001: Lock file naming](adr-001-lock-file-naming.md); - [ADR 002: Testing strategy](adr-002-testing-strategy.md); +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md); - [event schema](mpsc-log-event-schema.json); - [sidecar example](mpsc-log-sidecar.example.toml). @@ -26,7 +27,7 @@ spine before feature work starts, later slices can converge on one small CLI instead of repeatedly reopening syntax, locking, and retention decisions. This phase resolves the decisions that affect every subsequent pull request: -the accepted `jo` subset, public API boundary, filesystem support policy, +the selected `jo` field syntax, public API boundary, filesystem support policy, rotation naming, and the repository shape needed to validate a CLI whose main risks are concurrency and persistence. @@ -43,11 +44,13 @@ error handling, and future compatibility work. See mpsc-log-design.md §§3, 5, - Success: the ADR defines adjacent lock naming, `.lock` and self-sidecar journal rejection, sidecar sharing by stem, and the local-filesystem coordination boundary. -- [ ] 1.1.2. Record the accepted `jo` subset and duplicate-key behaviour in - an ADR. - - See mpsc-log-design.md §§2, 5, 13 and terms-of-reference.md §§6, 9. - - Success: the ADR names supported forms, rejected options, object-root - enforcement, object-path handling, and last-wins duplicate-key semantics. +- [x] 1.1.2. Record the selected `jo` field syntax and duplicate-key + behaviour in an ADR. + - See mpsc-log-design.md §§2, 5, 13, terms-of-reference.md §§6, 9, and + adr-003-jo-field-syntax-and-duplicate-keys.md. + - Success: the ADR names selected `jo`-inspired forms, rejected options, + object-root enforcement, object-path handling, and last-wins duplicate-key + semantics. - [ ] 1.1.3. Record the CLI-only v1 product boundary in an ADR. - See mpsc-log-design.md §§3, 10, 12-13 and terms-of-reference.md §§6, 9. - Success: the ADR states that `src/main.rs` owns process exit mapping and @@ -98,9 +101,10 @@ same CLI, domain, filesystem, and test seams. See mpsc-log-design.md §§2, 4, ## 2. Day-one structured journal entries -Idea: if one `mpsc-log` invocation can parse `jo`-style fields, apply sidecar -defaults, emit an RFC 3339 UTC timestamp, and append one JSONL object, the tool -already replaces the unsafe `jo >> file.jsonl` baseline for simple workflows. +Idea: if one `mpsc-log` invocation can parse selected `jo`-inspired fields, +apply sidecar defaults, emit an RFC 3339 UTC timestamp, and append one JSONL +object, the tool already replaces the unsafe `jo >> file.jsonl` baseline for +simple workflows. This phase delivers the narrowest useful end-to-end command. It intentionally starts without the full concurrency and rotation surface so that parser, @@ -109,10 +113,10 @@ examples before the filesystem protocol grows more complex. ### 2.1. Prove the CLI field contract can build one object record -This step answers whether the accepted `jo` subset can be implemented without -breaking the object-root logging contract. It informs sidecar coercion, error -classification, and the later combinatorial test matrix. See mpsc-log-design.md -§§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. +This step answers whether the selected `jo` field syntax can be implemented +without breaking the object-root logging contract. It informs sidecar coercion, +error classification, and the later combinatorial test matrix. See +mpsc-log-design.md §§2, 5, 10-12 and terms-of-reference.md §§2, 6, 8. - [ ] 2.1.1. Implement positional argument parsing for the journal path and raw field tail. @@ -120,14 +124,14 @@ classification, and the later combinatorial test matrix. See mpsc-log-design.md - See mpsc-log-design.md §§5, 10, 12. - Success: the parser preserves field words and coercion flags in order and rejects missing journal paths with `EX_USAGE`. -- [ ] 2.1.2. Implement field-word parsing for the accepted `jo` object forms. +- [ ] 2.1.2. Implement field-word parsing for the selected `jo` object forms. - Requires 2.1.1. - See mpsc-log-design.md §§2, 5 and terms-of-reference.md §§2, 6. - [ ] Cover `key=value`, `key@value`, object paths, array appends, and bracketed object insertion. - [ ] Reject unsupported `jo` options and any non-object root outcome. - - Success: representative `jo` examples produce typed intermediate values - or stable usage errors. + - Success: representative selected `jo`-inspired examples produce typed + intermediate values or stable usage errors. - [ ] 2.1.3. Implement explicit and inferred type coercion for CLI values. - Requires 2.1.2. - See mpsc-log-design.md §§5-6 and context.md. @@ -161,8 +165,8 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md mpsc-log-event-schema.json. - Success: sidecar defaults seed the record; CLI fields win over defaults and earlier duplicate paths; CLI coercion uses explicit `-s`, `-n`, or - `-b` flags before schema entries and default `jo` inference; and the - generated `timestamp` is inserted only if no timestamp exists after + `-b` flags before schema entries and default `jo`-inspired inference; and + the generated `timestamp` is inserted only if no timestamp exists after defaults and CLI fields. - [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. - Requires 1.2.3 and 2.2.2. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index ec09680..608135a 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -54,10 +54,10 @@ log easy to append, stream, grep, split, and ingest into later tools without requiring the whole file to be rewritten. The command accepts the log path as its first argument. Later arguments -describe fields using `jo`-style key and value words, including type coercion -flags and object paths. Unlike `jo`, the root value must always be an object: -array roots are out of scope because each log entry needs named fields and -configurable defaults. +describe fields using selected `jo`-inspired key and value words, including +type coercion flags and object paths. The CLI is not textually compatible with +all `jo` output: the root value must always be an object, and duplicate writes +to the same object path use last-wins semantics. The default entry includes a `timestamp` field containing a Coordinated Universal Time (UTC) timestamp captured when the command is invoked. The @@ -107,8 +107,8 @@ The current alternatives each solve part of the problem: - `logrotate` can rotate files, but it is scheduled external maintenance rather than a per-write safety mechanism integrated with concurrent appends. -The gap is a narrowly scoped CLI that combines `jo`-style record construction -with safe append and rotation behaviour for one local JSONL file. +The gap is a narrowly scoped CLI that combines selected `jo`-inspired record +construction with safe append and rotation behaviour for one local JSONL file. ## 4. Users and stakeholders @@ -148,8 +148,8 @@ artefact rather than trusting a caller's informal summary. - Accept a log-file path as the first CLI argument and interpret following arguments as record fields. -- Support the `jo`-style key/value syntax needed for object records, including - type coercion flags and object paths. +- Support the selected `jo`-inspired key/value syntax needed for object + records, including type coercion flags and object paths. - Reject any invocation that would produce a non-object root. - Add a default `timestamp` field using an invocation-time UTC timestamp unless configuration or CLI input explicitly overrides it according to the final @@ -187,8 +187,8 @@ artefact rather than trusting a caller's informal summary. - Preserving chronological file order by timestamp is out of scope. The default timestamp is captured at invocation time, so entries can appear out of order when one caller waits behind another. -- Full `jo` feature parity is out of scope where it conflicts with the - object-root logging contract. +- Full `jo` feature parity and textual duplicate-key preservation are out of + scope where they conflict with the object-root logging contract. - Cross-host distributed locking is out of scope unless a later design explicitly accepts the complexity and platform constraints. - Querying, filtering, formatting, or editing historical log entries is out of @@ -248,7 +248,8 @@ artefact rather than trusting a caller's informal summary. ### 8.1 Hard constraints - The first CLI parameter is the log-file path. -- Later parameters are key/value words using the accepted `jo`-style syntax. +- Later parameters are key/value words using the accepted `jo`-inspired + syntax. - The root record must be a JSON object. - The default lock timeout is five seconds. - The default rotation threshold is 1 MiB. @@ -298,8 +299,9 @@ artefact rather than trusting a caller's informal summary. ### 8.3 Dependencies -- The `jo` manual is the behavioural reference for compatible argument syntax, - type coercion flags, and object-path expectations.[^1] +- The `jo` manual is prior art for selected argument syntax, type coercion + flags, and object-path expectations; `mpsc-log` compatibility is limited by + the object-root and last-wins decisions.[^1] - RFC 3339 is the timestamp reference for default `timestamp` values.[^2] - The future technical design must choose Rust crates or platform APIs for JSON serialization, TOML parsing, file locking, gzip compression, time handling, @@ -309,9 +311,9 @@ artefact rather than trusting a caller's informal summary. | Question | Why it matters | Criteria for resolution | Suggested path | | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- | -| Which exact subset of `jo` syntax is in scope? | `jo` includes arrays, file-value operators, duplicate-key behaviour, object paths, and coercion flags that may conflict with an object-root log contract. | The design records accepted, rejected, and modified syntax with examples. | Design spike against `jo` examples. | +| Which exact subset of `jo` syntax is in scope? | `jo` includes arrays, file-value operators, duplicate-key behaviour, object paths, and coercion flags that may conflict with an object-root log contract. | The design records accepted, rejected, and modified syntax with examples. | Resolved by ADR 003. | | What are the sidecar precedence rules? | Defaults, schema coercion, CLI values, and `timestamp` overrides can conflict. | A precedence table defines every conflict outcome. | Technical design. | -| How should duplicate keys resolve? | JSON permits duplicate object names textually, but consumers often collapse them. | The product chooses reject, last-wins, first-wins, or compatibility behaviour. | ADR candidate. | +| How should duplicate keys resolve? | JSON permits duplicate object names textually, but consumers often collapse them. | The product chooses reject, last-wins, first-wins, or compatibility behaviour. | Resolved by ADR 003. | | What does "gzipping after 4 rotations" mean exactly? | Retention, naming, and compression timing affect concurrency and user expectations. | The rotation policy specifies filenames, retention count, compression trigger, and whether recent rotations remain plain text. | Technical design. | | What lock protects rotation and sidecar reads? | Append-only locking may not be enough when a process rotates while others are opening or creating files. | The design names the lock artefact and the critical sections it protects. | Technical design and stress tests. | | What platforms are supported at v1? | File-locking and atomic rename semantics differ across Unix, Windows, and network filesystems. | The project declares supported platforms and test coverage. | ADR candidate. | @@ -350,7 +352,6 @@ vocabulary for: ### 10.2 ADR candidates -- Accepted `jo` compatibility subset and deviations. - Locking and atomic-write strategy across supported platforms. - Rotation naming, retention, and compression policy. - CLI-only product boundary versus supported library API. @@ -358,8 +359,8 @@ vocabulary for: ### 10.3 Downstream readiness This document is complete enough to start a technical design. The design should -not begin implementation until the timestamp standard, `jo` compatibility -subset, sidecar precedence, rotation policy, and platform support questions are +not begin implementation until the timestamp standard, selected `jo` field +syntax, sidecar precedence, rotation policy, and platform support questions are resolved or explicitly deferred. ## Appendix A. References From 0d727dcee90bba46cab257d9a4b7bcbc261f9dcf Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Mon, 29 Jun 2026 18:38:09 +0100 Subject: [PATCH 08/27] Define corrupt final-line handling Clarify that partial-tail repair only applies to unterminated final bytes and that newline-terminated corrupt final records fail closed with `EX_DATAERR` without truncation. Update the roadmap, testing strategy, glossary, sidecar example, and terms of reference so implementation and operator guidance cover this case explicitly. --- docs/adr-002-testing-strategy.md | 14 ++++++---- docs/context.md | 2 ++ docs/mpsc-log-design.md | 45 ++++++++++++++++++++---------- docs/mpsc-log-sidecar.example.toml | 2 ++ docs/roadmap.md | 13 +++++---- docs/terms-of-reference.md | 3 +- 6 files changed, 53 insertions(+), 26 deletions(-) diff --git a/docs/adr-002-testing-strategy.md b/docs/adr-002-testing-strategy.md index 1ae4773..6d79f10 100644 --- a/docs/adr-002-testing-strategy.md +++ b/docs/adr-002-testing-strategy.md @@ -54,8 +54,10 @@ confidence. complete records as successful process exits. - Verify that size-only rotation, scheduled rotation, gzip compression, and retention preserve successful records. -- Verify that malformed tails and injected write failures do not leave partial - final records after the next invocation. +- Verify that malformed unterminated tails and injected write failures do not + leave partial final records after the next invocation. +- Verify that newline-terminated corrupt final records fail closed without + truncation. - Verify the selected `jo` field syntax, object-root enforcement, sidecar precedence, schema-guided coercion, explicit coercion flags, default timestamp insertion, and stable exit-code mapping. @@ -133,7 +135,7 @@ The testing prongs apply as follows: `config`, `record`, `errors`, `clock`, and rotation-planning helpers. - Behavioural tests with `rstest-bdd` cover user-visible CLI behaviours: successful append, unsupported syntax, sidecar precedence, diagnostics, - timeout, repair, and rotation scenarios. + timeout, repair, corrupt-final-line, and rotation scenarios. - Snapshot tests with `insta` cover stable outputs that reviewers benefit from seeing as artefacts: help text, one-line diagnostics, canonical fixture records after nondeterministic normalization, and rotation filename matrices. @@ -143,7 +145,7 @@ The testing prongs apply as follows: df12-build fixture commands. - Property tests with `proptest` cover input and state ranges for field parsing, object-path insertion, type coercion, merge precedence, partial-tail - repair, and rotation-plan invariants. + repair, final-line validation, and rotation-plan invariants. - Bounded model checks with `kani` cover compact pure planners where exhaustive small-state exploration is more useful than randomized testing, especially generation shifting and scheduled-period retention. @@ -184,8 +186,8 @@ The testing prongs apply as follows: 4. Add concurrency, repair, and rotation hardening as the filesystem adapter lands. - Exercise multi-process contention, lock timeout, partial-tail repair, - injected I/O failures, size rotation, scheduled rotation, gzip, and - retention. + newline-terminated corrupt final records, injected I/O failures, size + rotation, scheduled rotation, gzip, and retention. 5. Add model checking or proofs only when the implementation introduces a pure invariant-bearing planner or lemma. - Prefer `kani` for bounded state machines and Verus for explicit lemmas diff --git a/docs/context.md b/docs/context.md index 4abb92e..fda0085 100644 --- a/docs/context.md +++ b/docs/context.md @@ -10,11 +10,13 @@ | Agent workflow | A workflow that launches one or more autonomous agents or helper processes that can invoke command-line tools. | | Audit yield | The distribution of audit findings by severity, source, and remediation lane. | | CodeRabbit attempt | One attempt to run CodeRabbit review, including its start time, result, wait time, retry status, and whether the review was deferred. | +| Corrupt final line | A newline-terminated final JSON Lines record that is not a valid JSON object. The default repair path reports this instead of truncating it. | | Defect escape | A defect discovered after merge by later work, dogfooding, Continuous Integration (CI), or users. | | Journal | A shared JSON Lines file used as an append-only record of workflow events. | | Lock timeout | The maximum time an invocation waits for the journal lock before failing with a timeout diagnostic. | | Object path | A `jo`-inspired field path that writes a value into a nested JSON object rather than into a literal top-level key. | | Open Dynamic Workflows (ODW) | The workflow runtime used by `df12-build` to coordinate multi-agent roadmap execution. | +| Partial tail | Unterminated bytes after the final newline in the active log. Repair may truncate these bytes before appending the next record. | | Record | One JSON object serialized as one JSON Lines value and terminated with `\n`. | | Remediation lane | The route assigned to a review or audit follow-up, such as addendum, step task, later roadmap step, or dropped. | | Review round | One pass through a review loop, including any blocking findings and subsequent fix attempt. | diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 9f2ebf6..e2f2f38 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -222,7 +222,7 @@ sequenceDiagram M->>M: create parent directories M->>L: open or create lock file M->>L: acquire exclusive lock with timeout - M->>J: repair trailing partial record + M->>J: validate final record and repair partial tail M->>R: rotate and compress if threshold or schedule requires it M->>J: append record plus newline M->>L: release lock @@ -233,9 +233,25 @@ Figure 2: Critical-section protocol. The writer records the active file length before appending. If `write_all` returns an error, the writer truncates the active file back to that length -before returning. If the process dies during a write, the next invocation's -tail repair scans backward to the final `\n`, validates the remaining final -line if present, and truncates any partial tail before appending. +before returning. If the process dies during a write, the next invocation first +classifies the active file tail while holding the journal lock: + +- Empty files need no repair. +- Files ending without `\n` have an unterminated final line. When + `repair_partial_tail = true`, the writer scans backward to the previous `\n`, + truncates the unterminated bytes, and appends the new record. When + `repair_partial_tail = false`, the invocation fails with `EX_DATAERR` and + leaves the file unchanged. +- Files ending with `\n` have no partial tail. The complete final line is the + bytes between the previous `\n` and the terminal `\n`. That line must be a + valid JSON object unless the file has no lines. If the complete final line is + invalid JSON, not an object, or empty, the invocation fails closed with + `EX_DATAERR` and leaves the file unchanged. + +The repair path does not quarantine files and does not truncate a +newline-terminated corrupt record by default. It also does not scan every +historical line on each append; it validates only the active file's complete +final line and any unterminated tail that can affect the next append. The append path uses `OpenOptions::append(true).create(true)`. The lock, not append mode alone, provides cross-process serialization. The design treats NFS, @@ -351,15 +367,15 @@ mpsc-log .odw/run-20260629/journal.jsonl \ The library exposes a semantic error enum. The binary maps it to `sysexits` style process statuses: -| Exit | Name | Condition | -| ---- | -------------- | ---------------------------------------------------------------------------------------- | -| 0 | `OK` | Record appended. | -| 64 | `EX_USAGE` | Invalid CLI syntax, unsupported `jo` option, non-object root, or duplicate invalid path. | -| 65 | `EX_DATAERR` | Invalid JSON, invalid TOML, invalid schema coercion, or malformed sidecar value. | -| 73 | `EX_CANTCREAT` | Parent directory, lock file, journal, or rotated file cannot be created. | -| 74 | `EX_IOERR` | Write, flush, truncate, rename, compression, or filesystem metadata failure. | -| 75 | `EX_TEMPFAIL` | Lock timeout. | -| 78 | `EX_CONFIG` | Sidecar configuration is syntactically valid TOML but semantically invalid. | +| Exit | Name | Condition | +| ---- | -------------- | ---------------------------------------------------------------------------------------------------- | +| 0 | `OK` | Record appended. | +| 64 | `EX_USAGE` | Invalid CLI syntax, unsupported `jo` option, non-object root, or duplicate invalid path. | +| 65 | `EX_DATAERR` | Invalid JSON, invalid TOML, invalid schema coercion, corrupt final line, or malformed sidecar value. | +| 73 | `EX_CANTCREAT` | Parent directory, lock file, journal, or rotated file cannot be created. | +| 74 | `EX_IOERR` | Write, flush, truncate, rename, compression, or filesystem metadata failure. | +| 75 | `EX_TEMPFAIL` | Lock timeout. | +| 78 | `EX_CONFIG` | Sidecar configuration is syntactically valid TOML but semantically invalid. | Diagnostics go to standard error as one line: @@ -384,7 +400,8 @@ The implementation must satisfy these properties: | Concurrent successful invocations produce the same number of complete records as successes. | Stress test with many processes writing one journal. | | Rotation preserves successful records across active, plain rotated, scheduled rotated, and compressed rotated files. | End-to-end test that forces size and scheduled rotation under concurrent writers and counts decoded records. | | Detected write failures do not leave a partial final line. | Fault-injection adapter test that fails after partial writes and checks truncate repair. | -| Crash-like partial tails are repaired before the next append. | Fixture with malformed trailing bytes followed by a successful append. | +| Unterminated partial tails are repaired before the next append when repair is enabled. | Fixture with malformed trailing bytes followed by a successful append. | +| Newline-terminated corrupt final records fail closed. | Fixture ending in `\n` with an invalid final line returns `EX_DATAERR` without truncating. | | CLI coercion follows the selected `jo` field syntax. | Parameterized examples cover the accepted `jo`-inspired behaviours and last-wins duplicate paths. | | Sidecar/CLI precedence is deterministic. | Table-driven tests covering defaults, schema coercion, explicit flags, and `timestamp`. | diff --git a/docs/mpsc-log-sidecar.example.toml b/docs/mpsc-log-sidecar.example.toml index 7c41c1f..067a16d 100644 --- a/docs/mpsc-log-sidecar.example.toml +++ b/docs/mpsc-log-sidecar.example.toml @@ -7,6 +7,8 @@ gzip_after_plain_generations = true [locking] timeout_ms = 5000 +# Repairs only unterminated final bytes; newline-terminated corrupt records +# fail closed with EX_DATAERR. repair_partial_tail = true [defaults] diff --git a/docs/roadmap.md b/docs/roadmap.md index f436215..55e0e68 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -251,14 +251,16 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md - [ ] 3.2.2. Implement partial-tail repair before every append. - Requires 3.2.1. - See mpsc-log-design.md §§6-7, 11. - - Success: malformed trailing bytes and unterminated final records are - removed before a new valid record is appended. + - Success: unterminated final records are removed before a new valid record + is appended when repair is enabled, while newline-terminated invalid final + records fail closed with `EX_DATAERR` and leave the file unchanged. - [ ] 3.2.3. Build the fault-injection filesystem coverage for write, truncate, rename, compression, and metadata failures. - Requires 1.2.3, 3.2.1, and 3.2.2. - See mpsc-log-design.md §§7, 10-12. - Success: each documented filesystem failure class has a deterministic - assertion for journal state and exit-code mapping. + assertion for journal state and exit-code mapping, including the invalid + newline-terminated final-line case. ### 3.3. Demonstrate concurrent append correctness end to end @@ -447,8 +449,9 @@ and keeps deferred analysis work out of the core CLI. See terms-of-reference.md - Requires 5.2.1. - See mpsc-log-design.md §§7, 10-11 and terms-of-reference.md §§7-8. - Success: operators can distinguish invalid arguments, malformed sidecars, - timeout, directory creation failure, partial-tail repair, and rotation - failure from command output and exit status. + timeout, directory creation failure, partial-tail repair, + newline-terminated final-line corruption, and rotation failure from command + output and exit status. - [ ] 5.2.3. Add a release smoke script for the documented workflows. - Requires 5.2.1 and 5.2.2. - See mpsc-log-design.md §§5-11. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index 608135a..e3630ae 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -228,7 +228,8 @@ artefact rather than trusting a caller's informal summary. - A failed invocation leaves the previous log state readable and does not produce partial JSON records. - Diagnostics are useful to agents and humans: invalid arguments, timeout, - malformed sidecar configuration, and file-system failures are distinguishable. + malformed sidecar configuration, final-line corruption, and file-system + failures are distinguishable. - Journal writes add low enough overhead that agents can record telemetry at phase and attempt boundaries without changing workflow scheduling decisions. From a80b2708911235ea6d9f844f95ed9bd0e69af985 Mon Sep 17 00:00:00 2001 From: Payton McIntosh Date: Tue, 30 Jun 2026 01:48:49 +0100 Subject: [PATCH 09/27] Clarify scheduled rollover ordering Specify that scheduled rotation finalizes the previous period before checking `max_bytes` for the pending record against the fresh active file. Remove the brittle df12-build repository-relative path from the terms of reference and correct the plural wording for the document title. --- docs/mpsc-log-design.md | 22 +++++++++++++--------- docs/terms-of-reference.md | 20 +++++++++----------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index e2f2f38..14aa13a 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -305,19 +305,23 @@ The scheduled protocol is: 1. Determine the pending record period from the invocation timestamp. 2. Determine the active file period from the first complete record in the active log, falling back to the pending period when the active log is empty. -3. If the active period is older than the pending record period, rotate the - active log into the active period's scheduled filename. -4. If the active log plus the pending record would exceed `max_bytes`, rotate - the active log into the pending period's next size-split filename. +3. If the active period is older than the pending record period, finalize the + previous period first by rotating the active log into the active period's + scheduled filename. The active path is then a fresh empty file for the + pending period. +4. Evaluate `max_bytes` against the current active file after any + period-boundary rollover. If that active file plus the pending record would + exceed `max_bytes`, rotate it into the pending period's next size-split + filename and continue with a fresh active file. 5. Append the pending record to the active log. Within a scheduled period, reaching `max_bytes` before the time boundary creates an interim size split. Size splits add a numeric suffix within that -period: `run.2026-06-29.1.jsonl`, `run.2026-06-29.2.jsonl`, and so on. If a -period has no size splits, its final scheduled archive uses the unsuffixed base -name. If a period already has one or more size-split files, the final archive -at the next time boundary also uses the next numeric suffix so that every file -for that period has an ordered generation number. +period: `run..1.jsonl`, `run..2.jsonl`, and so on. If a period +has no size splits, its final scheduled archive uses the unsuffixed base name, +such as `run..jsonl`. If a period already has one or more size-split +files, the final archive at the next time boundary also uses the next numeric +suffix so that every file for that period has an ordered generation number. Retention and compression are period-based in scheduled mode. The newest four completed periods remain plain, including all size-split files inside those diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index e3630ae..2719ea9 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -32,19 +32,17 @@ continue. The tool must make the file-system coordination boring: safe file creation, file locking, write atomicity, timeout handling, and rotation belong inside the tool rather than being reimplemented by each caller. -The first concrete use case is -`../df12-build.worktrees/codex-annex/workflows/df12-build-odw.js`, a -multi-agent Open Dynamic Workflows (ODW) workflow for advancing -`docs/roadmap.md` through planning, design review, implementation, code review, -expert review, integration, audit, and remediation. Agents in that workflow -need to append journal events to a file in the workflow sidecar directory. The -journal should capture enough real-run telemetry to replace architectural -guesses with measured behaviour. +The first concrete use case is a multi-agent Open Dynamic Workflows (ODW) +workflow for advancing `docs/roadmap.md` through planning, design review, +implementation, code review, expert review, integration, audit, and +remediation. Agents in that workflow need to append journal events to a file in +the workflow sidecar directory. The journal should capture enough real-run +telemetry to replace architectural guesses with measured behaviour. This project is being defined before its design document. The existing -repository is a generated Rust application scaffold, so this terms of reference -treats the user brief and `Cargo.toml` package description as the authoritative -product inputs. +repository is a generated Rust application scaffold, so these terms of +reference treat the user brief and `Cargo.toml` package description as the +authoritative product inputs. ## 2. Domain From e928924b9aac0fb6eb763cf865eedec26ee13fbc Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 30 Jun 2026 12:33:56 +0200 Subject: [PATCH 10/27] Address design roadmap review notes Clarify df12-build workflow wording and make the workflow phase list parallel. Use a week-specific scheduled rotation filename example. --- docs/mpsc-log-design.md | 11 +++++------ docs/terms-of-reference.md | 6 +++--- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 14aa13a..4e09500 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -23,11 +23,10 @@ rewriting the same `jo`, `flock`, append, timeout, and rotation glue in every caller. The first integration target is the df12-build Open Dynamic Workflows (ODW) -workflow at `../df12-build.worktrees/codex-annex/workflows/df12-build-odw.js`. -Agents in that workflow will write journal records into a caller-owned workflow -sidecar directory. The records measure phase timing, review rounds, task shape, -CodeRabbit waits and HTTP 429s, audit yield, remediation lane outcomes, and -post-merge defect escape. +workflow. Agents in that workflow will write journal records into a +caller-owned workflow sidecar directory. The records measure phase timing, +review rounds, task shape, CodeRabbit waits and HTTP 429s, audit yield, +remediation lane outcomes, and post-merge defect escape. ## 2. Research baseline @@ -293,7 +292,7 @@ boundaries derived from the invocation timestamp: | -------- | --------------------------------------- | ------------------------- | | `hourly` | UTC hour beginning | `run.2026-06-29T14.jsonl` | | `daily` | UTC day beginning | `run.2026-06-29.jsonl` | -| `weekly` | UTC ISO week beginning, Monday 00:00:00 | `run.2026-06-29.jsonl` | +| `weekly` | UTC ISO week beginning, Monday 00:00:00 | `run.2026-W27.jsonl` | Scheduled rotation is opportunistic rather than daemon-driven. If no invocation occurs at the exact boundary, the next invocation rotates the previous active diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index 2719ea9..c892da0 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -34,9 +34,9 @@ inside the tool rather than being reimplemented by each caller. The first concrete use case is a multi-agent Open Dynamic Workflows (ODW) workflow for advancing `docs/roadmap.md` through planning, design review, -implementation, code review, expert review, integration, audit, and -remediation. Agents in that workflow need to append journal events to a file in -the workflow sidecar directory. The journal should capture enough real-run +implementation, code review, expert review, integration, audit, and remediation +phases. Agents in that workflow need to append journal events to a file in the +workflow sidecar directory. The journal should capture enough real-run telemetry to replace architectural guesses with measured behaviour. This project is being defined before its design document. The existing From f0857f860dc8e69a868fe18a6bbc9f6364834563 Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 19 Jul 2026 19:03:29 +0200 Subject: [PATCH 11/27] Allow flate2 crate identifier Record the crate name in the repository spelling overlay and generated configuration so design documentation passes the Markdown spelling gate. --- typos.local.toml | 3 ++- typos.toml | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/typos.local.toml b/typos.local.toml index f491eb4..271fd52 100644 --- a/typos.local.toml +++ b/typos.local.toml @@ -7,7 +7,7 @@ stems = [] [words] # Official names: Azure Architecture Center and GitHub Flavored Markdown. -accepted = [] +accepted = ["flate2"] [words.corrections] @@ -15,6 +15,7 @@ accepted = [] ignore = [ "\\bAzure Architecture\\s+Center \\| Microsoft Learn\\b", "\\bGitHub Flavored\\s+Markdown\\b", + "\\bflate2\\b", # Inline code spans quote identifiers verbatim, so they are not en-GB # prose; the shared base previously supplied this exemption, so this # restores it locally. diff --git a/typos.toml b/typos.toml index 9868d35..89afddd 100644 --- a/typos.toml +++ b/typos.toml @@ -36,6 +36,7 @@ extend-ignore-re = [ "Cranelift and mold", "\\bAzure Architecture\\s+Center \\| Microsoft Learn\\b", "\\bGitHub Flavored\\s+Markdown\\b", + "\\bflate2\\b", "\\brust-analyzer\\b", "`[^`\\n]+`", "`mold`", @@ -846,6 +847,7 @@ extend-ignore-re = [ "finalizers" = "finalizers" "finalizes" = "finalizes" "finalizing" = "finalizing" +"flate2" = "flate2" "formalisable" = "formalizable" "formalisably" = "formalizably" "formalisation" = "formalization" From 6955e006a071d578745246db02bc4e66d43abeea Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 19 Jul 2026 19:04:00 +0200 Subject: [PATCH 12/27] Remove duplicate contents spacing Keep the documentation index compliant with the Markdown blank-line rule. --- docs/contents.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/contents.md b/docs/contents.md index d204350..0fe8950 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -42,7 +42,6 @@ set. scripting stack, command execution patterns, and test expectations for helper scripts. - ## Design artefacts - [ADR 001: Lock file naming](adr-001-lock-file-naming.md) records how the From 91f01b9aeada566d6953094961a3c8b335a09a89 Mon Sep 17 00:00:00 2001 From: leynos Date: Mon, 20 Jul 2026 00:44:10 +0200 Subject: [PATCH 13/27] Clarify journal schema and rotation rules Require complete telemetry for each initial df12-build event, validate timestamp overrides before append, and avoid scheduled archive collisions. Align the roadmap and terms of reference with these established contracts. --- docs/mpsc-log-design.md | 38 ++++++++++-------- docs/mpsc-log-event-schema.json | 70 ++++++++++++++++++++++++++++++++- docs/roadmap.md | 22 ++++++----- docs/terms-of-reference.md | 11 +++--- 4 files changed, 107 insertions(+), 34 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 4e09500..6884c57 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -191,13 +191,14 @@ has four tables: Merge and coercion order is deterministic: -| Step | Rule | Winner | -| ---- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | -| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | -| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | -| 4 | Insert the generated invocation timestamp. | The generated canonical UTC `timestamp` is added only when no `timestamp` field exists after defaults and CLI fields. | -| 5 | Serialize the resulting object. | The merged record is written as one compact JSON object. | +| Step | Rule | Winner | +| ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | +| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | +| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | +| 4 | Validate an override timestamp, if present. | A sidecar or CLI `timestamp` must match the canonical RFC 3339 UTC format in the event schema; an invalid override returns `EX_DATAERR` before append. | +| 5 | Insert the generated invocation timestamp only when no override exists. | The generated canonical UTC `timestamp` is added only when defaults and CLI fields did not produce one. | +| 6 | Serialize the resulting object. | The merged record is written as one compact JSON object. | The sidecar schema never overrides an explicit CLI coercion flag. Schema entries affect only values supplied for the matching object path; they do not @@ -305,9 +306,11 @@ The scheduled protocol is: 2. Determine the active file period from the first complete record in the active log, falling back to the pending period when the active log is empty. 3. If the active period is older than the pending record period, finalize the - previous period first by rotating the active log into the active period's - scheduled filename. The active path is then a fresh empty file for the - pending period. + previous period first. Scan that period's existing archives: use its + unsuffixed scheduled filename only when no size-split archive exists and the + unsuffixed filename is unused; otherwise use the next unused positive + numeric suffix. Rotate the active log into that collision-free filename. The + active path is then a fresh empty file for the pending period. 4. Evaluate `max_bytes` against the current active file after any period-boundary rollover. If that active file plus the pending record would exceed `max_bytes`, rotate it into the pending period's next size-split @@ -316,11 +319,11 @@ The scheduled protocol is: Within a scheduled period, reaching `max_bytes` before the time boundary creates an interim size split. Size splits add a numeric suffix within that -period: `run..1.jsonl`, `run..2.jsonl`, and so on. If a period -has no size splits, its final scheduled archive uses the unsuffixed base name, -such as `run..jsonl`. If a period already has one or more size-split -files, the final archive at the next time boundary also uses the next numeric -suffix so that every file for that period has an ordered generation number. +period: `run..1.jsonl`, `run..2.jsonl`, and so on. The final +scheduled archive uses `run..jsonl` only when that filename is unused +and the period has no size splits. Otherwise it uses the next unused positive +numeric suffix, so it never overwrites an archive or collides with +`run..1.jsonl`. Retention and compression are period-based in scheduled mode. The newest four completed periods remain plain, including all size-split files inside those @@ -337,8 +340,9 @@ invocation before appending the new record. The normative JSON Schema lives in [mpsc-log-event-schema.json](mpsc-log-event-schema.json). The schema requires -`timestamp`, permits additional fields, and reserves structured namespaces for -`run`, `task`, `attempt`, `coderabbit`, `audit`, and `defect`. +`timestamp`, conditionally requires the table's fields for each named event, +permits additional fields, and reserves structured namespaces for `run`, `task`, +`attempt`, `coderabbit`, `audit`, and `defect`. The df12-build integration should emit these event names first: diff --git a/docs/mpsc-log-event-schema.json b/docs/mpsc-log-event-schema.json index 58c0034..fd2385a 100644 --- a/docs/mpsc-log-event-schema.json +++ b/docs/mpsc-log-event-schema.json @@ -80,5 +80,73 @@ "severity": { "type": "string" } } } - } + }, + "allOf": [ + { + "if": { "properties": { "event": { "const": "phase.started" } }, "required": ["event"] }, + "then": { + "required": ["run", "attempt"], + "properties": { + "run": { "required": ["id"] }, + "attempt": { "required": ["phase", "agent"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "phase.finished" } }, "required": ["event"] }, + "then": { + "required": ["run", "attempt"], + "properties": { + "run": { "required": ["id"] }, + "attempt": { "required": ["phase", "status", "duration_ms"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "review.round" } }, "required": ["event"] }, + "then": { + "required": ["task", "attempt"], + "properties": { + "task": { "required": ["id"] }, + "attempt": { "required": ["phase", "round", "status"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "task.finished" } }, "required": ["event"] }, + "then": { + "required": ["task"], + "properties": { + "task": { "required": ["id", "phase", "work_items", "changed_files", "dependency_depth"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "coderabbit.attempt" } }, "required": ["event"] }, + "then": { + "required": ["coderabbit"], + "properties": { + "coderabbit": { "required": ["status", "wait_ms", "retry", "deferred"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "audit.finding" } }, "required": ["event"] }, + "then": { + "required": ["audit"], + "properties": { + "audit": { "required": ["severity", "lane", "finding_count"] } + } + } + }, + { + "if": { "properties": { "event": { "const": "defect.escape" } }, "required": ["event"] }, + "then": { + "required": ["defect"], + "properties": { + "defect": { "required": ["source", "escaped_from", "severity"] } + } + } + } + ] } diff --git a/docs/roadmap.md b/docs/roadmap.md index 55e0e68..0365b34 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -151,8 +151,9 @@ generated timestamp can merge deterministically. Its outcome informs the external JSONL contract and df12-build event examples. See mpsc-log-design.md §§2, 6, 9, 11-12, mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. -- [ ] 2.2.1. Implement sidecar path derivation, TOML loading, and semantic - validation. +- [ ] 2.2.1. Implement sidecar path derivation, TOML v1.0 parsing, and + semantic validation for the `[rotation]`, `[locking]`, `[defaults]`, and + `[schema]` tables. - Requires 2.1.1. - See mpsc-log-design.md §§2, 6, 10 and mpsc-log-sidecar.example.toml. @@ -164,15 +165,13 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md - See mpsc-log-design.md §§6, 9, 11 and mpsc-log-event-schema.json. - Success: sidecar defaults seed the record; CLI fields win over defaults - and earlier duplicate paths; CLI coercion uses explicit `-s`, `-n`, or - `-b` flags before schema entries and default `jo`-inspired inference; and - the generated `timestamp` is inserted only if no timestamp exists after - defaults and CLI fields. + and earlier duplicate paths; and CLI coercion uses explicit `-s`, `-n`, or + `-b` flags before schema entries and default `jo`-inspired inference. - [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. - Requires 1.2.3 and 2.2.2. - See mpsc-log-design.md §§2, 6 and terms-of-reference.md §§2, 6, 8. - - Success: records receive an invocation-time UTC timestamp unless a prior - merge step already produced `timestamp`. + - Success: records receive an invocation-time UTC timestamp unless defaults + or CLI fields already produced `timestamp`. ### 2.3. Append the first useful JSONL record @@ -227,8 +226,11 @@ critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, 7 - [ ] 3.1.2. Implement exclusive journal locking with a configurable timeout. - Requires 3.1.1. - See mpsc-log-design.md §§2, 4, 6-7, 10 and context.md. - - Success: contending processes serialize through the same lock file, and - lock timeout failures return `EX_TEMPFAIL`. + - Success: a pure function derives the lock path by appending `.lock` to the + complete journal filename in the same directory, rejects caller-supplied + `.lock` journal filenames, and gives `run` and `run.jsonl` distinct locks. + Tests cover those documented names; contending processes serialize through + the same lock file; and lock timeout failures return `EX_TEMPFAIL`. - [ ] 3.1.3. Read sidecar configuration inside the journal critical section. - Requires 3.1.2. - See mpsc-log-design.md §§4, 6-7 and terms-of-reference.md §8.2. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index c892da0..c57e825 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -165,8 +165,11 @@ artefact rather than trusting a caller's informal summary. - Optionally rotate on hourly, daily, or weekly UTC time boundaries, with interim size splits if the active log reaches the size threshold before the next boundary. -- Compress rotated logs after the fourth rotation, subject to the final naming - and retention policy. +- For size-only rotation, retain the newest four rotated generations as plain + files and gzip older retained generations. +- For scheduled rotation, retain every size-split file in the newest four + completed periods as plain files, gzip only older retained periods, and never + compress files in the current period. - Read a sidecar TOML file for rotation configuration, schema-guided type coercion, and default field values. - Surface failures through stable exit codes and diagnostics suitable for @@ -310,11 +313,7 @@ artefact rather than trusting a caller's informal summary. | Question | Why it matters | Criteria for resolution | Suggested path | | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- | -| Which exact subset of `jo` syntax is in scope? | `jo` includes arrays, file-value operators, duplicate-key behaviour, object paths, and coercion flags that may conflict with an object-root log contract. | The design records accepted, rejected, and modified syntax with examples. | Resolved by ADR 003. | -| What are the sidecar precedence rules? | Defaults, schema coercion, CLI values, and `timestamp` overrides can conflict. | A precedence table defines every conflict outcome. | Technical design. | -| How should duplicate keys resolve? | JSON permits duplicate object names textually, but consumers often collapse them. | The product chooses reject, last-wins, first-wins, or compatibility behaviour. | Resolved by ADR 003. | | What does "gzipping after 4 rotations" mean exactly? | Retention, naming, and compression timing affect concurrency and user expectations. | The rotation policy specifies filenames, retention count, compression trigger, and whether recent rotations remain plain text. | Technical design. | -| What lock protects rotation and sidecar reads? | Append-only locking may not be enough when a process rotates while others are opening or creating files. | The design names the lock artefact and the critical sections it protects. | Technical design and stress tests. | | What platforms are supported at v1? | File-locking and atomic rename semantics differ across Unix, Windows, and network filesystems. | The project declares supported platforms and test coverage. | ADR candidate. | | What are the stable exit codes? | Agents need to distinguish retryable timeout from invalid input or corrupted configuration. | The user guide lists exit codes and diagnostics. | User-guide update during implementation. | | Is a Rust library API part of the product? | A public library API expands compatibility and documentation obligations. | The roadmap states whether the crate is CLI-only or also exposes supported library functions. | Product decision. | From 63b131fe447cb6f40585e53439f760bce6dfa56b Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 21 Jul 2026 01:01:29 +0200 Subject: [PATCH 14/27] Refine rollover prose and sidecar derivation roadmap Address review feedback on the design and roadmap documents: - Add a comma after "Otherwise" in the scheduled-rollover final-archive naming sentence in docs/mpsc-log-design.md. - Expand roadmap item 2.2.1 to specify extension-replacement sidecar derivation (run, run.jsonl, and run.ndjson derive run.toml) and add an acceptance criterion plus tests rejecting .toml journal filenames whose derived sidecar path equals the journal path, while preserving the existing defaults and error-code requirements. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/mpsc-log-design.md | 2 +- docs/roadmap.md | 18 ++++++++++++------ 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 6884c57..3724202 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -321,7 +321,7 @@ Within a scheduled period, reaching `max_bytes` before the time boundary creates an interim size split. Size splits add a numeric suffix within that period: `run..1.jsonl`, `run..2.jsonl`, and so on. The final scheduled archive uses `run..jsonl` only when that filename is unused -and the period has no size splits. Otherwise it uses the next unused positive +and the period has no size splits. Otherwise, it uses the next unused positive numeric suffix, so it never overwrites an archive or collides with `run..1.jsonl`. diff --git a/docs/roadmap.md b/docs/roadmap.md index 0365b34..9979fb5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -151,14 +151,20 @@ generated timestamp can merge deterministically. Its outcome informs the external JSONL contract and df12-build event examples. See mpsc-log-design.md §§2, 6, 9, 11-12, mpsc-log-sidecar.example.toml, and mpsc-log-event-schema.json. -- [ ] 2.2.1. Implement sidecar path derivation, TOML v1.0 parsing, and - semantic validation for the `[rotation]`, `[locking]`, `[defaults]`, and - `[schema]` tables. +- [ ] 2.2.1. Implement sidecar path derivation by extension replacement, TOML + v1.0 parsing, and semantic validation for the `[rotation]`, `[locking]`, + `[defaults]`, and `[schema]` tables. - Requires 2.1.1. - - See mpsc-log-design.md §§2, 6, 10 and + - See mpsc-log-design.md §§2, 6, 10, adr-001-lock-file-naming.md, and mpsc-log-sidecar.example.toml. - - Success: missing sidecars use defaults, malformed TOML returns - `EX_DATAERR`, and semantically invalid configuration returns `EX_CONFIG`. + - Success: derivation replaces the journal filename extension with `.toml`, + appending `.toml` when the journal has no extension, so `run`, `run.jsonl`, + and `run.ndjson` each derive `run.toml`; missing sidecars use defaults; + malformed TOML returns `EX_DATAERR`; and semantically invalid configuration + returns `EX_CONFIG`. + - Success: a `.toml` journal filename whose derived sidecar path equals the + journal path is rejected with `EX_USAGE`. Tests cover the shared-stem + derivations and reject the self-sidecar `.toml` journal case. - [ ] 2.2.2. Implement deterministic record merging and schema-guided coercion. - Requires 2.1.3 and 2.2.1. From 2c9b09bce8d377bfdc96d2e34b6cbb8e9a851b99 Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 21 Jul 2026 10:55:14 +0200 Subject: [PATCH 15/27] Guard empty active file and clarify retention scope Address a second round of review feedback on the design, roadmap, and terms of reference: - Guard the scheduled size-split step so an empty active file is never rotated into a spurious empty archive: when the active file holds no data, append the oversized pending record directly and only rotate a data-bearing active file into a collision-free archive name. - Expand roadmap item 2.2.3 to require validating defaults- and CLI-supplied timestamp values against the RFC 3339 UTC contract before append, preserve valid overrides, fail invalid overrides with EX_DATAERR, and generate an invocation-time timestamp only when no valid one exists. - Scope roadmap item 3.1.2's serialization guarantee to supported local filesystems and defer network-filesystem (NFS and CIFS) support to the named filesystem verification matrix in step 6.2.1, preserving the lock naming, contention, and EX_TEMPFAIL claims. - Clarify in the terms of reference that rotation and compression retention define local generation retention, and scope the non-goal to centralized or downstream retention policy so local retention is clearly in scope. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/mpsc-log-design.md | 8 +++++--- docs/roadmap.md | 14 ++++++++++---- docs/terms-of-reference.md | 8 +++++--- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 3724202..682b308 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -312,9 +312,11 @@ The scheduled protocol is: numeric suffix. Rotate the active log into that collision-free filename. The active path is then a fresh empty file for the pending period. 4. Evaluate `max_bytes` against the current active file after any - period-boundary rollover. If that active file plus the pending record would - exceed `max_bytes`, rotate it into the pending period's next size-split - filename and continue with a fresh active file. + period-boundary rollover. If the active file already contains data and that + data plus the pending record would exceed `max_bytes`, rotate it into the + pending period's next size-split filename and continue with a fresh active + file. When the active file is empty, append the oversized pending record + directly rather than rotating an empty active file into an archive. 5. Append the pending record to the active log. Within a scheduled period, reaching `max_bytes` before the time boundary diff --git a/docs/roadmap.md b/docs/roadmap.md index 9979fb5..60c9426 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -176,8 +176,11 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md - [ ] 2.2.3. Generate the default RFC 3339 UTC `timestamp` field. - Requires 1.2.3 and 2.2.2. - See mpsc-log-design.md §§2, 6 and terms-of-reference.md §§2, 6, 8. - - Success: records receive an invocation-time UTC timestamp unless defaults - or CLI fields already produced `timestamp`. + - Success: defaults- and CLI-supplied `timestamp` values are validated + against the documented RFC 3339 UTC contract before append; valid overrides + are preserved, invalid overrides fail with `EX_DATAERR`, and an + invocation-time RFC 3339 UTC `timestamp` is generated when no valid + timestamp already exists. ### 2.3. Append the first useful JSONL record @@ -235,8 +238,11 @@ critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, 7 - Success: a pure function derives the lock path by appending `.lock` to the complete journal filename in the same directory, rejects caller-supplied `.lock` journal filenames, and gives `run` and `run.jsonl` distinct locks. - Tests cover those documented names; contending processes serialize through - the same lock file; and lock timeout failures return `EX_TEMPFAIL`. + Tests cover those documented names; contending processes on a supported + local filesystem serialize through the same lock file; and lock timeout + failures return `EX_TEMPFAIL`. Network-filesystem support (NFS and CIFS) + remains deferred until the named filesystem verification matrix in step + 6.2.1. - [ ] 3.1.3. Read sidecar configuration inside the journal critical section. - Requires 3.1.2. - See mpsc-log-design.md §§4, 6-7 and terms-of-reference.md §8.2. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index c57e825..dfbe26e 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -165,6 +165,7 @@ artefact rather than trusting a caller's informal summary. - Optionally rotate on hourly, daily, or weekly UTC time boundaries, with interim size splits if the active log reaches the size threshold before the next boundary. +- Define local generation retention on the writer's filesystem, as follows. - For size-only rotation, retain the newest four rotated generations as plain files and gzip older retained generations. - For scheduled rotation, retain every size-split file in the newest four @@ -180,9 +181,10 @@ artefact rather than trusting a caller's informal summary. ### 6.2 Non-goals -- Centralized log ingestion, search, dashboards, alerting, and retention policy - are out of scope; users needing those should ship the JSONL output into an - observability system. +- Centralized log ingestion, search, dashboards, alerting, and centralized or + downstream retention policy are out of scope; users needing those should ship + the JSONL output into an observability system. Local rotation and compression + retention remain in scope, per the goals above. - A long-running daemon or background service is out of scope; the product is a short-lived CLI. - Preserving chronological file order by timestamp is out of scope. The default From 3d8f73dcddfd550843560d4369ed3a909655c75f Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 21 Jul 2026 11:56:24 +0200 Subject: [PATCH 16/27] Tie timestamp generation to override absence Refine roadmap item 2.2.3 so an invocation-time RFC 3339 UTC timestamp is generated only when no `timestamp` override exists, matching the write protocol's step 5. The previous "when no valid timestamp already exists" wording could be read as generating a timestamp after an invalid override, whereas an invalid default or CLI value fails with EX_DATAERR before append. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/roadmap.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 60c9426..a057169 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -179,8 +179,8 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md - Success: defaults- and CLI-supplied `timestamp` values are validated against the documented RFC 3339 UTC contract before append; valid overrides are preserved, invalid overrides fail with `EX_DATAERR`, and an - invocation-time RFC 3339 UTC `timestamp` is generated when no valid - timestamp already exists. + invocation-time RFC 3339 UTC `timestamp` is generated only when no + `timestamp` override exists. ### 2.3. Append the first useful JSONL record From f6a67fb5d2ad0ef49c6aa418361eb46fa1309c33 Mon Sep 17 00:00:00 2001 From: leynos Date: Tue, 21 Jul 2026 20:27:23 +0200 Subject: [PATCH 17/27] Frame rotation counts as defaults and standardise journal terms Address review feedback across the design, roadmap, and terms of reference: - Present the size-only rotation threshold and plain/compressed generation counts in docs/mpsc-log-design.md as the `[rotation]` sidecar defaults (`max_bytes`, `plain_generations`, `compressed_generations`) that callers can override, and use the configured plain and compressed retention counts when describing scheduled-period retention. - Expand roadmap items 4.3.2 and 4.3.3 with observable success conditions: period-boundary rollover runs before the size check, collision-free suffix selection when archives already exist, and appending an oversized record directly to an empty active journal without creating an empty archive. - Reframe the terms-of-reference retention goals so four plain generations and four completed periods read as configurable defaults rather than universal limits, and align the compression hard-constraint bullet. - Standardise the terms of reference on "journal" and "journal path" for the target file and its argument, preserving the JSON Lines (JSONL) format name, the glossary log terms, and the distinct workflow sidecar directory. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/mpsc-log-design.md | 23 ++++++++++------ docs/roadmap.md | 8 ++++++ docs/terms-of-reference.md | 55 ++++++++++++++++++++------------------ 3 files changed, 52 insertions(+), 34 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 682b308..bff4600 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -264,15 +264,21 @@ Rotation runs while holding the journal lock. The sidecar `schedule` value is one of `none`, `hourly`, `daily`, or `weekly`; the default is `none`. There is no `max_age` setting in the v1 configuration surface. -When `schedule = "none"`, rotation is size-only: - -- active threshold: 1 MiB; -- plain generations: four; -- compressed generations: 32; +When `schedule = "none"`, rotation is size-only. The active threshold, plain +generation count, and compressed generation count are the `[rotation]` sidecar +settings `max_bytes`, `plain_generations`, and `compressed_generations`; the +values below are the defaults and are overridable through the sidecar: + +- active threshold (`max_bytes`): 1 MiB; +- plain generations (`plain_generations`): four; +- compressed generations (`compressed_generations`): 32; - newest plain rotation: `.1`; - oldest plain rotation before compression: `.4`; - compressed rotations: `.5.gz` through `.36.gz`. +The rotation names above follow from the default counts; overriding the counts +shifts the highest plain and compressed suffixes accordingly. + For `run.jsonl`, the active path is `run.jsonl`; the newest plain rotation is `run.1.jsonl`; the first compressed rotation is `run.5.jsonl.gz`. @@ -327,9 +333,10 @@ and the period has no size splits. Otherwise, it uses the next unused positive numeric suffix, so it never overwrites an archive or collides with `run..1.jsonl`. -Retention and compression are period-based in scheduled mode. The newest four -completed periods remain plain, including all size-split files inside those -periods. Older retained periods are gzipped, and periods beyond +Retention and compression are period-based in scheduled mode. The newest +`plain_generations` completed periods remain plain, including all size-split +files inside those periods. Older retained periods are gzipped up to +`compressed_generations`, and periods beyond `plain_generations + compressed_generations` are deleted. This keeps hourly, daily, and weekly retention predictable even when one busy period produces many size-split files. diff --git a/docs/roadmap.md b/docs/roadmap.md index a057169..332fd99 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -375,11 +375,19 @@ and context.md. - See mpsc-log-design.md §8. - Success: the next invocation after a time break archives the previous active segment under the period that produced its records. + - Success: a unit or behavioural test shows period-boundary rollover runs + before the `max_bytes` size check, so the size split is evaluated against + the fresh active file for the pending period. - [ ] 4.3.3. Implement interim size-split suffixes inside scheduled periods. - Requires 4.3.2. - See mpsc-log-design.md §8. - Success: busy periods produce ordered `.n` suffixes, and final scheduled archives use the next suffix when a period already has size splits. + - Success: a unit or behavioural test covers collision-free suffix selection + when archives for the period already exist, so rotation never overwrites an + existing archive. + - Success: a unit or behavioural test covers appending an oversized record + directly to an empty active journal without creating an empty archive. - [ ] 4.3.4. Implement scheduled-mode period retention and compression. - Requires 4.2.3 and 4.3.3. - See mpsc-log-design.md §8. diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index dfbe26e..6905757 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -18,12 +18,12 @@ ## 1. Background and motivation `mpsc-log` exists for multi-agent and scripted workflows that need several -independent processes to record structured events in one append-only log file. +independent processes to record structured events in one append-only journal. The immediate problem is not creating JSON. Tools such as `jo` already turn shell arguments into JSON objects.[^1] The gap is that agents also need a small -command-line interface (CLI) that appends each record to a shared JSON Lines -(JSONL) file without overwriting existing data, losing concurrent writes, or -corrupting the file during rotation. +command-line interface (CLI) that appends each record as a JSON Lines (JSONL) +entry to a shared journal without overwriting existing data, losing concurrent +writes, or corrupting the journal during rotation. The motivating context is local automation where several agents may run at the same time and cannot coordinate through a long-lived service. Each agent should @@ -51,7 +51,7 @@ records are JSON objects serialized as one JSON value per line. JSONL keeps the log easy to append, stream, grep, split, and ingest into later tools without requiring the whole file to be rewritten. -The command accepts the log path as its first argument. Later arguments +The command accepts the journal path as its first argument. Later arguments describe fields using selected `jo`-inspired key and value words, including type coercion flags and object paths. The CLI is not textually compatible with all `jo` output: the root value must always be an object, and duplicate writes @@ -61,8 +61,8 @@ The default entry includes a `timestamp` field containing a Coordinated Universal Time (UTC) timestamp captured when the command is invoked. The timestamp uses RFC 3339, the Internet timestamp profile of ISO 8601.[^2] -The tool also reads a sidecar TOML file next to the log file. The sidecar has -the same base filename as the log file and a `.toml` extension. It defines +The tool also reads a sidecar TOML file next to the journal. The sidecar has +the same base filename as the journal and a `.toml` extension. It defines rotation configuration, type-coercion schema, and default field values. The term "sidecar" has two relevant meanings in the current domain. The ODW @@ -106,7 +106,7 @@ The current alternatives each solve part of the problem: than a per-write safety mechanism integrated with concurrent appends. The gap is a narrowly scoped CLI that combines selected `jo`-inspired record -construction with safe append and rotation behaviour for one local JSONL file. +construction with safe append and rotation behaviour for one local journal. ## 4. Users and stakeholders @@ -125,7 +125,7 @@ Table 1: Stakeholder mapping for the initial product boundary. When an agent workflow launches several independent command-line processes, the workflow author wants each process to append a structured event to the same -local log file, so they can inspect the run later without reconstructing events +local journal, so they can inspect the run later without reconstructing events from scattered output. When a df12-build ODW run processes roadmap tasks through planning, review, @@ -144,7 +144,7 @@ artefact rather than trusting a caller's informal summary. ### 6.1 Goals -- Accept a log-file path as the first CLI argument and interpret following +- Accept a journal path as the first CLI argument and interpret following arguments as record fields. - Support the selected `jo`-inspired key/value syntax needed for object records, including type coercion flags and object paths. @@ -156,21 +156,23 @@ artefact rather than trusting a caller's informal summary. invocation. - Prevent concurrent calls from overwriting existing logs or colliding with each other while creating, appending to, or rotating the file. -- Create missing parent directories for the log-file path automatically when +- Create missing parent directories for the journal path automatically when filesystem permissions allow it. - Use locking and safe writes with a five-second default timeout. -- Gracefully handle simultaneous attempts to create the log file and sidecar - coordination artefacts. +- Gracefully handle simultaneous attempts to create the journal file and + sidecar coordination artefacts. - Rotate by default after the active log reaches 1 MiB. - Optionally rotate on hourly, daily, or weekly UTC time boundaries, with interim size splits if the active log reaches the size threshold before the next boundary. - Define local generation retention on the writer's filesystem, as follows. -- For size-only rotation, retain the newest four rotated generations as plain - files and gzip older retained generations. -- For scheduled rotation, retain every size-split file in the newest four - completed periods as plain files, gzip only older retained periods, and never - compress files in the current period. +- For size-only rotation, retain the newest rotated generations (four by + default, via `plain_generations`) as plain files, and gzip older + retained generations (per `compressed_generations`). +- For scheduled rotation, retain every size-split file in the newest + completed periods (four by default, via `plain_generations`) as plain + files, gzip only older retained periods (per `compressed_generations`), + and never compress files in the current period. - Read a sidecar TOML file for rotation configuration, schema-guided type coercion, and default field values. - Surface failures through stable exit codes and diagnostics suitable for @@ -213,10 +215,10 @@ artefact rather than trusting a caller's informal summary. basic type coercions. - Stress tests with many concurrent invocations produce the same number of valid JSONL records as successful command exits. -- Simultaneous first writes to a missing log file create exactly one usable log +- Simultaneous first writes to a missing journal create exactly one usable log and do not truncate, replace, or interleave records. -- A first write to a log path in a missing directory tree creates the required - parent directories when permissions allow it. +- A first write to a journal path in a missing directory tree creates the + required parent directories when permissions allow it. - Rotation during concurrent writes leaves every successful entry in either the active file or a rotated file. - A df12-build run can record one journal entry per meaningful phase, review @@ -251,7 +253,7 @@ artefact rather than trusting a caller's informal summary. ### 8.1 Hard constraints -- The first CLI parameter is the log-file path. +- The first CLI parameter is the journal path. - Later parameters are key/value words using the accepted `jo`-inspired syntax. - The root record must be a JSON object. @@ -260,12 +262,13 @@ artefact rather than trusting a caller's informal summary. - The default scheduled rotation policy is `none`. - Scheduled rotation modes are `hourly`, `daily`, and `weekly`; they use UTC period boundaries and do not imply a `max_age` retention setting. -- Rotated logs are compressed after four rotations. -- The sidecar configuration file is TOML and derives its path from the log file - path by replacing the filename extension with `.toml`. +- Rotated logs are compressed after the configured plain generation count, + four by default. +- The sidecar configuration file is TOML and derives its path from the + journal path by replacing the filename extension with `.toml`. - The default record includes `timestamp` unless the final precedence rules say otherwise. -- Missing parent directories for the log-file path are created automatically +- Missing parent directories for the journal path are created automatically when permissions allow it. - The tool must tolerate concurrent writers and concurrent initial file creation. From 76fbf2df4b6fde57b7ecd93100ac2058eefc7c5a Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 22 Jul 2026 22:07:12 +0200 Subject: [PATCH 18/27] Parameterise size-only rotation by configured generation counts Address review feedback on the rotation normative steps and sidecar naming: - Rewrite the size-only rotation algorithm in docs/mpsc-log-design.md so every generation index derives from the configured plain generation count P (`plain_generations`) and compressed generation count C (`compressed_generations`) rather than the hard-coded 36, 5, 4, 3, 2, and 1. Define both counts as non-negative integers (negatives rejected with EX_CONFIG) and specify the zero-count behaviour: P = 0 gzips the active file straight into generation 1, C = 0 deletes the oldest plain generation instead of gzipping, and both zero retains only the active file. Deletion, renaming, compression, and retention now target the computed generations. - Clarify the terms-of-reference sidecar filename description to state that the sidecar path replaces the journal's final extension with `.toml`, appending `.toml` only when the journal has no extension, replacing the ambiguous "same base filename" wording. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/mpsc-log-design.md | 52 +++++++++++++++++++++++++++----------- docs/terms-of-reference.md | 7 ++--- 2 files changed, 41 insertions(+), 18 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index bff4600..454185e 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -264,17 +264,20 @@ Rotation runs while holding the journal lock. The sidecar `schedule` value is one of `none`, `hourly`, `daily`, or `weekly`; the default is `none`. There is no `max_age` setting in the v1 configuration surface. -When `schedule = "none"`, rotation is size-only. The active threshold, plain -generation count, and compressed generation count are the `[rotation]` sidecar -settings `max_bytes`, `plain_generations`, and `compressed_generations`; the -values below are the defaults and are overridable through the sidecar: +When `schedule = "none"`, rotation is size-only. The active threshold +(`max_bytes`), the plain generation count `P` (`plain_generations`), and the +compressed generation count `C` (`compressed_generations`) are `[rotation]` +sidecar settings; the values below are the defaults and are overridable through +the sidecar: - active threshold (`max_bytes`): 1 MiB; -- plain generations (`plain_generations`): four; -- compressed generations (`compressed_generations`): 32; +- plain generations `P` (`plain_generations`): four; +- compressed generations `C` (`compressed_generations`): 32; - newest plain rotation: `.1`; -- oldest plain rotation before compression: `.4`; -- compressed rotations: `.5.gz` through `.36.gz`. +- oldest plain rotation before compression: `.P` (`.4` + by default); +- compressed rotations: `.(P+1).gz` through `.(P+C).gz` + (`.5.gz` through `.36.gz` by default). The rotation names above follow from the default counts; overriding the counts shifts the highest plain and compressed suffixes accordingly. @@ -282,15 +285,34 @@ shifts the highest plain and compressed suffixes accordingly. For `run.jsonl`, the active path is `run.jsonl`; the newest plain rotation is `run.1.jsonl`; the first compressed rotation is `run.5.jsonl.gz`. +`plain_generations` and `compressed_generations` are non-negative integers that +count retained archive generations; the active file is always separate. +Generation `1` is the newest archive and generation `P + C` is the oldest, so +the indices below derive from `P` and `C` rather than fixed numbers. Semantic +validation rejects negative counts with `EX_CONFIG`. + +Zero counts are valid. When `P` is zero, no plain archives exist and the active +file is gzipped straight into generation `1`. When `C` is zero, no archive is +compressed and the oldest plain generation is deleted rather than gzipped. When +both are zero, rotation retains only the active file and discards the previous +contents. + The size-only rotation order is oldest-to-newest: -1. Delete generation 36 if present. -2. Rename compressed generations upward. -3. Gzip generation 4 into generation 5 and remove generation 4 only after the - gzip output is complete. -4. Rename plain generations 3 to 4, 2 to 3, and 1 to 2. -5. Rename the active file to generation 1. -6. Create a fresh active file by appending the pending record. +1. Delete generation `P + C` if present. +2. Rename each compressed generation `i` from `P + C - 1` down to `P + 1` into + generation `i + 1`. +3. Gzip plain generation `P` into generation `P + 1`, removing generation `P` + only after the gzip output is complete. Skip this step when `C` is zero + (generation `P` was already deleted in step 1) or when `P` is zero (the + active file is compressed in step 5 instead). +4. Rename each plain generation `i` from `P - 1` down to `1` into generation + `i + 1`. +5. Rename the active file to generation `1` when `P` is at least one; when `P` + is zero and `C` is at least one, gzip the active file into generation `1`. +6. Create a fresh active file by appending the pending record. When both `P` and + `C` are zero there is no archive generation, so the previous active contents + are discarded. When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period boundaries derived from the invocation timestamp: diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index 6905757..0d4ed07 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -61,9 +61,10 @@ The default entry includes a `timestamp` field containing a Coordinated Universal Time (UTC) timestamp captured when the command is invoked. The timestamp uses RFC 3339, the Internet timestamp profile of ISO 8601.[^2] -The tool also reads a sidecar TOML file next to the journal. The sidecar has -the same base filename as the journal and a `.toml` extension. It defines -rotation configuration, type-coercion schema, and default field values. +The tool also reads a sidecar TOML file whose path is derived from the journal +path: it replaces the journal's final extension with `.toml`, or appends +`.toml` when the journal has no extension. It defines rotation configuration, +type-coercion schema, and default field values. The term "sidecar" has two relevant meanings in the current domain. The ODW workflow has a workflow sidecar directory where run artefacts belong. From 51de2f5fd2256d8eac2cd8120b1ed4e216475bf0 Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 29 Jul 2026 14:28:05 +0200 Subject: [PATCH 19/27] Reject zero-retention rotation and close rotation failure windows The size-only rotation rules allowed plain_generations and compressed_generations to both be zero, in which case rotation discarded the previous journal. That state transition cannot be made atomic and contradicts the terms-of-reference constraint that a failed invocation leaves the previous journal readable. - Require `P + C >= 1` in docs/mpsc-log-design.md, so semantic validation rejects zero total retention with EX_CONFIG alongside negative counts, and explain why discarding the rotated journal is a configuration error rather than a supported mode. A single zero count remains valid. - Make step 5 state that the previous contents always reach generation 1 before any append: a rename when P >= 1, and otherwise a gzip whose active file is removed only after the output is committed, so a failed compression aborts with the previous journal intact. - Note in step 6 that an append failure therefore leaves the previous contents readable in the rotated journal. - Add a roadmap 2.2.1 acceptance criterion covering both rejected count configurations. Co-Authored-By: Claude Opus 5 (1M context) --- docs/mpsc-log-design.md | 31 +++++++++++++++++++------------ docs/roadmap.md | 4 ++++ 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 454185e..c4a5c88 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -289,13 +289,17 @@ For `run.jsonl`, the active path is `run.jsonl`; the newest plain rotation is count retained archive generations; the active file is always separate. Generation `1` is the newest archive and generation `P + C` is the oldest, so the indices below derive from `P` and `C` rather than fixed numbers. Semantic -validation rejects negative counts with `EX_CONFIG`. - -Zero counts are valid. When `P` is zero, no plain archives exist and the active -file is gzipped straight into generation `1`. When `C` is zero, no archive is -compressed and the oldest plain generation is deleted rather than gzipped. When -both are zero, rotation retains only the active file and discards the previous -contents. +validation rejects negative counts with `EX_CONFIG`, and also rejects +`P + C == 0`: rotation must retain at least one generation. A configuration +that would leave rotation with nowhere to put the previous journal is a +configuration error rather than a supported mode, because discarding the +rotated journal cannot be made atomic and would contradict the guarantee that a +failed invocation leaves the previous journal readable. + +One zero count is valid because `P + C` is still at least one. When `P` is +zero, no plain archives exist and the active file is gzipped straight into +generation `1`. When `C` is zero, no archive is compressed and the oldest plain +generation is deleted rather than gzipped. The size-only rotation order is oldest-to-newest: @@ -308,11 +312,14 @@ The size-only rotation order is oldest-to-newest: active file is compressed in step 5 instead). 4. Rename each plain generation `i` from `P - 1` down to `1` into generation `i + 1`. -5. Rename the active file to generation `1` when `P` is at least one; when `P` - is zero and `C` is at least one, gzip the active file into generation `1`. -6. Create a fresh active file by appending the pending record. When both `P` and - `C` are zero there is no archive generation, so the previous active contents - are discarded. +5. Move the active file into generation `1`. When `P` is at least one this is a + rename, which is atomic on a supported local filesystem. When `P` is zero, + gzip the active file into generation `1` instead, and remove the active file + only after that gzip output is committed, so a failed compression leaves the + previous journal in place and aborts before any append. +6. Create a fresh active file by appending the pending record. Because step 5 + always commits the previous contents to generation `1` first, a failure here + leaves those contents readable in the rotated journal. When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period boundaries derived from the invocation timestamp: diff --git a/docs/roadmap.md b/docs/roadmap.md index 332fd99..e4bb988 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -165,6 +165,10 @@ external JSONL contract and df12-build event examples. See mpsc-log-design.md - Success: a `.toml` journal filename whose derived sidecar path equals the journal path is rejected with `EX_USAGE`. Tests cover the shared-stem derivations and reject the self-sidecar `.toml` journal case. + - Success: `[rotation]` generation counts are validated as non-negative and + as retaining at least one generation, so a negative count and + `plain_generations = 0` with `compressed_generations = 0` each return + `EX_CONFIG`. - [ ] 2.2.2. Implement deterministic record merging and schema-guided coercion. - Requires 2.1.3 and 2.2.1. From eee8fac151808f819759341bdef4e4b2c1813fe9 Mon Sep 17 00:00:00 2001 From: leynos Date: Wed, 29 Jul 2026 14:31:49 +0200 Subject: [PATCH 20/27] Defer the oldest-generation eviction to the rotation commit point Size-only rotation deleted generation P + C as its first act, so a later failure in the rename or gzip sequence had already destroyed the oldest archive and left a partially shifted layout. That also contradicted roadmap item 4.2.3, which requires eviction only after newer retained files are safely in place. - Stage the eviction instead of deleting it: rename generation P + C aside to a reserved staging name in the same directory, so the rename stays within one filesystem and is atomic. - Unlink the staged eviction only in a new final commit step, after the pending record has been appended, so no failure path deletes a record-bearing file. - State the failure guarantee explicitly: steps 1 to 5 are atomic renames and atomic-write-file commits, so a failure aborts before the append and unwinds the completed renames in reverse order to the prior generation layout, and a process killed mid-rotation leaves a staged eviction that the next invocation reclaims under the journal lock. - Update step 3's C = 0 note, which previously referred to the deletion. - Add a roadmap 4.2.3 acceptance criterion for staging, commit-point unlinking, and staged-eviction reclaim. Co-Authored-By: Claude Opus 5 (1M context) --- docs/mpsc-log-design.md | 31 +++++++++++++++++++++---------- docs/roadmap.md | 3 +++ 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index c4a5c88..85bde75 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -301,25 +301,36 @@ zero, no plain archives exist and the active file is gzipped straight into generation `1`. When `C` is zero, no archive is compressed and the oldest plain generation is deleted rather than gzipped. -The size-only rotation order is oldest-to-newest: +The size-only rotation order is oldest-to-newest. No record-bearing file is +unlinked until the invocation commits: -1. Delete generation `P + C` if present. +1. Stage the eviction. If generation `P + C` exists, rename it aside to a + reserved staging name in the same directory rather than deleting it. The + staging name shares the journal directory so the rename stays within one + filesystem and is therefore atomic. 2. Rename each compressed generation `i` from `P + C - 1` down to `P + 1` into generation `i + 1`. 3. Gzip plain generation `P` into generation `P + 1`, removing generation `P` - only after the gzip output is complete. Skip this step when `C` is zero - (generation `P` was already deleted in step 1) or when `P` is zero (the - active file is compressed in step 5 instead). + only after the gzip output is committed. Skip this step when `C` is zero + (generation `P` is the staged eviction) or when `P` is zero (the active file + is compressed in step 5 instead). 4. Rename each plain generation `i` from `P - 1` down to `1` into generation `i + 1`. 5. Move the active file into generation `1`. When `P` is at least one this is a rename, which is atomic on a supported local filesystem. When `P` is zero, gzip the active file into generation `1` instead, and remove the active file - only after that gzip output is committed, so a failed compression leaves the - previous journal in place and aborts before any append. -6. Create a fresh active file by appending the pending record. Because step 5 - always commits the previous contents to generation `1` first, a failure here - leaves those contents readable in the rotated journal. + only after that gzip output is committed. +6. Create a fresh active file by appending the pending record. +7. Commit the rotation by unlinking the staged eviction, once the append has + succeeded. + +Steps 1 to 5 are atomic renames and `atomic-write-file` commits, so a failure in +any of them aborts the invocation before the append and unwinds the renames it +has already made, in reverse order, back to the prior generation layout. A +failure in step 6 leaves the previous contents readable in generation `1`. +Because the eviction is unlinked only in step 7, no failure path deletes a +record-bearing file. A process killed mid-rotation can leave a staged eviction +file behind; the next invocation reclaims it while holding the journal lock. When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period boundaries derived from the invocation timestamp: diff --git a/docs/roadmap.md b/docs/roadmap.md index e4bb988..c5a52f7 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -358,6 +358,9 @@ mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. - See mpsc-log-design.md §§6, 8. - Success: files beyond `plain_generations + compressed_generations` are deleted only after newer retained files are safely in place. + - Success: the evicted oldest generation is staged aside rather than deleted, + is unlinked only after the append commits, and a staged eviction left by a + killed process is reclaimed by the next invocation under the journal lock. ### 4.3. Deliver UTC scheduled rotation with size splits From 334e53d7fac482896226d550e96b6f2d4b5ed116 Mon Sep 17 00:00:00 2001 From: leynos Date: Fri, 14 Aug 2026 22:37:51 +0200 Subject: [PATCH 21/27] Make rotation recovery settle on one layout, never a mixture Step 3 removed plain generation P once its gzip was committed, so the claimed unwind "back to the prior generation layout" could not hold: a reverse rename cannot restore a removed plain source. Interruption also had no restart rule beyond reclaiming the staged eviction, leaving a partially shifted layout unspecified. - Defer every removal to the commit point. Superseded sources, including the gzipped plain generation P and the active file when P is zero, are renamed aside to staging names instead of being unlinked, so the prepare phase only creates and renames files and every source generation stays recoverable. - Record the planned layout transition in a rotation manifest written through atomic-write-file before any file is touched, and remove it at the commit point, so its presence means a rotation was interrupted. - State that each prepare step is idempotent, applied only when its source exists and its target does not, so replaying the manifest can neither duplicate nor skip a generation. - Specify the three failure paths so rotation always settles into either the pre-rotation or the post-rotation layout: a prepare error reverses the applied renames, an append error still commits because the rotation itself completed, and a killed process leaves the manifest for the next invocation to complete or reverse under the journal lock before handling its own record. - Expand roadmap item 4.2.3 to fault-inject each rotation commit point and assert recovery: partial renames, uncommitted and committed gzip output, staged source removal, append failure, and process interruption. Steps 1 to 7 are now one unbroken list so the phase split no longer trips MD029; the phase boundary is stated in the lead-in instead. Co-Authored-By: Claude Opus 5 (1M context) --- docs/mpsc-log-design.md | 67 +++++++++++++++++++++++++++-------------- docs/roadmap.md | 14 +++++++-- 2 files changed, 56 insertions(+), 25 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 85bde75..b697f87 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -299,38 +299,61 @@ failed invocation leaves the previous journal readable. One zero count is valid because `P + C` is still at least one. When `P` is zero, no plain archives exist and the active file is gzipped straight into generation `1`. When `C` is zero, no archive is compressed and the oldest plain -generation is deleted rather than gzipped. +generation is evicted rather than gzipped. -The size-only rotation order is oldest-to-newest. No record-bearing file is -unlinked until the invocation commits: +Size-only rotation is a prepare phase followed by a commit point. The prepare +phase only creates files and renames them within the journal directory, so +every source generation stays on disk and recoverable until the rotation +commits. Superseded sources are renamed aside to staging names rather than +unlinked; staging names and the manifest below are reserved names derived from +the journal filename, alongside the `.lock` name from ADR 001. + +Before touching any file, the invocation writes a rotation manifest recording +the planned layout transition. The manifest is written through +`atomic-write-file` in the journal directory and is removed at the commit +point, so its presence means a rotation was interrupted. + +Steps 1 to 5 prepare the new layout, oldest-to-newest; steps 6 and 7 commit it: 1. Stage the eviction. If generation `P + C` exists, rename it aside to a - reserved staging name in the same directory rather than deleting it. The - staging name shares the journal directory so the rename stays within one - filesystem and is therefore atomic. + staging name. Staging names share the journal directory, so every rename + stays within one filesystem and is atomic. 2. Rename each compressed generation `i` from `P + C - 1` down to `P + 1` into generation `i + 1`. -3. Gzip plain generation `P` into generation `P + 1`, removing generation `P` - only after the gzip output is committed. Skip this step when `C` is zero - (generation `P` is the staged eviction) or when `P` is zero (the active file - is compressed in step 5 instead). +3. Gzip plain generation `P` into generation `P + 1`, then rename generation + `P` aside to a staging name rather than unlinking it. Skip this step when + `C` is zero (generation `P` is already the staged eviction) or when `P` is + zero (the active file is compressed in step 5 instead). 4. Rename each plain generation `i` from `P - 1` down to `1` into generation `i + 1`. 5. Move the active file into generation `1`. When `P` is at least one this is a rename, which is atomic on a supported local filesystem. When `P` is zero, - gzip the active file into generation `1` instead, and remove the active file - only after that gzip output is committed. + gzip the active file into generation `1` and then rename the active file + aside to a staging name. 6. Create a fresh active file by appending the pending record. -7. Commit the rotation by unlinking the staged eviction, once the append has - succeeded. - -Steps 1 to 5 are atomic renames and `atomic-write-file` commits, so a failure in -any of them aborts the invocation before the append and unwinds the renames it -has already made, in reverse order, back to the prior generation layout. A -failure in step 6 leaves the previous contents readable in generation `1`. -Because the eviction is unlinked only in step 7, no failure path deletes a -record-bearing file. A process killed mid-rotation can leave a staged eviction -file behind; the next invocation reclaims it while holding the journal lock. +7. Unlink the staged files, then remove the manifest. + +Every prepare step is idempotent: it is applied only when its source exists and +its target does not, so replaying the manifest can neither duplicate nor skip a +generation. Because nothing is unlinked before step 7, the prepare phase is +also reversible by renaming each staged or shifted file back. + +Rotation therefore settles into either the pre-rotation or the post-rotation +layout, never a mixture of the two: + +- An error during prepare reverses the renames already applied, discards any + uncommitted gzip temporary, removes the manifest, and aborts before the + append, leaving the pre-rotation layout intact. +- An error during the append in step 6 leaves the rotated layout in place with + the previous contents readable in generation `1`. The invocation still + performs step 7 before reporting the append failure, because the rotation + itself completed. +- A process killed at any point leaves the manifest behind. The next + invocation acquires the journal lock, finds the manifest, and drives the + recorded plan to a settled state before evaluating its own rotation: it + completes the outstanding prepare steps and commits when the plan can still + be completed, and otherwise reverses the applied renames and removes the + manifest. Only then does it handle its own record. When `schedule` is `hourly`, `daily`, or `weekly`, the command uses UTC period boundaries derived from the invocation timestamp: diff --git a/docs/roadmap.md b/docs/roadmap.md index c5a52f7..d76a095 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -358,9 +358,17 @@ mpsc-log-design.md §§2, 8, 11 and terms-of-reference.md §§6-7. - See mpsc-log-design.md §§6, 8. - Success: files beyond `plain_generations + compressed_generations` are deleted only after newer retained files are safely in place. - - Success: the evicted oldest generation is staged aside rather than deleted, - is unlinked only after the append commits, and a staged eviction left by a - killed process is reclaimed by the next invocation under the journal lock. + - Success: the evicted oldest generation and every superseded source + generation are staged aside rather than unlinked, and are removed only at + the commit point after the append. + - [ ] Fault-inject a failure at each rotation commit point: a partial rename + sequence, an uncommitted and a committed gzip output, staged source + removal, a failed append, and a process killed mid-rotation. + - Success: after every injected fault the journal settles into either the + pre-rotation or the post-rotation layout and never a mixture, no + record-bearing generation is lost, and the next invocation under the + journal lock completes or reverses the recorded plan and clears the + manifest before handling its own record. ### 4.3. Deliver UTC scheduled rotation with size splits From 17f855f9671db97b78418a2fe3d739ba003e1f42 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 20 Aug 2026 18:45:23 +0200 Subject: [PATCH 22/27] Resolve the lock-timeout circularity and define append commit points Address four review findings across the design, roadmap, and lock-naming ADR. Lock timeout. The timeout was documented as sidecar-configurable while the lock had to be held before the sidecar could be read, so no document said which timeout governed the only acquisition attempt. Resolve it with an unlocked advisory pre-read: the single attempt uses a five-second default that a pre-read of `[locking] timeout_ms` may override, the pre-read only chooses how long this invocation waits and never feeds repair, rotation, coercion, or defaults, an absent or invalid pre-read falls back to five seconds, and no second attempt is made. Apply the same wording to the design architecture and write-protocol sections, the sidecar table, the critical -section diagram, roadmap items 3.1.2 and 3.1.3, and ADR 001, whose ordering claims now refer to the authoritative sidecar read. Append commit point. The writer only defined rollback for a `write_all` error, although the exit-code table maps flush and filesystem metadata failures to EX_IOERR. Define a record as committed once its bytes and terminating newline are written and flushed, extend the restore-to-recorded -length rollback to every in-process failure after bytes reach the file, state that a failed rollback truncate still returns EX_IOERR and leaves the tail for partial-tail repair, and record that a retry after EX_IOERR cannot duplicate a committed record. Note the residual at-least-once caveat when a process dies between the write and the exit. Expand roadmap items 3.2.1 and 3.2.3 to fault-inject those paths. Size-only rotation predicate. The size-only subsection named `max_bytes` but never said when rotation triggers. State that rotation is evaluated after tail repair and before the append, compares the repaired active length plus the serialized pending record against `max_bytes`, and rotates only a non-empty active file, so an oversized record appends directly to an empty active file. The scheduled subsection already carried this rule. Non-goals. Record that `mpsc-log` is `jo`-inspired rather than `jo` compatible, cross-referencing the CLI contract and ADR 003 rather than duplicating the option list. Co-Authored-By: Claude Opus 5 (1M context) --- docs/adr-001-lock-file-naming.md | 22 +++++++++------- docs/mpsc-log-design.md | 43 ++++++++++++++++++++++++++++++-- docs/roadmap.md | 19 ++++++++++++++ 3 files changed, 73 insertions(+), 11 deletions(-) diff --git a/docs/adr-001-lock-file-naming.md b/docs/adr-001-lock-file-naming.md index 2933da1..295b1d1 100644 --- a/docs/adr-001-lock-file-naming.md +++ b/docs/adr-001-lock-file-naming.md @@ -12,10 +12,10 @@ same directory. ## Context and problem statement -`mpsc-log` serializes repair, sidecar reads, rotation, compression, and append -through one advisory lock per journal. Every safety claim in the design depends -on all cooperating invocations choosing the same lock for the same journal and -choosing different locks for different journals. +`mpsc-log` serializes repair, the authoritative sidecar read, rotation, +compression, and append through one advisory lock per journal. Every safety +claim in the design depends on all cooperating invocations choosing the same +lock for the same journal and choosing different locks for different journals. The obvious rule, `.lock` in the same directory, is simple but needs collision boundaries. Names such as `run`, `run.jsonl`, @@ -60,9 +60,12 @@ can safely create under contention. ### Technical requirements - Create parent directories before opening the lock file. -- Open the lock file with create semantics and acquire the exclusive lock before - reading sidecar configuration, repairing tails, rotating, compressing, or - appending. +- Open the lock file with create semantics and acquire the exclusive lock + before the authoritative sidecar configuration read, repairing tails, + rotating, compressing, or appending. +- Permit an unlocked, advisory pre-read of the sidecar `timeout_ms` value + solely to choose the lock-acquisition timeout; it never substitutes for + the authoritative read. - Treat `.lock` journal paths as invalid because `.lock` is reserved for coordination artefacts. - Treat `.toml` journal paths as invalid when the derived sidecar path would be @@ -176,8 +179,9 @@ configuration must choose distinct stems or directories, such as - Reject paths whose derived sidecar path equals the journal path. 2. Implement lock-path derivation as a pure function and test the examples in Table 1. -3. Use the derived lock path for the complete critical section: sidecar read, - tail repair, rotation, compression, and append. +3. Use the derived lock path for the complete critical section: the + authoritative sidecar read, tail repair, rotation, compression, and + append. 4. Document reserved suffixes and sidecar sharing in the users' guide when the CLI implementation lands. diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index b697f87..7219dd3 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -106,6 +106,9 @@ atomic-output integration, dependency risk, and portability.[^gzp] [^gzippy] - `mpsc-log` does not provide distributed locking across hosts. - `mpsc-log` does not preserve textual duplicate JSON object keys. - `mpsc-log` does not guarantee chronological record order by `timestamp`. +- `mpsc-log` is `jo`-inspired rather than `jo` compatible: it omits `jo` + formatting, pretty-printing, array-root, and version options; see + section 5 and [ADR 003](adr-003-jo-field-syntax-and-duplicate-keys.md). ## 4. Architecture @@ -139,6 +142,17 @@ first, opens the lock file with create semantics, acquires an exclusive lock, and only then reads configuration, repairs the journal tail, rotates, compresses, and appends. +The lock timeout is the one setting that cannot come from the locked +configuration read, because the lock must already be held to read +configuration authoritatively. The single acquisition attempt uses a +five-second default, which an unlocked advisory pre-read of the sidecar's +`[locking] timeout_ms` may override. That pre-read is advisory only: it +selects how long this invocation waits for the lock and never feeds repair, +rotation, coercion, or defaults. A missing, unreadable, or invalid pre-read +falls back to five seconds without failing the invocation. Once the lock is +held, the authoritative configuration read supplies one coherent view for +everything else. There is no second acquisition attempt. + ## 5. CLI contract The first positional argument is always the journal path: @@ -185,7 +199,7 @@ has four tables: | Table | Responsibility | | ------------ | --------------------------------------------------------------------------------------------------- | | `[rotation]` | `schedule`, `max_bytes`, plain generation count, compressed generation count, and gzip policy. | -| `[locking]` | Lock timeout and partial-tail repair mode. | +| `[locking]` | Lock timeout, read by an unlocked advisory pre-read, and partial-tail repair mode. | | `[defaults]` | Default JSON fields inserted before CLI fields. | | `[schema]` | Object paths mapped to coercion names: `string`, `integer`, `number`, `boolean`, `json`, or `null`. | @@ -221,7 +235,7 @@ sequenceDiagram A->>M: mpsc-log path fields... M->>M: create parent directories M->>L: open or create lock file - M->>L: acquire exclusive lock with timeout + M->>L: acquire exclusive lock with pre-read or default timeout M->>J: validate final record and repair partial tail M->>R: rotate and compress if threshold or schedule requires it M->>J: append record plus newline @@ -248,6 +262,23 @@ classifies the active file tail while holding the journal lock: invalid JSON, not an object, or empty, the invocation fails closed with `EX_DATAERR` and leaves the file unchanged. +A record is committed once its bytes and terminating newline are written +and flushed; before that point it is uncommitted. The writer restores the +recorded pre-append length on any in-process failure after bytes reach the +file, including flush and filesystem metadata failures, and all of these +failures map to `EX_IOERR`. If the rollback truncate itself fails, the +invocation still returns `EX_IOERR` and leaves the unterminated tail for +the next invocation's partial-tail repair; this is why repair classifies +the tail rather than trusting the previous writer. + +Because a failed invocation commits nothing, a caller retrying after +`EX_IOERR` cannot duplicate a committed record: the previous attempt either +committed and returned success, or committed nothing. One caveat remains: a +process killed between the write and the exit can leave a complete record +the caller never saw acknowledged, so a retry then appends a second copy. +`mpsc-log` is therefore at-least-once under process death, and callers +needing deduplication should carry their own idempotency key in the record. + The repair path does not quarantine files and does not truncate a newline-terminated corrupt record by default. It also does not scan every historical line on each append; it validates only the active file's complete @@ -301,6 +332,14 @@ zero, no plain archives exist and the active file is gzipped straight into generation `1`. When `C` is zero, no archive is compressed and the oldest plain generation is evicted rather than gzipped. +Rotation is evaluated after tail repair and before the append. The +invocation compares the repaired active file length plus the length of the +serialized pending record, including its terminating newline, against +`max_bytes`. It rotates only when the active file is non-empty and that +combined length exceeds `max_bytes`. When the active file is empty, the +pending record is appended directly, so an oversized record never rotates +an empty active file into an archive. + Size-only rotation is a prepare phase followed by a commit point. The prepare phase only creates files and renames them within the journal directory, so every source generation stays on disk and recoverable until the rotation diff --git a/docs/roadmap.md b/docs/roadmap.md index d76a095..dffd830 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -247,11 +247,19 @@ critical-section protocol and timeout behaviour. See mpsc-log-design.md §§4, 7 failures return `EX_TEMPFAIL`. Network-filesystem support (NFS and CIFS) remains deferred until the named filesystem verification matrix in step 6.2.1. + - Success: the single lock acquisition attempt uses a five-second default + timeout, which an unlocked advisory pre-read of `[locking] timeout_ms` + may override; an absent, unreadable, or invalid pre-read falls back to + five seconds without failing the invocation; and no second acquisition + attempt is made. - [ ] 3.1.3. Read sidecar configuration inside the journal critical section. - Requires 3.1.2. - See mpsc-log-design.md §§4, 6-7 and terms-of-reference.md §8.2. - Success: each invocation uses one coherent sidecar view for repair, rotation, compression, and append. + - Success: the advisory pre-read is used only to choose the lock timeout + and never supplies repair, rotation, coercion, or default values, which + come solely from the authoritative read taken under the lock. ### 3.2. Preserve complete records across write and crash-like failures @@ -266,6 +274,10 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md - See mpsc-log-design.md §§7, 10-11. - Success: injected write failures truncate the active file back to its recorded pre-append length before returning an error. + - Success: rollback also covers flush and filesystem metadata failures + after bytes reach the file, all of which map to `EX_IOERR`; a failed + rollback truncate still returns `EX_IOERR` and leaves the unterminated + tail for the next invocation's partial-tail repair. - [ ] 3.2.2. Implement partial-tail repair before every append. - Requires 3.2.1. - See mpsc-log-design.md §§6-7, 11. @@ -279,6 +291,13 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md - Success: each documented filesystem failure class has a deterministic assertion for journal state and exit-code mapping, including the invalid newline-terminated final-line case. + - [ ] Fault-inject a flush failure and a filesystem metadata failure after + bytes have reached the active file, plus a failed rollback truncate. + - Success: no injected post-write failure leaves a committed record, a + retry after `EX_IOERR` does not duplicate a committed record, and a + complete record left by a process killed after the write is repaired or + retained per the documented at-least-once behaviour rather than silently + duplicated. ### 3.3. Demonstrate concurrent append correctness end to end From 445763c1ba48b6d4516a85476aa78253583cc2c5 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 20 Aug 2026 20:36:37 +0200 Subject: [PATCH 23/27] Admit duplication when a failed rollback truncate hides commit status The append contract contradicted itself. It assumed a failed rollback truncate leaves an unterminated tail, then concluded that a retry after EX_IOERR could not duplicate a committed record. Neither holds when write_all completes, flush fails, and the rollback truncate then fails: the active file ends with a complete newline-terminated record, which the repair rules preserve rather than remove, so a retry appends a second copy. - State that a failed rollback truncate leaves the commit status unknown, that an unterminated tail is removed by partial-tail repair while a complete newline-terminated record is preserved and therefore committed, and that the invocation returns EX_IOERR without knowing which occurred. - Replace the no-duplication guarantee with at-least-once whenever the writer cannot confirm the outcome, covering both the failed rollback truncate and process death between the write and the exit. - Record that `mpsc-log` keeps no unknown-commit reconciliation state, so callers needing exactly-once must carry an idempotency key and deduplicate when reading. - Reword roadmap items 3.2.1 and 3.2.3 so the acceptance criteria allow the duplication instead of asserting it cannot happen, and assert it as the documented outcome rather than a defect. The "exactly one record per successful invocation" claims elsewhere are unaffected, since at-least-once applies only to unconfirmed outcomes. Co-Authored-By: Claude Opus 5 (1M context) --- docs/mpsc-log-design.md | 30 ++++++++++++++++++------------ docs/roadmap.md | 18 ++++++++++-------- 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 7219dd3..3fe7339 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -266,18 +266,24 @@ A record is committed once its bytes and terminating newline are written and flushed; before that point it is uncommitted. The writer restores the recorded pre-append length on any in-process failure after bytes reach the file, including flush and filesystem metadata failures, and all of these -failures map to `EX_IOERR`. If the rollback truncate itself fails, the -invocation still returns `EX_IOERR` and leaves the unterminated tail for -the next invocation's partial-tail repair; this is why repair classifies -the tail rather than trusting the previous writer. - -Because a failed invocation commits nothing, a caller retrying after -`EX_IOERR` cannot duplicate a committed record: the previous attempt either -committed and returned success, or committed nothing. One caveat remains: a -process killed between the write and the exit can leave a complete record -the caller never saw acknowledged, so a retry then appends a second copy. -`mpsc-log` is therefore at-least-once under process death, and callers -needing deduplication should carry their own idempotency key in the record. +failures map to `EX_IOERR`. A successful rollback leaves the record +uncommitted, so a later retry cannot duplicate it. + +A failed rollback truncate leaves the commit status unknown. The written +bytes stay on disk, and which repair rule applies depends on how far the +write got: an unterminated tail is removed by partial-tail repair, whereas +a complete newline-terminated record is preserved by the rule above and is +therefore committed. The invocation returns `EX_IOERR` without knowing +which case occurred, which is why repair classifies the tail rather than +trusting the previous writer. + +`mpsc-log` is therefore at-least-once whenever the writer cannot confirm +the outcome. A retry after `EX_IOERR` can duplicate a record that a failed +rollback truncate left complete, and a process killed between the write and +the exit can leave a complete record the caller never saw acknowledged. +`mpsc-log` keeps no unknown-commit reconciliation state, so callers needing +exactly-once semantics must carry their own idempotency key in the record +and deduplicate when reading. The repair path does not quarantine files and does not truncate a newline-terminated corrupt record by default. It also does not scan every diff --git a/docs/roadmap.md b/docs/roadmap.md index dffd830..7943310 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -275,9 +275,10 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md - Success: injected write failures truncate the active file back to its recorded pre-append length before returning an error. - Success: rollback also covers flush and filesystem metadata failures - after bytes reach the file, all of which map to `EX_IOERR`; a failed - rollback truncate still returns `EX_IOERR` and leaves the unterminated - tail for the next invocation's partial-tail repair. + after bytes reach the file, all of which map to `EX_IOERR`; a successful + rollback leaves the record uncommitted; and a failed rollback truncate + still returns `EX_IOERR` while leaving the commit status unknown for the + next invocation to classify. - [ ] 3.2.2. Implement partial-tail repair before every append. - Requires 3.2.1. - See mpsc-log-design.md §§6-7, 11. @@ -293,11 +294,12 @@ and validates the design's append-plus-repair claim. See mpsc-log-design.md newline-terminated final-line case. - [ ] Fault-inject a flush failure and a filesystem metadata failure after bytes have reached the active file, plus a failed rollback truncate. - - Success: no injected post-write failure leaves a committed record, a - retry after `EX_IOERR` does not duplicate a committed record, and a - complete record left by a process killed after the write is repaired or - retained per the documented at-least-once behaviour rather than silently - duplicated. + - Success: a successful rollback leaves no committed record; a failed + rollback truncate leaves either an unterminated tail that repair removes + or a complete record that repair preserves; and a retry after `EX_IOERR` + may duplicate a preserved record. The tests assert that duplication as + the documented at-least-once outcome rather than treating it as a + defect, because `mpsc-log` keeps no unknown-commit reconciliation state. ### 3.3. Demonstrate concurrent append correctness end to end From 7e748f634790179fb03408eb0685483c5636af96 Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 23 Aug 2026 01:26:16 +0200 Subject: [PATCH 24/27] Own the record model in the domain and document the port boundary The design put `serde_json::Map` in the domain core, so the record model was defined by a serialization crate and the module table did not say which modules were domain and which were adapters. Contributors had no documented architecture to place new code against. Design: - Replace the `serde_json::Map` core with domain `Record` and `Value` types. Add section 4.1 defining `Value` variants and the required behaviour for object paths, nested objects, arrays, scalars, coercion results, and last-wins replacement. - Add section 4.2 declaring the `Clock` and `JournalStore` ports, and state that no domain module names a filesystem, locking, TOML, JSON, or `sysexits` API. - Add section 4.3 defining the adapters, including the TOML adapter that converts sidecar defaults and schema data into domain types at the input boundary and the JSON adapter that emits one compact JSON object at the output boundary. - Give the section 12 module table a Layer column, and record that JSON output serialization and process exit mapping are boundary concerns rather than domain modules. - Reword the merge table and inference prose so sidecar defaults convert into domain `Value` equivalents rather than JSON equivalents. Developer guide: add an implementation architecture section covering the eight planned modules and their layers, the `src/main.rs` startup and `sysexits` responsibility, the adapter boundary and its concerns, the two ports, and links to the design and ADRs 001, 002, and 003. ADR 003: replace the `serde_json::Map` mandate with the domain `Record` object map of `Value` values serialized by the JSON adapter, keeping the accepted last-wins semantics and the textual duplicate-name non-goal. Contents: note that the developer guide now covers the architecture. Compact JSON output and last-wins duplicate handling are unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- ...-003-jo-field-syntax-and-duplicate-keys.md | 21 +- docs/contents.md | 3 +- docs/developers-guide.md | 70 ++++++ docs/mpsc-log-design.md | 212 ++++++++++++------ 4 files changed, 228 insertions(+), 78 deletions(-) diff --git a/docs/adr-003-jo-field-syntax-and-duplicate-keys.md b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md index 027b8b0..a8a1434 100644 --- a/docs/adr-003-jo-field-syntax-and-duplicate-keys.md +++ b/docs/adr-003-jo-field-syntax-and-duplicate-keys.md @@ -15,12 +15,13 @@ Duplicate writes to the same object path resolve with last-wins semantics. `mpsc-log` should feel familiar to workflow authors who already use `jo` to build JSON from shell arguments. The product goal is not to be a drop-in `jo` replacement. It is a reliable multi-process JSON Lines writer whose root record -is always an object and whose output is produced through `serde_json::Map`. +is always an object, held in the domain as a `Record` object map of domain +`Value` values and serialized by the JSON adapter at the output boundary. That distinction matters for duplicate object keys. Textual JSON can contain -repeated object names, and `jo` can produce those names in its output. A JSON +repeated object names, and `jo` can produce those names in its output. An object map cannot preserve repeated textual keys. Once `mpsc-log` chooses a map -as its internal representation, duplicate writes must either be rejected or +as its domain representation, duplicate writes must either be rejected or collapsed into one value. The project needs an explicit compatibility statement before implementing the @@ -53,7 +54,10 @@ argument parser, merge rules, sidecar schema interaction, and tests. ### Technical requirements -- Store records in `serde_json::Map` before serialization. +- Store records in a domain `Record` object map whose values are domain + `Value` values, and keep serialization crates out of domain modules. +- Serialize that `Record` to compact JSON through `serde_json` in the JSON + adapter at the output boundary. - Process CLI field words in argument order. - Let later writes to the same object path replace earlier writes. - Pair the duplicate-key behaviour with parameterized tests covering top-level @@ -89,9 +93,9 @@ awkward and complicates the existing precedence model. ### Option C: Preserve textual duplicate JSON object names This option would attempt closer textual compatibility with `jo` by emitting -duplicate JSON object names. It conflicts with `serde_json::Map`, makes schema -coercion and object-path updates harder to reason about, and produces records -that many consumers collapse differently. +duplicate JSON object names. It conflicts with the domain `Record` object map, +makes schema coercion and object-path updates harder to reason about, and +produces records that many consumers collapse differently. ### Option D: Full `jo` compatibility @@ -137,7 +141,8 @@ textual JSON names. - Goals: - Make the compatibility boundary honest for users and implementers. - Preserve the familiar shell-friendly field forms needed for logging. - - Keep object-root JSON serialization map-based and deterministic. + - Keep the object-root `Record` map-based and its serialization + deterministic. - Align duplicate-path behaviour with merge precedence. - Non-goals: - Preserve repeated textual JSON object names. diff --git a/docs/contents.md b/docs/contents.md index 0fe8950..b2cab73 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -15,7 +15,8 @@ set. review-sized implementation tasks. - [User guide](users-guide.md) explains how to use the generated project and its public build and test commands. -- [Developer guide](developers-guide.md) explains the local workflow and +- [Developer guide](developers-guide.md) explains the planned implementation + architecture, module layering, adapter boundary, local workflow, and implementation tooling for contributors. - [Repository layout](repository-layout.md) explains the generated project's top-level files, directories, and ownership boundaries. diff --git a/docs/developers-guide.md b/docs/developers-guide.md index bbbe3f5..e4db3bd 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -2,6 +2,76 @@ This guide explains the contributor workflow for the generated mpsc-log project. + +## mpsc-log implementation architecture + +`mpsc-log` is planned as a small domain core surrounded by adapters. The domain +owns the record model and the write, rotation, and retention policy. Adapters +own every external format and effect. The authoritative specification is the +[design document](mpsc-log-design.md); this section summarizes the boundary it +defines so contributors can place new code correctly. + +`src/main.rs` owns process startup and `sysexits` mapping only. It constructs +the adapters, runs the domain, and translates the semantic error type into an +exit code. It carries no parsing, merging, rotation, or filesystem logic. + + +### Planned modules + +| Module | Layer | Responsibility | +| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `args` | Adapter | Read process arguments and hand on the journal path and the raw selected `jo` field tail in order. | +| `fields` | Domain | Interpret field words, object paths, coercion flags, and file-value forms into domain `Value` results. | +| `config` | Adapter | Parse and validate the sidecar TOML at the input boundary, converting `[defaults]` and `[schema]` data into domain types. | +| `record` | Domain | Build the `Record`: merge defaults and CLI fields, apply coercion policy and object-path updates, and insert the generated timestamp. | +| `journal` | Domain | Plan tail repair, rotation, retention, and append, and drive them through the `JournalStore` port. | +| `errors` | Domain | Define the semantic error type. Exit-code mapping belongs to `src/main.rs`, not here. | +| `clock` | Domain port | Declare the `Clock` port; an infrastructure implementation supplies the real instant. | +| `fs` | Adapter | Implement `JournalStore` over the real filesystem, alongside the fault-injection test double. | + + +### The adapter boundary + +The domain never names an external API. It works only with the domain `Record` +and `Value` types and with ports. Everything that reaches outside the process, +or that encodes an external format, is an adapter concern: + +- CLI argument reads. +- TOML parsing of the sidecar. +- JSON serialization of the record. +- Filesystem operations. +- Locking. +- Compression. +- Process exit mapping. + +The domain reaches time and persistence-related effects through two ports. +`Clock` supplies the invocation instant used for the generated `timestamp` +field. `JournalStore` performs every journal-directory effect: creating parent +directories, reading the sidecar, acquiring and releasing the journal lock, +inspecting and repairing the active tail, appending, truncating, renaming, +writing compression output, and syncing metadata. + +Because both are traits, tests substitute a fixed clock and a fault-injecting +store without mutating global process state. See +[reliable testing in Rust via dependency injection](reliable-testing-in-rust-via-dependency-injection.md) +for the injection patterns this repository expects. + +When adding code, put format and effect handling in an adapter and keep policy +in the domain. If a domain module needs a new external effect, add a port +rather than importing the concrete API. + + +### Further reading + +- [Design](mpsc-log-design.md) specifies the CLI contract, record model, + ports, adapters, write protocol, and rotation protocol. +- [ADR 001: Lock file naming](adr-001-lock-file-naming.md) records how the + journal lock path is derived and which suffixes are reserved. +- [ADR 002: Testing strategy](adr-002-testing-strategy.md) records how the + required testing prongs apply to this design. +- [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md) + records the selected field syntax and last-wins duplicate-path semantics. + ## Spelling policy Run `make spelling` to enforce en-GB-oxendict prose spelling. The generated diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 3fe7339..19e5b90 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -39,7 +39,7 @@ The CLI field grammar is `jo`-inspired and uses selected `jo` field syntax where that serves object-record logging. `jo` supports `key=value`, `key@value`, type coercion flags, object paths, file-value prefixes, array construction, and duplicate object keys.[^jo] `mpsc-log` rejects array roots -and serializes through `serde_json`, so duplicate paths resolve by last write +and builds an object-map `Record`, so duplicate paths resolve by last write rather than producing repeated textual JSON object names. That compatibility boundary is recorded in [ADR 003: `jo` field syntax and duplicate keys](adr-003-jo-field-syntax-and-duplicate-keys.md). @@ -64,7 +64,7 @@ The Rust implementation baseline is conservative: | Need | Choice | Reason | | ----------- | ------------------------------ | ------------------------------------------------------------------------------------------------- | | CLI parsing | `clap` builder API | Handles help/version/error rendering while allowing raw selected `jo` field tails.[^clap] | -| JSON | `serde`, `serde_json` | Standard Rust serialization stack; object maps naturally enforce last-wins duplicate handling. | +| JSON | `serde`, `serde_json` | Standard Rust serialization stack, used by the JSON output adapter rather than by domain modules. | | TOML | `toml` crate, TOML v1.0 subset | Current crate supports TOML parsing; the product contract remains v1.0. | | Time | `jiff` | Provides `Timestamp::now()` and RFC 3339-style instant formatting with nanosecond support.[^jiff] | | Locking | `fs4` sync feature | Provides cross-platform file locks without requiring a daemon.[^fs4] | @@ -107,16 +107,17 @@ atomic-output integration, dependency risk, and portability.[^gzp] [^gzippy] - `mpsc-log` does not preserve textual duplicate JSON object keys. - `mpsc-log` does not guarantee chronological record order by `timestamp`. - `mpsc-log` is `jo`-inspired rather than `jo` compatible: it omits `jo` - formatting, pretty-printing, array-root, and version options; see - section 5 and [ADR 003](adr-003-jo-field-syntax-and-duplicate-keys.md). + formatting, pretty-printing, array-root, and version options; see section 5 + and [ADR 003](adr-003-jo-field-syntax-and-duplicate-keys.md). ## 4. Architecture -The design is a synchronous CLI with a small domain core and filesystem -adapter. The core parses fields, merges configuration, produces a -`serde_json::Map`, and computes rotation actions. The adapter owns directory -creation, sidecar reads, lock acquisition, append, truncate repair, rename, and -gzip compression. +The design is a synchronous CLI with a small domain core surrounded by +adapters. The domain interprets field words, merges configuration, builds a +`Record`, and computes rotation actions. It reaches every external effect +through a port. Adapters own directory creation, sidecar reads, lock +acquisition, append, truncate repair, rename, gzip compression, and the +translation between external formats and domain types. ```mermaid flowchart LR @@ -143,15 +144,80 @@ and only then reads configuration, repairs the journal tail, rotates, compresses, and appends. The lock timeout is the one setting that cannot come from the locked -configuration read, because the lock must already be held to read -configuration authoritatively. The single acquisition attempt uses a -five-second default, which an unlocked advisory pre-read of the sidecar's -`[locking] timeout_ms` may override. That pre-read is advisory only: it -selects how long this invocation waits for the lock and never feeds repair, -rotation, coercion, or defaults. A missing, unreadable, or invalid pre-read -falls back to five seconds without failing the invocation. Once the lock is -held, the authoritative configuration read supplies one coherent view for -everything else. There is no second acquisition attempt. +configuration read, because the lock must already be held to read configuration +authoritatively. The single acquisition attempt uses a five-second default, +which an unlocked advisory pre-read of the sidecar's `[locking] timeout_ms` may +override. That pre-read is advisory only: it selects how long this invocation +waits for the lock and never feeds repair, rotation, coercion, or defaults. A +missing, unreadable, or invalid pre-read falls back to five seconds without +failing the invocation. Once the lock is held, the authoritative configuration +read supplies one coherent view for everything else. There is no second +acquisition attempt. + +### 4.1 Domain record model + +The domain owns its record types so that no domain module depends on a +serialization crate. Two types carry every journal entry: + +- `Value` is the domain value tree, with null, boolean, integer, float, + string, array, and object variants. The object variant is a map from key to + `Value`. +- `Record` is the root of one entry. It is always an object map from key to + `Value`, so a record can never serialize as an array or a bare scalar. + +The domain defines this behaviour over those types: + +- Object paths. A path is the sequence of key segments produced by splitting + a field key on the configured delimiter. Writing a path creates any missing + intermediate objects. +- Nested objects. Writing `task.id` creates `task` as an object variant and + sets `id` within it. Writing through a segment whose existing value is not an + object replaces that value with a new object. +- Arrays. The `key[]` form appends to an array at that path, creating an + empty array first when the path is unset. +- Scalars. Null, boolean, integer, float, and string values are stored as the + matching `Value` variant rather than as text. +- Coercion results. Coercion is a domain policy mapping one raw field word + and its coercion source to one `Value`. Explicit `-s`, `-n`, and `-b` flags + win, then the matching `[schema]` entry, then default `jo`-inspired inference. +- Last-wins replacement. Writing a path that already holds a value replaces + that value. Ordering follows section 6, so sidecar defaults seed the record + and each later CLI write wins at its path. A `Record` therefore holds one + value per path and never repeats a key. + +### 4.2 Ports + +The domain reaches external effects only through ports. A port is a trait +declared by the domain in domain terms; an adapter implements it with a +concrete API. + +| Port | Responsibility | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Clock` | Supply the invocation instant used for the generated `timestamp` field, so tests can fix time without mutating process state. | +| `JournalStore` | Perform every journal-directory effect: create parent directories, read the sidecar, acquire and release the journal lock, inspect and repair the active tail, append, truncate, rename, write compression output, and sync metadata. | + +Table 2: Domain ports. + +No domain module names a filesystem API, a locking crate, a TOML parser, a JSON +serializer, or a `sysexits` code. The domain returns semantic errors, and the +process boundary maps them to exit codes as section 10 describes. + +### 4.3 Adapters + +Adapters sit on both sides of the domain: + +- The CLI adapter reads process arguments and hands the domain raw field + words in their original order. +- The TOML adapter parses the sidecar at the input boundary and converts + `[defaults]` and `[schema]` data into domain types, so the domain receives + `Value` defaults and coercion names rather than TOML values. +- The JSON adapter converts a `Record` into one compact JSON object at the + output boundary through `serde_json`, emitting object keys in a deterministic + order. +- The filesystem adapter implements `JournalStore` over the real filesystem. + A test adapter implements the same port for fault injection. +- The process adapter, `src/main.rs`, maps the semantic error type to + `sysexits` codes. ## 5. CLI contract @@ -205,20 +271,21 @@ has four tables: Merge and coercion order is deterministic: -| Step | Rule | Winner | -| ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | Start with sidecar `[defaults]` converted from TOML values to their JSON equivalents. | Sidecar defaults seed the record. | -| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | -| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | -| 4 | Validate an override timestamp, if present. | A sidecar or CLI `timestamp` must match the canonical RFC 3339 UTC format in the event schema; an invalid override returns `EX_DATAERR` before append. | -| 5 | Insert the generated invocation timestamp only when no override exists. | The generated canonical UTC `timestamp` is added only when defaults and CLI fields did not produce one. | -| 6 | Serialize the resulting object. | The merged record is written as one compact JSON object. | +| Step | Rule | Winner | +| ---- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Start with sidecar `[defaults]`, which the TOML adapter converts into domain `Value` equivalents. | Sidecar defaults seed the record. | +| 2 | Process CLI fields in argument order and write each value to its object path. | Each CLI field wins over sidecar defaults and over earlier CLI fields at the same path. | +| 3 | Coerce each CLI value before writing it. | Explicit `-s`, `-n`, and `-b` flags win for that word; otherwise the matching `[schema]` entry wins; otherwise default `jo`-inspired inference wins. | +| 4 | Validate an override timestamp, if present. | A sidecar or CLI `timestamp` must match the canonical RFC 3339 UTC format in the event schema; an invalid override returns `EX_DATAERR` before append. | +| 5 | Insert the generated invocation timestamp only when no override exists. | The generated canonical UTC `timestamp` is added only when defaults and CLI fields did not produce one. | +| 6 | Serialize the resulting object. | The merged record is written as one compact JSON object. | The sidecar schema never overrides an explicit CLI coercion flag. Schema entries affect only values supplied for the matching object path; they do not change sidecar defaults or the generated timestamp. When default `jo`-inspired -inference is used, valid JSON values parse as JSON, empty `key=` becomes -`null`, and other values remain strings. +inference is used, a field word that is valid JSON parses into the matching +`Value` variant, empty `key=` becomes a null `Value`, and other words remain +strings. ## 7. Journal write protocol @@ -262,28 +329,27 @@ classifies the active file tail while holding the journal lock: invalid JSON, not an object, or empty, the invocation fails closed with `EX_DATAERR` and leaves the file unchanged. -A record is committed once its bytes and terminating newline are written -and flushed; before that point it is uncommitted. The writer restores the -recorded pre-append length on any in-process failure after bytes reach the -file, including flush and filesystem metadata failures, and all of these -failures map to `EX_IOERR`. A successful rollback leaves the record -uncommitted, so a later retry cannot duplicate it. - -A failed rollback truncate leaves the commit status unknown. The written -bytes stay on disk, and which repair rule applies depends on how far the -write got: an unterminated tail is removed by partial-tail repair, whereas -a complete newline-terminated record is preserved by the rule above and is -therefore committed. The invocation returns `EX_IOERR` without knowing -which case occurred, which is why repair classifies the tail rather than -trusting the previous writer. - -`mpsc-log` is therefore at-least-once whenever the writer cannot confirm -the outcome. A retry after `EX_IOERR` can duplicate a record that a failed -rollback truncate left complete, and a process killed between the write and -the exit can leave a complete record the caller never saw acknowledged. -`mpsc-log` keeps no unknown-commit reconciliation state, so callers needing -exactly-once semantics must carry their own idempotency key in the record -and deduplicate when reading. +A record is committed once its bytes and terminating newline are written and +flushed; before that point it is uncommitted. The writer restores the recorded +pre-append length on any in-process failure after bytes reach the file, +including flush and filesystem metadata failures, and all of these failures map +to `EX_IOERR`. A successful rollback leaves the record uncommitted, so a later +retry cannot duplicate it. + +A failed rollback truncate leaves the commit status unknown. The written bytes +stay on disk, and which repair rule applies depends on how far the write got: +an unterminated tail is removed by partial-tail repair, whereas a complete +newline-terminated record is preserved by the rule above and is therefore +committed. The invocation returns `EX_IOERR` without knowing which case +occurred, which is why repair classifies the tail rather than trusting the +previous writer. + +`mpsc-log` is therefore at-least-once whenever the writer cannot confirm the +outcome. A retry after `EX_IOERR` can duplicate a record that a failed rollback +truncate left complete, and a process killed between the write and the exit can +leave a complete record the caller never saw acknowledged. `mpsc-log` keeps no +unknown-commit reconciliation state, so callers needing exactly-once semantics +must carry their own idempotency key in the record and deduplicate when reading. The repair path does not quarantine files and does not truncate a newline-terminated corrupt record by default. It also does not scan every @@ -338,13 +404,13 @@ zero, no plain archives exist and the active file is gzipped straight into generation `1`. When `C` is zero, no archive is compressed and the oldest plain generation is evicted rather than gzipped. -Rotation is evaluated after tail repair and before the append. The -invocation compares the repaired active file length plus the length of the -serialized pending record, including its terminating newline, against -`max_bytes`. It rotates only when the active file is non-empty and that -combined length exceeds `max_bytes`. When the active file is empty, the -pending record is appended directly, so an oversized record never rotates -an empty active file into an archive. +Rotation is evaluated after tail repair and before the append. The invocation +compares the repaired active file length plus the length of the serialized +pending record, including its terminating newline, against `max_bytes`. It +rotates only when the active file is non-empty and that combined length exceeds +`max_bytes`. When the active file is empty, the pending record is appended +directly, so an oversized record never rotates an empty active file into an +archive. Size-only rotation is a prepare phase followed by a commit point. The prepare phase only creates files and renames them within the journal directory, so @@ -542,18 +608,26 @@ filesystem and mount configuration. ## 12. Module structure `src/main.rs` owns CLI startup and process exit mapping only. The library owns -the implementation: - -| Module | Responsibility | -| --------- | ------------------------------------------------------------------------------ | -| `args` | Parse the journal path and raw selected `jo` field tail. | -| `fields` | Parse field words, object paths, coercion flags, and file-value forms. | -| `config` | Load and validate sidecar TOML. | -| `record` | Merge defaults, CLI fields, schema coercions, and generated timestamp. | -| `journal` | Create directories, acquire locks, repair tails, rotate, compress, and append. | -| `errors` | Semantic error enum and exit-code mapping. | -| `clock` | Injectable timestamp source for deterministic tests. | -| `fs` | Filesystem adapter boundary for fault injection. | +the implementation, split into domain modules, ports, and adapters: + +| Module | Layer | Responsibility | +| --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| `args` | Adapter | Read process arguments and hand on the journal path and the raw selected `jo` field tail in order. | +| `fields` | Domain | Interpret field words, object paths, coercion flags, and file-value forms into domain `Value` results. | +| `config` | Adapter | Parse and validate sidecar TOML at the input boundary, converting `[defaults]` and `[schema]` data into domain types. | +| `record` | Domain | Build the `Record`: merge defaults and CLI fields, apply coercion policy and object-path updates, and insert the generated timestamp. | +| `journal` | Domain | Plan tail repair, rotation, retention, and append, and drive them through the `JournalStore` port. | +| `errors` | Domain | Semantic error type. Exit-code mapping belongs to the process boundary, not to this module. | +| `clock` | Domain port | Declare the `Clock` port; an infrastructure implementation supplies the real instant. | +| `fs` | Adapter | Implement `JournalStore` over the real filesystem, alongside the fault-injection test double. | + +Table 3: Module layers and responsibilities. + +Two boundary concerns are not domain modules. JSON output serialization is an +adapter step at the output boundary, converting the domain `Record` into one +compact JSON object through `serde_json` immediately before the bytes reach the +`JournalStore` port. Process exit mapping belongs to `src/main.rs`, which +translates the semantic error type from `errors` into a `sysexits` code. The first implementation should expose no stable library API beyond what the binary needs. Public library exports remain internal support until a roadmap From 52a8dcfe860c79297c0cfe2b4666533dc2e45aef Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 23 Aug 2026 03:23:19 +0200 Subject: [PATCH 25/27] Show the port and adapter boundary in the topology diagram The architecture prose, module table, and ADR already described the domain, its `Record` and `Value` types, and the `Clock` and `JournalStore` ports, but Figure 1 still drew the earlier topology: CLI to argument parser and sidecar loader, into a record builder, into a journal writer. A reader of the architecture section saw no domain, port, or adapter boundary in the diagram. Redraw Figure 1 with input adapters, the domain, the ports, and the output adapters as separate groups, showing the domain reaching the outside world only through `Clock` and `JournalStore`, the filesystem adapter implementing `JournalStore`, and `main.rs` owning the `sysexits` mapping. Extend the caption to state the boundary in prose. Co-Authored-By: Claude Opus 5 (1M context) --- docs/mpsc-log-design.md | 53 +++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/docs/mpsc-log-design.md b/docs/mpsc-log-design.md index 19e5b90..bc0684c 100644 --- a/docs/mpsc-log-design.md +++ b/docs/mpsc-log-design.md @@ -121,18 +121,51 @@ translation between external formats and domain types. ```mermaid flowchart LR - Agent[Agent or workflow] --> CLI[mpsc-log CLI] - CLI --> Parser[Argument parser] - CLI --> Config[Sidecar loader] - Parser --> Builder[Record builder] - Config --> Builder - Builder --> Writer[Journal writer] - Writer --> Lock[(Journal lock)] - Writer --> Journal[(JSONL journal)] - Writer --> Rotated[(Rotated logs)] + Agent[Agent or workflow] + + subgraph inbound [Input adapters] + Args[args: process arguments] + Cfg[config: sidecar TOML] + end + + subgraph core [Domain] + Fields[fields: field interpretation] + Rec[record: builds the Record] + Jnl[journal: plans repair and rotation] + Errs[errors: semantic errors] + end + + subgraph ports [Ports] + Clk[[Clock]] + Store[[JournalStore]] + end + + subgraph outbound [Output adapters] + Json[JSON serializer] + Fs[fs: filesystem] + Exit[main.rs: sysexits mapping] + end + + Agent --> Args + Args --> Fields + Cfg --> Rec + Fields --> Rec + Clk --> Rec + Rec --> Jnl + Jnl --> Json + Json --> Store + Jnl --> Store + Errs --> Exit + Store -. implemented by .-> Fs + Fs --> Lock[(Journal lock)] + Fs --> JournalFile[(JSONL journal)] + Fs --> Rotated[(Rotated logs)] ``` -Figure 1: Runtime component topology. +Figure 1: Runtime component topology. The domain reaches the outside world only +through the `Clock` and `JournalStore` ports; adapters own the process +arguments, the sidecar TOML, JSON serialization, the filesystem, and the +`sysexits` mapping. The lock file is the coordination boundary. The naming decision is recorded in [ADR 001: Lock file naming](adr-001-lock-file-naming.md): `mpsc-log` appends From 258f969f5f4d82b0c9360c94783533e559e14da2 Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 23 Aug 2026 03:39:26 +0200 Subject: [PATCH 26/27] Require parameterized duplicate-path coverage in the pairwise suite ADR 003 pairs the accepted last-wins decision with parameterized tests over top-level keys, nested object paths, sidecar defaults, and explicit coercion flags, but roadmap item 3.3.2 listed only the pairwise axes, so nothing in the delivery plan required that coverage. Add a success criterion to 3.3.2 requiring parameterized duplicate-path tests for duplicate writes at the same object path, across top-level keys, nested object paths, a sidecar default overridden by a CLI field, and the explicit `-s`, `-n`, and `-b` coercion flags, asserting that the later write replaces the earlier value at that object path. The existing pairwise axes criterion is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmap.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index 7943310..4085555 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -318,6 +318,11 @@ terms-of-reference.md §§5, 7. - See mpsc-log-design.md §11 and adr-002-testing-strategy.md. - Success: the suite covers `jo` syntax form, coercion source, object path, sidecar default, rotation schedule, rotation state, and lock contention. + - Success: parameterized duplicate-path tests cover duplicate writes at the + same object path across top-level keys, nested object paths, a sidecar + default overridden by a CLI field, and explicit `-s`, `-n`, and `-b` + coercion flags, asserting last-wins: the later write replaces the earlier + value at that object path. ## 4. Rotation and retention without record loss From 810af68b63fc73c38959e4664f28cb9fbf71676c Mon Sep 17 00:00:00 2001 From: leynos Date: Sun, 23 Aug 2026 20:47:22 +0200 Subject: [PATCH 27/27] Apply mdformat output to this branch's Markdown main is now a fixed point of `make fmt`, so bring this branch's own files to the same state instead of reverting the formatter on every run. docs/adr-001-lock-file-naming.md and docs/terms-of-reference.md are files this branch adds, and their mdformat drift was this branch's own. The rebase onto the reformatted main also left four stray blank lines before headings in docs/developers-guide.md, which mdformat removes. Formatting only: whitespace-normalised content is unchanged in all three files. Co-Authored-By: Claude Opus 5 (1M context) --- docs/adr-001-lock-file-naming.md | 7 +++---- docs/developers-guide.md | 4 ---- docs/terms-of-reference.md | 10 +++++----- 3 files changed, 8 insertions(+), 13 deletions(-) diff --git a/docs/adr-001-lock-file-naming.md b/docs/adr-001-lock-file-naming.md index 295b1d1..dd21254 100644 --- a/docs/adr-001-lock-file-naming.md +++ b/docs/adr-001-lock-file-naming.md @@ -64,8 +64,8 @@ can safely create under contention. before the authoritative sidecar configuration read, repairing tails, rotating, compressing, or appending. - Permit an unlocked, advisory pre-read of the sidecar `timeout_ms` value - solely to choose the lock-acquisition timeout; it never substitutes for - the authoritative read. + solely to choose the lock-acquisition timeout; it never substitutes for the + authoritative read. - Treat `.lock` journal paths as invalid because `.lock` is reserved for coordination artefacts. - Treat `.toml` journal paths as invalid when the derived sidecar path would be @@ -180,8 +180,7 @@ configuration must choose distinct stems or directories, such as 2. Implement lock-path derivation as a pure function and test the examples in Table 1. 3. Use the derived lock path for the complete critical section: the - authoritative sidecar read, tail repair, rotation, compression, and - append. + authoritative sidecar read, tail repair, rotation, compression, and append. 4. Document reserved suffixes and sidecar sharing in the users' guide when the CLI implementation lands. diff --git a/docs/developers-guide.md b/docs/developers-guide.md index e4db3bd..e2d2cd5 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -2,7 +2,6 @@ This guide explains the contributor workflow for the generated mpsc-log project. - ## mpsc-log implementation architecture `mpsc-log` is planned as a small domain core surrounded by adapters. The domain @@ -15,7 +14,6 @@ defines so contributors can place new code correctly. the adapters, runs the domain, and translates the semantic error type into an exit code. It carries no parsing, merging, rotation, or filesystem logic. - ### Planned modules | Module | Layer | Responsibility | @@ -29,7 +27,6 @@ exit code. It carries no parsing, merging, rotation, or filesystem logic. | `clock` | Domain port | Declare the `Clock` port; an infrastructure implementation supplies the real instant. | | `fs` | Adapter | Implement `JournalStore` over the real filesystem, alongside the fault-injection test double. | - ### The adapter boundary The domain never names an external API. It works only with the domain `Record` @@ -60,7 +57,6 @@ When adding code, put format and effect handling in an adapter and keep policy in the domain. If a domain module needs a new external effect, add a port rather than importing the concrete API. - ### Further reading - [Design](mpsc-log-design.md) specifies the CLI contract, record model, diff --git a/docs/terms-of-reference.md b/docs/terms-of-reference.md index 0d4ed07..ae8622d 100644 --- a/docs/terms-of-reference.md +++ b/docs/terms-of-reference.md @@ -168,12 +168,12 @@ artefact rather than trusting a caller's informal summary. next boundary. - Define local generation retention on the writer's filesystem, as follows. - For size-only rotation, retain the newest rotated generations (four by - default, via `plain_generations`) as plain files, and gzip older - retained generations (per `compressed_generations`). + default, via `plain_generations`) as plain files, and gzip older retained + generations (per `compressed_generations`). - For scheduled rotation, retain every size-split file in the newest - completed periods (four by default, via `plain_generations`) as plain - files, gzip only older retained periods (per `compressed_generations`), - and never compress files in the current period. + completed periods (four by default, via `plain_generations`) as plain files, + gzip only older retained periods (per `compressed_generations`), and never + compress files in the current period. - Read a sidecar TOML file for rotation configuration, schema-guided type coercion, and default field values. - Surface failures through stable exit codes and diagnostics suitable for