RFC: Chronicle depends on shared control plane; TokenOps stays above Chronicle

[susheem-k]

Summary

Proposal: introduce a shared control plane package as the system of record. Chronicle depends on it (persist envelopes). TokenOps continues to depend on Chronicle (govern via @boundary hooks) and the same plane (ledger / budgets / Admin).

Packages (depend-on →):

  tokenops  →  chronicle  →  control-plane (new or extracted)

Runtime shape:

@boundary → Chronicle session → plane.persist(envelope)
                ↓ on_enter / on_crossing
            TokenOps governor → plane.ledger / budgets / halt

Today TokenOps embeds “plane + SDK + UI” and Chronicle is freestanding (in-memory / optional JSONL EnvelopeStore). This RFC extracts the store/API so both products write to one DB without TokenOps being the only persistence path for traces.

Why

Package	Owns	Must not own
control-plane	Shared SQLite (or remote API): runs, registrations, envelopes, ledger accumulators, governance config seed	Agent frameworks, Streamlit UI, OpenAI SDKs
chronicle	@boundary, envelopes schema, record/replay, session hooks	Budgets, HALT policies, pricing
tokenops	Policies, tokenops_run, Admin/Dashboard, pricing books	Re-implementing envelope storage

Design principles

1. One DB file / one TOKENOPS_URL — envelopes and spend share run_id / trace_id.
2. Chronicle stays useful offline — plane client has an embedded mode (local SQLite) and optional HTTP mode; no plane process required for unit tests / replay fixtures.
3. Hooks stay Chronicle-owned — on_enter / on_crossing / on_leave remain on the session; TokenOps (and later others) register adapters. Plane persistence is a Chronicle→plane writer, not TokenOps-only.
4. No circular imports — control-plane has zero imports of chronicle/tokenops. Chronicle never imports tokenops. Tokenops imports both.
5. Stable IDs — envelope trace_id ↔ TokenOps run_id (or explicit foreign key); document the join in Admin.

Phased plan

Phase 0 — Spec (1–2 days)

• Name the package (agentplane-control / agent-control-plane — pick one).
• Schema sketch:
  • Keep today’s TokenOps tables (runs, budgets, ledger_*, policy_instances, …).
  • Add envelopes (or chronicle_envelopes): envelope_id, trace_id, run_id?, boundary_id, kind, sequence, payload_json, recorded_at.
• API sketch (minimal):
  • register_run / get_run (already TokenOps-shaped)
  • append_envelope(trace_id|run_id, envelope)
  • list_envelopes(run_id|trace_id)
• Decide: embedded SQLite only in v1 vs HTTP from day one (recommend embedded first, HTTP mirrors TokenOps ControlPlaneClient later).
• Write an ADR: “Chronicle depends on plane; TokenOps depends on Chronicle; no TokenOps→plane bypass for envelopes.”

Exit: ADR + OpenAPI/table list reviewed.

Phase 1 — Extract control-plane package (3–5 days)

• Lift from TokenOps: Store SQLite schema, migrations/seed, ControlPlaneClient embedded path (and thin HTTP if already there).
• Publish agent-control-plane 0.1.0 (or keep private until TokenOps absorbs it).
• TokenOps depends on it internally first (re-export Store / from_env) — behavior unchanged, Chronicle still unused by plane.
• CI: TokenOps green on extracted store.

Exit: TokenOps uses extracted plane; no Chronicle change yet.

Phase 2 — Chronicle → plane for envelopes (3–5 days)

• Chronicle depends on agent-control-plane>=….
• Session option: plane: ControlPlaneClient | None; if set (or env), each record_envelope also append_envelope.
• Keep in-memory + optional JSONL EnvelopeStore for fixture workflows; plane is additive.
• Bind trace_id to ambient run when TokenOps (or caller) set it — or Chronicle creates a plane “trace” row if no run yet.
• Tests: boundary LIVE → row in plane DB; replay still works from fixtures without plane.

Exit: pip install agent-chronicle pulls plane; demo writes envelopes to SQLite.

Phase 3 — TokenOps uses Chronicle+plane as the story (2–4 days)

• Ensure tokenops_run + install_crossing_hook:
  • Opens/uses same plane client Chronicle will write to (TOKENOPS_DB / TOKENOPS_URL).
  • Sets session plane handle (or env) so envelopes land in the same DB as ledger.
• Admin Dashboard: “Run detail” shows envelopes + spend (read list_envelopes).
• Deprecate dual mental model in docs: “JSONL is for fixtures; plane DB is product truth.”
• Bump: chronicle minor, tokenops minor; release order: plane → chronicle → tokenops.

Exit: One tokenops.db has runs + ledger + envelopes; UI shows both.

Phase 4 — Harden & optional HTTP (ongoing)

• Multi-process: plane HTTP; agents + Chronicle client append envelopes remotely.
• Retention/redaction: plane applies Chronicle redactors before persist.
• Migration guide for existing EnvelopeStore JSONL users.
• Consider extracting UI into tokenops only (plane stays headless).

Release / versioning order (every cut)

1. agent-control-plane
2. agent-chronicle (depends on plane)
3. agent-tokenops (depends on chronicle + plane, or chronicle only if plane is transitive)

Never release TokenOps before Chronicle if TokenOps needs new Chronicle session APIs.

Risks & mitigations

Risk	Mitigation
Chronicle PyPI users forced into plane	Embedded plane default; no server required; fixture path unchanged
Circular features (plane wants Chronicle types)	Plane stores opaque JSON (+ indexed columns); Chronicle owns schema validation
TokenOps Store vs plane drift	Single implementation; TokenOps becomes a thin re-export
Bigger install / coupling	Accept for AgentPlane product; document “Chronicle standalone = embedded plane only”

Non-goals (v1)

• Rewriting replay to require the plane
• Moving policies into Chronicle
• Making Streamlit part of control-plane
• Breaking @boundary transparency contract

Suggested first milestone

“One SQLite, two readers”: extract plane + Chronicle append_envelope + TokenOps Dashboard tab listing envelopes for run_id — still no Chronicle→TokenOps import.

Open decisions

1. Package name / repo for control-plane (new repo vs monorepo package).
2. Must run_id == trace_id, or separate with a join table?
3. Is JSONL EnvelopeStore kept forever or deprecated after plane ships?
4. Embedded-only for first release, or HTTP parity with today’s TOKENOPS_URL?

Related

• TokenOps already consumes Chronicle @boundary / on_enter / on_crossing (e.g. agent-tokenops ≥0.1.2 + agent-chronicle ≥0.3.0).
• Testbench showcase: LLM @boundary on live providers; TokenOps Admin shows ledger only — envelopes currently stay in-process unless EnvelopeStore / export_trace is wired.