diff --git a/_698-PRO-REVIEW-R2.md b/_698-PRO-REVIEW-R2.md new file mode 100644 index 00000000..7ac04ff5 --- /dev/null +++ b/_698-PRO-REVIEW-R2.md @@ -0,0 +1,191 @@ +# GPT-5.6 Pro round-2 review of RFC v2 — VERDICT: REQUEST CHANGES +Extracted 2026-08-16 from https://chatgpt.com/c/6a81be80-1ba0-83ea-89fe-6d9a5f5cdde4 +(sent 11:21 ET, "Worked for 17m 41s"; full text verified against page) + +## Round-diff of its 15 findings +3 ADOPTED-CORRECTLY (dependency graph/plan lock; kernel capability typing; +shared core + country extensions), 11 ADOPTED-BUT-MUTATED, 1 LOST. + +The LOST one: the semantic-versioning triad — v2's four identity classes +"solve a different problem"; semantic authority_version has no field, bump +rules, public-contract diff, precedence, or cache/logbook effects. + +Key mutations flagged: +- Node keys adopted BUT global spec_binding also flows into QRF/transfer-bank + bindings and "resume requires equality of all four classes" — global + identities must describe the run, node identities must determine reuse; + both gating reuse defeats node granularity. +- Canonicalization strong but no schema-migration translator, compiler + implementation identity, or immutable schema-content digest. +- Five surfaces sound (fifth = chain state, good) but the YAML examples + violate the separation (k, device, Supabase store, HF destination inside + bundle files; sim batch size double-classified). +- Catalogs re-declare owner/lineage class = new drift pair; entity keys, + membership cardinalities, artifact keys, executable row-scope language + missing. +- legacy-v1 as named-stream map = documentation unless enforced (see (d)). +- Equivalence: fixed the tautology but needs FOUR builds (constants A/B, + bundle C/D; require A=B, C=D, A=C) to separate authority difference from + nondeterminism; comparison set must derive from plan.lock, not "the three + H5 files"; must include model banks, final release artifact, manifests, + mass history, receipts. +- ELIGIBILITY GUARD TOO WEAK: family-level required_concepts:[eligibility] + with a generic eligibility tag does NOT make the CHAMPVA defect + structurally impossible (drop veteran_va, keep own_coverage → still + passes). Must be target-specific concept sets (CHAMPVA → veteran_status + + military_coverage_context). +- Vintages: no typed compatibility constraints; engine pin duplicated + between vintages.yaml and sources.yaml. +- Calibration/exact-k: names, not an executable mathematical contract (no + scaling, weighting, zero/negative targets, optimizer, dtype, stopping, + infeasibility, target priority; post-selection weight semantics absent). +- Sealed attempts: "sealed" conflicting with a running status; no atomic + seal, idempotency, state transitions, orphan reconciliation; release DAG + never actually defined as distinct from the strict-linear audit chain + (needs derived_from / supersedes / revokes). +- Machine-decidable gates: metrics still lack populations, formulas, + baselines, support minima, uncertainty, thresholds, missing-slice rules. + +## Fusion adjudication +(a) Four classes = sound vocabulary, wrong composition: grammar already +folded into spec_sha256's envelope; global kernel_set can invalidate +unrelated nodes; runtime lock/output contract duplicate other classes; node +key lacks root-seed VALUE / behavior-relevant run inputs. Prescribes: +run_provenance_identity {source_grammar_receipt, global_resolved_spec_digest, +global_code_inventory_digest, global_artifact_protocol_inventory, +run_request, execution_receipt} for receipts/logbook, and node_reuse_key = +H(compiler_ir_abi_and_digest, resolved_transitive_node_slice, +behavior_relevant_run_inputs, transitive_input_content_hashes, +per_node_implementation_and_dependency_digest, +rng_protocol_and_actual_seed_material, input_and_output_artifact_contracts, +per_artifact_materializer_abi, output_sensitive_backend_abi) for reuse. +Schema migrations: schema_version selects an immutable migration chain with +recorded IDs + implementation digests; semantics-preserving migration +changes the audit receipt, not node reuse. + +(b) Execution profile class in the node key contradicts the surface model. +Proven byte-invariant → attempt receipt only. Can change bits → resolved +backend/numeric ABI in the AFFECTED node keys. Suspected → temporary +cache-compatibility fence, never called output-invariant. device:auto can +never be output-invariant by declaration; receipt the resolved backend, +dtype, library stack, deterministic-algorithm mode; CPU/GPU share a key only +after per-kernel byte-equality conformance. Release labels never invalidate +computational nodes. + +(c) F0 fast path = dual-authority trap as written, salvageable: manual +constants+bundle dual-editing is not honest authority. F0 needs nearly the +complete compiler FRONT END (parse/canonicalize/defaults/migrate, cross-ref, +typed entities/artifacts/scopes/columns, full stage DAG + producer graph +compile, seed-protocol resolution, complete normalization, compile-to-legacy +adapter covering every normative field, usage/coverage report proving no +field ignored, round-trip + mutation tests) — though not the executor. Safe +fast path: author the bundle ONCE; the compiler GENERATES the legacy payload +the constants-era executor consumes; record config_authority=constants_adapter +until F1; call spec_sha256 a mirror-attested configuration identity, not +proof the bundle executor built the artifact. + +(d) legacy-v1 normative only if the runtime is FORCED to obey it: versioned +RNG broker; direct np.random/SeedSequence/random/framework RNG construction +outside the broker fails static + runtime checks; stochastic kernels receipt +consumed stream IDs. Normative: draw-site IDs, site→stream map, literal base +seeds, RNG family/version, spawn count/index, consumed order, reset/reuse +boundaries, entity/clone ordering, digest width/endianness/encoding, and the +immutable protocol implementation ID + digest. Rationale/code-paths = +descriptive. + +## Fresh attack (new MAJORs) +1. Five-surface model contradicted by example schemas + migration map (k, + device, logbook store, HF destination, batch size) — compile five + physically separate typed objects; hash only the normative projection; + declared precedence for default-vs-run-request values. +2. fingerprint → spec_sha256 boundary named, not specified: gen-0 keeps raw + fingerprint; gen-1 raw = transport/package-integrity receipt only; + spec_sha256 = semantic authority receipt; node reuse via compiled slices; + {legacy_fingerprint, canonical_spec_sha256} compatibility map; NO second + resource manifest (country_package.json only); immutable digests for + schemas/migrators/composition. A Belgian SMOKE BUILD must precede the + identity cutover (compile-only insufficient). +3. Producer graph: field coverage ≠ graph-semantic losslessness — needs + explicit read-after-write edges, deterministic order for incomparable + nodes (commute-or-disjoint rule), ordered fallback precedence with + exhaustive/disjoint predicates, temp/validation-only outputs, + entity-key/cardinality effects, typed link/membership/order/weight/mass + mutations, unique ownership per CELL SEGMENT, retry/idempotence; plus a + closed executable row-scope predicate algebra (labels like acs_rows can't + police writes otherwise). +4. Take-up enum mixes ownership, mechanism, invocation optimization, policy + interaction, and an untyped escape hatch (dedicated_stage undermines + "kernels are the only escape hatch"; batched_seeded isn't a treatment; + near_universal is a rate/edge policy). Fix: orthogonal ownership + (measured|transferred|modeled|engine) × ordered typed pipeline-step kinds + (probability_seed, count_calibration, …) + dependence group + + final_owner_stage. Column/entity derivation needs a total-and-injective + assertion. +5. Catalogs/vintages = authored drift pairs: split catalog into normative + column contract / derived lineage report / non-normative docs; vintages = + typed reference records (tax_period_ref, survey_period_ref, + target_period_ref, geography_vintage_ref, policy_engine_surface_ref, + release_series_ref) with validated compatibility relationships. +6. Equivalence gate: four-build structure; plan-derived comparison set; and + adversarial rare/boundary fixtures (CHAMPVA-scale donor scarcity, + empty/saturated take-up domains, county complements, crosswalk + boundaries, overlapping producer fallbacks, zero/negative calibration + targets, infeasible exact-k, clone tails, mixed-ownership columns) — a + generic small fixture + one f004 may never exercise the motivating + failures. +7. Deletion checklist additions: no is-guards/constant imports/alternate + dispatch; node-invalidation matrix (unrelated edit reuses unaffected + nodes, invalidates descendants); schema-migration + old-bundle reader + fixtures; four-build determinism; fault injection around + checkpoint/manifest/seal/promotion/logbook append; cross-profile + byte-equivalence for every claimed-invariant profile; RNG-broker + + ambient-read enforcement; real Belgian build + UK walking-skeleton + EXECUTION (not compile). +8. Publication state machine: append-only attempt events → one immutable + terminal seal (running ≠ sealed); temp artifact namespace, atomic + manifest seal, output verification, idempotency key, promotion + transaction, recovery for seal-ok/db-fail and db-ok/alias-fail, orphan + + expiry reconciliation; strict_linear stays the tamper-evident AUDIT + sequence, a separate release relationship graph (derived_from, + supersedes, revokes) is the release DAG. +9. Machine-decidable gates: per-gate formula, input artifact/stage, + population/denominator, slices, reference release/digest, minimum + support, absolute+relative thresholds, uncertainty/multi-seed rule, + missing-slice treatment, fail/warn/report status, typed reason mapping; + calibration math contract; exact-k post-selection weight semantics. +10. Generic executor cannot enforce ambient reads (module globals, env, + files, network, time, independent RNG) — brokers for file/env/clock/RNG; + orthogonal capability fields: determinism | numeric_reproducibility | + effects | structural_delta (none|filter|expand|join|relink|reorder| + reweight) | retry_safety; structural_effect replaced by the specific + delta + pre/postconditions. +11. MINOR: D6 can't stay an untyped open decision inside normative YAML — + resolve before golden bundles freeze, or mark the release line + explicitly provisional/non-normative and excluded from spec_sha256. + +## Final verdict +REQUEST CHANGES — "v2 is not ready for owner sign-off as an implementation +contract … directionally much stronger than v1 … the remaining blockers are +internal contradictions in identity, authority, and rollout — not polish. +The most serious is that v2 promises per-node reuse while still globally +binding spec_sha256 and all four identity classes into checkpoint and bank +eligibility." + +Ranked minimal v3 list: (1) separate run provenance from node reuse; (2) +restore the versioning triad with precedence/bump rules + immutable +schema/migrator/canonicalizer/compiler digests; (3) make F0 single-authored +(bundle → generated legacy payload; full front-end coverage, compile-back, +usage accounting, mutation tests before CHAMPVA-class edits use the path); +(4) reconcile the five surfaces mechanically; (5) legacy-v1 executable via +RNG broker + draw-site inventory + receipts; (6) correct domain schemas +(target-specific eligibility concepts; orthogonal take-up; no +dedicated_stage); (7) complete graph/contract semantics (row-scope predicate +language, deterministic ordering, fallback disjointness, segmented +ownership, derived catalogs, typed vintage references); (8) strengthen +certification/deletion (four builds, plan-derived comparison, adversarial +fixtures, node-invalidation tests, cross-profile determinism, migration +fixtures, crash/recovery injection, real BE + UK smoke executions); (9) +finish publication semantics (attempt events, atomic seal, idempotent +promotion, audit chain ≠ release DAG); (10) executable quality/calibration +gates; (11) resolve D6 or exclude the provisional release line from +normative identity. diff --git a/_698-SOL-REVIEW-R2.md b/_698-SOL-REVIEW-R2.md new file mode 100644 index 00000000..628ad416 --- /dev/null +++ b/_698-SOL-REVIEW-R2.md @@ -0,0 +1,709 @@ +# Round-2 adversarial review of the spec-engine RFC v2 (PR #698) + +Reviewed locally at `d865ba40` against repository source and the three held +wave-1 branch heads: `per-family-predictor-sets` at `016cb662`, +`lineage-column-closure-697` at `323f6c69`, and +`block-first-geography-696` at `7fff4489`. I did not use the network, a virtual +environment, a pipeline build, or certification inputs. This is a +design/source review, not an implementation review. + +**Bottom line: request changes.** V2 fixes several important R1 defects, but it +is not sign-off-ready. Three R1 findings are asserted away without a schema +capable of representing the current system, and six more are only partial. The +new machinery adds independent blockers: F0 contains the compiler work assigned +to F1, the node key can collide across different stochastic run requests, the +frozen split ledger is factually wrong (five/37, not four/32), take-up engine +validation is circular, catalogs/vintages recreate dual authority, and the +equivalence vector stops before final publication. + +## 1. Round-diff audit + +Verdict count: **11 RESOLVED, 6 PARTIALLY RESOLVED, 3 +RENAMED-NOT-RESOLVED, 0 REGRESSED**. + +| R1 | Verdict | V2 text | Code-grounded adjudication | +|---|---|---|---| +| M1 | **RESOLVED** | `docs/spec-engine.md:217-231,630-635` puts one namespaced `spec_binding` in configured/base identity, makes both authorities load the same bundle before branching, and replaces singleton canonicality. | This fixes the R1 ordering contradiction. Current namespace routing is `_configured_stacked_identity` (`tools/build_us_multispine_pool.py:1150-1178`); stage discovery reconstructs and exactly compares `_stacked_checkpoint_base_identity` (`:1043-1147,1181-1244`); stage and bank identities inherit that base (`:1298-1322,2602-2633`). Loading and hashing before the current configured-identity call at `:4192` is implementable. The real `is`-based guards are at `stacked_spine.py:3089-3147,3349-3376,3672-3695`, and v2 explicitly replaces them. The missing concrete `identity_generation` field is charged to M12 rather than double-counted here. | +| M2 | **RESOLVED** | `docs/spec-engine.md:623-652` now requires two cold isolated roots, forbids checkpoint/model-bank resume, compares all three deterministic stage H5 files (or an exhaustive frame digest), normalizes terminal gates, and emits mismatch diagnostics. | This addresses both original failures for the original three pool cutpoints plus terminal gates: a changed ordinary cell is covered by checkpoint content, and run two cannot resume run one. The serializer is intentionally deterministic (`frame_checkpoint.py:1-7,91-155,318-396`) and current shared-root discovery/resume is real (`tools/build_us_multispine_pool.py:4192-4239`). V2's broader bundle now claims publication and downstream surfaces outside that original vector; that new scope insufficiency is N6 below. | +| M3 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:258-279` correctly preserves `legacy-v1` for the flip and defers stateless `derived-v2` to F3; `specs/us/bundle.yaml:13-14` selects it. | The phasing defect is fixed, but the promised named-stream declaration does not exist, and a static stream map cannot declaratively reproduce code-internal, data-dependent consumption. QRF creates two `SeedSequence` children and advances one shared fit RNG in target order (`qrf.py:1077-1098,1128-1148,1333-1380,1428-1429`); ACS uses exact NUL-separated labels, SHA-256 truncation, and little-endian decoding (`acs_transfer.py:2902-2916`). V2 does not say what is spec-normative, kernel-contract-normative, or merely receipt-descriptive. | +| M4 | **RESOLVED** | `docs/spec-engine.md:408-442,669,719-735` and `specs/us/geography.yaml:3-50` put block-first geography and the ASEC complement in F3 and require exact legacy geography through the flip. | This is coherent with the current gap: the legacy block ladder lacks PUMA and samples inside a prior CD, while the PUMA ladder preserves/draws PUMA and never assigns a block. An implementer no longer has to fake block-first behavior during byte equivalence. The held branch's failure to include the later complement ruling is a rollout issue (N8), not a recurrence of M4's phase error. | +| M5 | **RESOLVED** | `docs/spec-engine.md:152-182` defines closed-world typed resolution, canonical envelope/file boundaries, numeric normalization, explicit ordered-vs-set fields, golden bytes/hashes, and preservation of the current serializers during equivalence; corrected ASEC/clone-role examples are at `:388-406`. | Following this contract avoids v1's ambiguous concatenation and ordering/default/numeric failures. The current serializers really differ (`tools/build_us_multispine_pool.py:1315-1322`; `stacked_spine.py:2349-2357`), and v2 correctly does not silently unify them at the flip. | +| M6 | **RESOLVED** | `docs/spec-engine.md:281-303,719-723,757-759` requires one projected-input/patch-output executor, full structural diffing, capability types, and adversarial read/write tests before bundle mode drives production. | This is an implementable replacement for the current metadata-only `ProducerContract` and opaque callback (`late_producer_dag.py:139-163,426-518`) and the full-frame dispatcher. V2 no longer claims today's specialized guards already provide the generic discipline. | +| M7 | **RENAMED-NOT-RESOLVED** | `docs/spec-engine.md:305-312,508-516,668` calls `producer_graph` lossless and promises byte-identical compile-back. | The only sketched record is not lossless. It omits `producing_stage` and tolerated receipt IDs, flattens the OR-of-AND `alternatives`, puts an invalid `value_kind: amount` on the logical input, spells `coverage_scope` as `coverage`, and invents a per-output `final_owner` Boolean. Actual fields are in `late_producer_dag.py:47-137,176-209`; kind-specific virtual resources and complete receipt semantics are in `us_late_producer_registry.py:1597-1736,2047-2103`; final ownership is an 18-row target × origin × clone-role matrix (`us_late_overlap_ownership.py:60-199,219-261`). An implementer following the shown shape cannot reproduce the current payload. | +| M8 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:325-354` adds a closed artifact profile, bidirectional checkpoint/final closure, predicates, skip receipts, canonical empty outputs, and exhaustive `(entity,column,row_scope,stage,write_policy)` segments. | Those provisions directly address mixed-cell ownership and conditional presence. But `:356-359` immediately reauthors one catalog row per column with `owner` and `lineage class` and says closure derives from it. That is again a whole-column second authority and conflicts with `:327-335`, where owner/class derive from producer outputs. The runtime segment model can be correct only if catalog owner/class become compiler-generated, not authored. | +| M9 | **RENAMED-NOT-RESOLVED** | `docs/spec-engine.md:519-558` and `specs/us/take_up.yaml:17-51` replace one draw per flag with a closed treatment enum and ordered pipelines. | SNAP is materially better, but the supposedly schema-conforming inventory is still false. `wic` and `social_security` are not among the 13 engine take-up programs; EITC is conditioned by approximated child count, not `filer_conditioning`; and `dedicated_stage` is an untyped escape hatch for unlike SSI, ACA, and SNAP mechanisms. SSI and housing cannot be represented honestly by one enum value without forced fits. The complete inventory follows below. | +| M10 | **RENAMED-NOT-RESOLVED** | `docs/spec-engine.md:128-150,678-679` and `specs/us/bundle.yaml:4-8` repeatedly call the bundle a `CountrySpec` extension and require package data/one composition boundary. | The intended consolidation is right, and v2 now explicitly retires the old lineage file, test, and emitter. But “new resource kinds” do not exist in the current manifest or loader: resources are bare filenames (`country_spec.py:835-839`), every resource is parsed as a JSON mapping (`:849-860`), typed behavior is selected by hard-coded filenames (`:862-907`), `CountrySpec` has fixed fields (`:756-784`), and compilation only builds source/geography `StagePlan`s (`:923-996`). YAML is currently forbidden by the package contract (`test_spec_only_country_packages.py:11,44-57,80-87`). `bundle.yaml.files` also adds a second file inventory beside `country_package.json.resources`. This is a loader/manifest/dataclass/compiler replacement unless v3 defines the adapter and consumer migration. | +| M11 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:659-686` adds the missing pipeline/runtime, PUF support/tail, resume/operations, calibration/selection, and default-leakage classes, plus a generated authority inventory and static allow/deny test; `:743-765` gates deletion. | The classification is much more complete. It is not yet total enough to authorize deletion because new surfaces and their old consumers have no explicit tombstones: catalog/vintage duplicates, country-specific producer constants/receipts, the two greedy splitters and max-width constant, engine ABI snapshots, and the full stochastic-callsite surface. N9 gives the required zero-reference gates. A general sentence saying an inventory assigns “every item” is not a deletion protocol. | +| M12 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:184-225` defines four jointly required identity classes and exact resume interaction; `:653-657,707-714,760-763` declares an `identity_generation` cutover. | The four-class policy is coherent with the repository's independent semantic/materializer/checkpoint versions and correctly adds `builder_code_identity`. But `identity_generation` has no field location, legacy default, reader branch, discovery behavior, or promotion/logbook representation; `rg` finds no source implementation. Current routing hashes configured identity and discovery reconstructs/exact-compares base identity (`tools/build_us_multispine_pool.py:1150-1244`). “Bump” alone does not make generation 0/1 machine-decidable. | +| M13 | **RESOLVED** | `docs/spec-engine.md:314-323,460-465,674` requires schema-complete adapters, every build-facing kwarg passed explicitly, a monkeypatched-default test, and a static invocation contract. | This directly closes the current leakage from QRF defaults at production call sites (`puf_qrf_chain.py:219-225`; `acs_transfer.py:1341-1345`). Standalone library convenience defaults remain appropriately code-owned. | +| M14 | **RESOLVED** | `docs/spec-engine.md:233-256,370-386,663,677` separates normative config, identity-bound run request, output-invariant execution profile, operational bindings, and external chain state; paths are bijective/hash-verified but not identity-hashed. | This matches current logical input pins versus absolute-path receipts and cleanly classifies checkpoint roots, credentials, paths, workers, and logbook predecessor. The contradictory inclusion of execution profile in the new node key is N2, not the original source/path conflation. | +| M15 | **RESOLVED** | `docs/spec-engine.md:106-115,146-150,707-710,735-736` requires a country-neutral core plus discriminated extensions, UK two-stage/OA geography and synthetic prior to be expressible before sign-off, a minimal UK compile, and Belgian compatibility compile. | This addresses the actual UK mismatch (`uk_runtime/geography_ladder.py:1-74,96-121`; `national_frame.py:61`; `spi_support.py:29-43`) rather than promising literal reuse of US shapes. F0's size is separately wrong, but the schema direction no longer forces US PUF/FIPS/take-up conventions onto the UK. | +| M16 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:630-638,649-657,724-728,743-765` names `--config-authority`, rejects interaction with `--legacy-two-spine`, requires both-mode PR fixtures, restricted cold certification, a full bundle-mode release, a cold generation cutover, and no retro-labeling. | Most lifecycle policy is now present. It is not fully implementable until M12's generation field/read protocol exists, and no retention/cleanup date is stated—only a checklist requiring one later. Without explicit absent/0/1/unknown reader rules, “generation 0 readable forever but never promotable” is prose rather than artifact policy. | +| MIN1 | **RESOLVED** | `docs/spec-engine.md:408-442` and `specs/us/geography.yaml:20-34` make `state_minus_identified_counties` the sole rule, name a pinned source id, and prohibit a sampled fallback. | The conflicting `else_state` rule is gone and the behavior is correctly postponed to F3. | +| MIN2 | **RESOLVED** | `docs/spec-engine.md:84-91` scopes closure to the selected country/profile registry namespace, always rejects unknown/duplicate referenced IDs, and permits unused library-only implementations. | This avoids failing a US load merely because UK, diagnostic, test, or general library kernels are installed. | +| MIN3 | **RESOLVED** | `docs/spec-engine.md:251-256` says failed resolution has no `spec_sha256`; failure rows carry status, attempted grammar/canonicalizer, a raw file-set digest when available, and the validation error. | This is coherent with terminal attempts being opened before validation (`tools/build_us_multispine_pool.py:3985-4015,4138-4167`). | +| NIT1 | **PARTIALLY RESOLVED** | `docs/spec-engine.md:365-368` relabels snippets “representative, schema-conforming” and says placeholders are gone or outside YAML. | The claim remains unreviewable because no schemas exist, 8 of 11 bundle files are absent, and `specs/us/take_up.yaml:48-51` still contains an ellipsis promising a future inventory. More importantly, three rows already present are factually invalid under the claimed exact engine-coverage rule. “Draft” is honest; “schema-conforming” is not yet supportable. | + +### M1/M12 pressure point: where `identity_generation` must live + +The same-bundle-before-branch rule now works, provided the loader runs before +`_configured_stacked_identity()` at `tools/build_us_multispine_pool.py:4192`. +The generation cutover needs one equally concrete representation: + +```text +configured identity: + identity_generation: 1 + spec_binding: {...} + +stacked checkpoint base identity: + identity_generation: 1 + spec_binding: {...} +``` + +`_discover_stacked_checkpoint_identity()` must pass both through when it +reconstructs the expected base identity. Stage identities and the primary-QRF / +ACS-transfer bank identities already derive from the base, so they inherit the +cutover. The same fields must be copied into outer `run_config`, checkpoint and +publication manifests, terminal attempts, and Logbook rows. Readers need a +closed rule: absent or `0` = historic/readable/non-promotable; `1` = binding +required and fully validated; unknown = refuse. Without this, discovery cannot +distinguish a genuine generation-0 artifact from malformed new output. + +### M3 pressure point: a behavior-preserving `legacy-v1` boundary + +A legacy declaration can preserve behavior, but it cannot truthfully expose +QRF's internal advances as independently addressable streams. + +- **Spec-normative:** literal/run-request seed source; which stages/families + share it; exact ACS label grammar, digest, truncation, and byte order; target + order; and a versioned kernel algorithm-contract ID. +- **Kernel-contract-normative:** `SeedSequence(seed).spawn(2)`, child roles, + bit-generator type, ordered traversal, and regime/data-dependent RNG + consumption. These stay protected by the kernel/code digest. +- **Receipt-descriptive:** realized derived seeds, target order, saved RNG + states, and checkpoint evidence. + +The flip should pass the declared base seed unchanged to the pinned legacy +kernel. It must not replace the shared generator with per-target streams. Add +golden ACS label/seed vectors and a full multi-regime QRF chain fixture. +`derived-v2`, not `legacy-v1`, is where streams become stateless declarations. + +### M7 pressure point: minimum lossless producer graph + +A compilable graph needs, at minimum: + +- graph-level external stages and the scope-coverage relation currently hashed + in `late_producer_dag.py:373-400`; +- each input's `entity`, `column`, `required_scope`, `producing_stage`, ordered + tolerated-absence receipt IDs, and alternatives as OR-of-AND lists of + `{entity,column,value_kind}`; +- outputs `{entity,column,coverage_scope}`; +- typed virtual resource declarations for manifests, resolved weights, + execution configs, transition/producer receipts, and target banks, including + each kind's semantic payload and digest rules; +- transfer groups and the full overlap matrix keyed by target, origin, and clone + role, with finalization plus every non-owner action; and +- the execution-receipt and transition-authority contracts in + `us_late_producer_schedule_payload()` (`us_late_producer_registry.py:2047-2103`). + +The RFC's example at `docs/spec-engine.md:508-516` must either be replaced with +that shape or be labeled non-normative pseudocode. In its present form the +promised compile-back equality is impossible. + +### M9 pressure point: actual take-up inventory + +The checked-in contract has exactly 13 programs +(`us/take_up_contract.json:17-207`); tests require exact equality with installed +engine metadata (`test_us_take_up_contract.py:33-38,152-168`). The actual +mechanisms across that contract, `source_stages.json`, and runtime modules are: + +| Actual program(s) | Current contract label | Actual production mechanism | Honest v2 representation | +|---|---|---|---| +| SNAP | `out_of_scope` | National reported-anchor/FNS-rate prior, then reporter-anchored state count calibration that overwrites the flag (`source_stages.json:1950-2036`). | Ordered two-stage pipeline; final semantic treatment can be `anchored_count_calibrated`. FNS counts are targets, not the “anchor” claimed at `take_up.yaml:32`. | +| TANF | `seed` | Generic stable-ID Bernoulli kernel at a scalar administrative rate (`take_up.py:242-294,307-386`). | `seeded_rate`; batching is a separate invocation-group property. | +| EITC | `seed` | Same kernel, with rate selected by approximated qualifying-child count (`take_up.py:200-239,262-269`). | `seeded_rate` + `rate_by_num_children`; not `filer_conditioning`. | +| Medicaid | `count_calibrated` | Reporter anchor, runtime state prior, greedy state count calibration (`source_stages.json:2788-2836`). | `anchored_count_calibrated`. | +| CHIP, Basic Health Program, DC PTC, Early Head Start | `rate_unsourced` | The engine's default-true leaf survives, with explicit source/debt follow-up. | `engine_default_with_debt`. | +| Medicare | `out_of_scope` | Measured ASEC `MCARE` mapping and support-clone propagation (`source_stages.json:2068-2109`). | `measured` pipeline. | +| SSI | `count_calibrated` in the contract | Reporter-anchored, target-derived age-band Bernoulli prior plus delivered-recipient gate; it explicitly **never count-matches flags** (`take_up_contract.json:121-145`; test at `test_us_take_up_contract.py:86-116`). | Needs a typed `target_derived_seeded_with_delivery_gate` operation/pipeline. `anchored_count_calibrated` is false. | +| Head Start | `out_of_scope` | Weighted QRF trained on direct SIPP response and transferred to frame identities (`source_stages.json:1357-1486`). | `imputed_transferred`, with donor/training/recipient scopes. | +| Housing assistance | `out_of_scope` | Measured ASEC receipt on one row surface and ACS/QRF-imputed receipt on PUF support; the flag equals the receipt (`source_stages.json:2390-2530`). | Mixed row-scope `measured` + `imputed_transferred`; one whole-column treatment is insufficient. | +| ACA | `out_of_scope` | Dedicated Marketplace assignments and several calibration operations (`source_stages.json:2680-2785`). | A typed dedicated pipeline, not a bare escape-hatch label. | + +No current contract program is `near_universal` or `model_simulated`. WIC and +Social Security are not contract programs. `batched_seeded` mixes execution +grouping with semantic treatment, and `dedicated_stage` makes the supposedly +closed enum open-ended. V3 should separate engine class/debt status, semantic +pipeline operations, row-scope ownership, and kernel invocation group, then +commit all 13 real rows. Column mapping must come from an engine ABI projection, +not the insufficient optional-suffix naming rule (`takes_up_eitc` and +`takes_up_dc_ptc` already show why). + +### M10 pressure point: this is a replacement unless the seam is explicit + +Current `CountrySpec` combines four assumptions: + +1. one flat `country_package.json.resources: [filename]` manifest; +2. JSON-object parsing for every declared resource; +3. filename-specific typed fields on the `CountrySpec` dataclass; and +4. source/geography-only `StagePlan` compilation. + +V2 adds another file manifest, YAML parsing, closed schemas/default injection, +arbitrary kernel registries, catalogs/vintages, a typed execution IR, node keys, +and canonical typed hashing. Calling that “new resource kinds” does not define +an extension: the existing manifest has no kind field. Minimal coherent wording +is: + +- replace the bare filename list with one authoritative typed resource table + `{path,kind,schema_id}`, or make the versioned bundle manifest itself replace + `country_package.json`; do not keep both file lists; +- return one `ResolvedCountrySpec` carrying both migration-era compatibility + projections (`sources`, `gates`, etc.) and the compiled spec-engine IR; +- intentionally update the spec-only package test to allow YAML and kernel IDs; +- define which generation exposes raw `fingerprint`, which exposes + `spec_sha256`, and migrate every validator/receipt/attestation consumer. In + particular, `gate_battery.py:787-792,958-983` currently emits a + `spec_fingerprint` composed only from `gates.json`, while + `docs/gate-battery-contract.md:89-90` calls it the whole country package; and +- list direct runtime consumers such as `load_take_up_contract()` and UK + `load_country_spec("uk").gates` (`uk_runtime/national_build.py:467-472`) in the + compatibility/deletion plan. + +V2 does get one part right: it explicitly retires the old lineage surface and +its consumers. The active direct consumers are +`packages/microcosm-build/tests/test_imputation_lineage_spec.py:31-124` and +`tools/emit_lineage_dashboard.py:21-83` (plus the stale package comment at +`packages/microcosm-build/pyproject.toml:20-22`). They must be replaced, not left +pointing at the retired file. + +## 2. New v2 findings + +All new findings below are **MAJOR**. There are no standalone new MINOR or NIT +findings: each defect can invalidate identity, equivalence, the authority flip, +or the promised fast path. + +### MAJOR N1 — F0's “fast mirror” contains F1's compiler/compile-back work + +**Evidence.** F0 promises closed schemas, the canonicalizer, one typed +`ResolvedSpec`, a full constants-generated US bundle, UK and Belgian compile +compatibility, the `CountrySpec` loader/wheel rewrite, both authority modes, +the identity cutover, a complete legacy seed map, and byte equality of all +resolved constants/bundle payloads (`docs/spec-engine.md:707-718`). F1 is then +said to introduce the producer graph, generic executor, compile-back, and +bundle-built authorities (`:719-723`). Those boundaries are circular: +high-level references, defaults, graph topology, ownership, and kernel config +can be compared to today's low-level plan/schedule payloads only after the +common typed IR and adapters/extractors exist. A textual YAML comparison would +not prove resolved equality. + +The current mirror is deliberately narrower. It checks ordered family/target +parity, predictor sets, a few model/default values, and computed producer +outputs (`test_imputation_lineage_spec.py:43-112`); it is not a complete plan +compiler. The existing `CountrySpec` incompatibilities in M10 make F0 larger +again. Likewise, `spec_sha256` cannot truthfully answer “what built this” at +`docs/spec-engine.md:715` while constants outside the compared subset still +drive behavior. + +**Rollout failure.** Either F0 balloons into F1 and blocks the CHAMPVA-class +work it exists to unblock, or F0 certifies a partial mirror while attaching a +whole-build hash and identity generation to unbound behavior. The latter is +worse: it gives a false provenance claim. + +**Concrete fix.** Define an honest F0a with one of two orders: + +1. Preferred: land the predictor/CHAMPVA value change first under today's #695 + mirror plus its required OOS/statistical gate, then generate the legacy + baseline from that commit. +2. If schema work must land first, F0a only packages/parses a narrow + `PredictorMirrorPayload` covering exact ordered families/targets, + predictor columns/blocks, and every model argument used by the held edit. + CI compares that payload to constants. Constants remain explicitly + authoritative and the digest is descriptive, not `spec_sha256`. + +Move the full `ResolvedCountrySpec`, all-file canonical bundle hash, +`plan.lock.json`, CountrySpec migration, global identity cutover, full seed +inventory, derived closure, dual authority, and plan/schedule/ownership +compile-back to F1. Alternatively rename the current F0 “compiler binding,” +estimate it as F1-sized, and stop advertising it as the value-fix fast path. + +### MAJOR N2 — The node key overkeys output-invariant profiles and underkeys run values + +**Evidence.** The node formula includes “execution profile class” +(`docs/spec-engine.md:211-214`), while `:243-245` defines execution profiles as +receipted and proven output-invariant. Conversely, root seed, rung/sample +fraction, `k`, and release label are identity-bound run-request values +(`:240-242`), but the node formula names only a `seed stream id`, not the root +seed or a value-bearing run-request digest. A stream identifier names a +derivation function; it does not distinguish seed 1 from seed 2. `plan.lock` +is described as compiled bundle IR (`:208-210`), so it cannot be assumed to +contain per-run values. + +Current checkpoint identities bind exact values: period/model seed, sample +fraction and seed, clone fraction and seed, engine, and inputs are explicit in +`tools/build_us_multispine_pool.py:1067-1093`; the configured namespace binds +the same run controls at `:1150-1165`. + +**Rollout failure.** Different stochastic runs can share a node key and resume +one another, while harmless worker/device/batch profiles unnecessarily split +cache namespaces. “Runtime lock” is too vague to repair the omission; it also +does not distinguish semantic dependency/code identity from operational +scheduling compatibility. + +**Concrete fix.** Define two records: + +```text +semantic node key = H( + identity_generation, + grammar + canonicalizer, + resolved semantic node plan, + exact consumed run-request values (including root seed), + direct input content hashes, + kernel ABI + implementation/output-affecting dependency digest, + seed stream id, + artifact/materializer + output contract +) + +attempt/scheduling receipt = { + semantic_node_key, + output-invariant execution profile, + operational bindings +} +``` + +If a device, batch size, worker count, or backend can change bytes, reclassify +that exact field as semantic/code identity and include it; do not include a +vague profile “class.” Golden tests should prove both that changing root seed +changes the key and that changing a certified output-invariant worker count +does not. + +### MAJOR N3 — The frozen split ledger is wrong, and F3 reunification changes more than RNG order + +**Evidence.** V2 says today's +`puf_tax_itemization__batch_1..4` are 32 targets +(`docs/spec-engine.md:479-483,732-734`). The committed mirror has **five** +batches: four groups of eight and `batch_5` with five—**37 targets** +(`specs/us_imputation_lineage.yaml:280-356`; the operator documentation confirms +all five at `docs/us-multispine-operator-ordering.md:724-728`). The overall +registry requires 19 bounded groups/70 targets +(`us_late_producer_registry.py:1393-1404`). + +The split is reproducible given today's ordered input and width: runtime +normalization preserves target order and sorts family names, then greedily +packs atoms while keeping the immigration pair together +(`acs_transfer.py:2268-2380`). But it is not yet an independent declaration: +the registry duplicates the splitter (`us_late_producer_registry.py:1338-1390`) +and ownership literals name particular batches +(`us_late_overlap_ownership.py:29-33`). Those copies can co-drift. + +Reunification has at least four behavioral effects: + +- the family label changes `_family_seed`/`_pattern_seed` for every target + (`acs_transfer.py:2902-2916`); +- QRF's shared RNG is consumed across a different target sequence + (`qrf.py:1077-1098,1333-1380`); +- the donor complete-case mask is now the intersection across all 37 targets, + not per old batch (`acs_transfer.py:1260-1270,1427-1439`); and +- later targets can condition on drawn targets across former batch boundaries + (`qrf.py:1523-1533`). + +**Rollout failure.** Literal implementation drops five live outputs before any +intended behavior edit and breaks registry/ownership equality. Treating F3 as +only “RNG consumption order” under-scopes donor selection, chained features, +bank identities/layout, and overlap-owner references. + +**Concrete fix.** At the flip, declare and golden the entire current ordered +split ledger—19 groups/70 targets overall and five/37 for tax itemization—with +an explicit reason for every frozen split. Compile it back against both current +splitter outputs and every literal owner/receipt consumer, then delete the two +greedy production splitters and width constant at F2. At F3 declare the full +37-target chain, assign fresh node/bank/materializer identities, and require +OOS/statistical tests that cover the new donor complete-case population, +family-label seed change, and cross-boundary chained predictors. + +### MAJOR N4 — Take-up derive-and-assert is circular on an engine bump + +**Evidence.** V2 removes authored `column`/`entity`, derives them from a naming +rule plus installed engine metadata, and asserts coverage against that same +installed metadata (`docs/spec-engine.md:530-557`; `specs/us/take_up.yaml:12-15`). +Today the checked-in contract is intentionally a reviewed snapshot of engine +facts: entity/value type/default/class keys are explicit +(`take_up_contract.py:49-58`), `assert_take_up_contract_current()` compares +every field and set against the installed engine (`:327-391`), and tests prove +the equality and its ability to fail (`test_us_take_up_contract.py:33-38, +152-168`). Deriving facts and then asserting them against their own source +removes that tripwire: existing variables whose entity/default/class changes +would be silently accepted unless another snapshot remains. + +The version authority is already split. Project metadata permits +`policyengine-us>=1.745.0,<2` (`packages/microcosm-build/pyproject.toml:30`), the +lock resolves 1.764.6 (`uv.lock:1364-1365`), the current contract records a +1.752.2 review vintage (`take_up_contract.json:5-9`), and v2 places 1.764.6 in +both sources and vintages (`docs/spec-engine.md:361-363,382-383`). + +**Rollout failure.** On an engine bump, either the compiler silently changes +the resolved take-up ABI, or unrelated duplicate pins fail in an undefined +order. A newly discovered default-true flag can ship without treatment if the +derived inventory is mistaken for reviewed bundle content. The optional-suffix +naming rule is also not a bijection for actual names such as `takes_up_eitc` +and `takes_up_dc_ptc`. + +**Concrete fix.** Choose one exact engine artifact/version pin. From precisely +that pin, compile a complete engine ABI projection +`{program_id -> variable, entity, type, default, engine_class, consumers}` into +a generated, committed, reviewable lock. CI compares a fresh derivation to the +lock before bundle compilation. Engine bumps require an explicit pin/ABI-lock +refresh and human review of all bundle-owned treatments/pipelines; added, +removed, or changed facts fail closed first. The engine owns its facts, the +bundle owns treatments and scope pipelines, and neither is derived from the +other. Other files refer to the one engine pin by ID. + +### MAJOR N5 — Catalogs and vintages recreate the drift pairs v2 claims to dissolve + +**Evidence: catalogs.** `docs/spec-engine.md:327-335` says closure owner/class +derive from bundle outputs and catalogs are human-facing. Lines `:356-359` then +author `owner` and `lineage class` in catalogs and make closure reporting derive +from them. That is a second `column_lineage.yaml` under another name. It also +conflicts with the repository rule that entity/dtype/period metadata come +through the RulesEngine adapter, never per-tool guesses (`DESIGN.md:71-72, +82-89`). + +**Evidence: vintages.** `docs/spec-engine.md:361-363` calls `vintages.yaml` the +one place for engine, geography, and “2024,” but the same RFC pins the engine in +`sources.yaml` (`:382-383`) and assigns the engine pin to sources in the +migration table (`:663`). The geography draft repeats `cd119` +(`specs/us/geography.yaml:35-40`). Current code has `POOL_TIME_PERIOD = 2024` +(`multispine_pool.py:238-242`), 2024 release parsing/IDs +(`tools/build_us_multispine_pool.py:290-293,1288-1292`), and +`CURRENT_CONGRESSIONAL_DISTRICT_VINTAGE = "119th_congress"` +(`congressional_district_vintage.py:16-25`). Source artifacts also carry their +own factual survey years, which should not be overwritten by one global year. + +**Rollout failure.** Catalog ownership can disagree with producer-graph +ownership, and an engine/year/geography change can update only one of +sources/vintages/code. If prose is part of `spec_sha256`, a spelling edit can +invalidate builds; if it is not, the schema still needs to distinguish the +non-normative overlay. + +**Concrete fix.** Authored catalog rows are documentation overlays keyed by a +compiler-derived stable column key: description, citations, and only display +units that are not engine metadata. The compiler injects entity, dtype/period, +presence profile, owner, row-scope segments, and lineage class. Hash prose in a +separate documentation digest, not normative `spec_sha256`. + +Make vintages a normalized reference/index over content-pinned source records: +survey/tax/geography vintages live on the source artifact they describe, and +other specs refer to a vintage/source ID. Put engine compatibility in exactly +one runtime/source-lock record and assert the installed engine against it. +Derive publication period from the dataset/run contract. Add a CI check that +rejects duplicate literal authorities for each normalized key. + +### MAJOR N6 — Three stage H5 hashes do not cover the bundle's claimed behavior, and resume-forbidden is underspecified and non-fail-fast + +**Evidence: output coverage.** The three durable stages are only `assembled`, +`transferred`, and `simulated` (`multispine_pool.py:200-201`). The transferred +checkpoint precedes derive/seed/simulate (`tools/build_us_multispine_pool.py: +2947-3175`); the simulated checkpoint is written at `:3182-3250`; terminal +gates run afterward at `:3264-3275`. V2 separately compares normalized gates, +which is good, but final publication is still outside the SHA vector: it uses a +different H5 materializer plus publication run ID and writes the final manifest +and diagnostics later (`tools/build_us_multispine_pool.py:3664-3748`). The +manifest explicitly says calibration is downstream (`:3507-3512`), even though +calibration/selection/publication are bundle surfaces in v2. + +Operational/model-bank evidence is deliberately excluded from canonical stage +H5 metadata: `_split_checkpoint_stage_receipts()` removes primary-QRF resume +status/routing and ACS target-bank receipts to a sidecar +(`tools/build_us_multispine_pool.py:2103-2152`). Thus H5 equality cannot prove +that both paths executed cold. + +**Evidence: enforceability.** Existing receipts are sufficient for a post-run +no-resume assertion, but v2 neither enumerates the required assertion nor +provides a fail-fast policy. Stage provenance exposes +`deepest_resumed_stage` and per-stage source +(`tools/build_us_multispine_pool.py:1614-1714`); primary QRF exposes an +aggregate `initialized|resumed` status (`:2326-2368,2462-2470`); and ACS target +banks expose per-target `load_status: resumed` / `source: checkpoint` +(`acs_transfer_bank.py:107-228,337-377`). The aggregate primary status is +adequate: the chain resumes a contiguous prefix when a manifest exists, and it +rejects a nonempty manifestless bank (`puf_qrf_chain.py:254-312`). A harness +can therefore require null `deepest_resumed_stage`, primary +`resume_status == initialized`, and no resumed/checkpoint-sourced ACS target. +What is missing is one specified policy/audit spanning those receipts and a +pre-load refusal; empty roots are only convention. + +**Rollout failure.** Bundle and constants modes can agree at the three pool +cutpoints while differing in final serialization/manifest behavior or a later +bundle-owned calibration/selection/release node. An implementer can also check +only stage provenance and miss model-bank reuse because “fail if either run +reports a resume” does not define the three-receipt assertion above. The gate +is enforceable today, but not specified as an executable predicate. + +**Concrete fix.** Define an equivalence artifact vector, scoped explicitly to +every node under bundle authority: canonical logical digest of every stage +frame; final published logical frame/schema/period/materializer; normalized +final manifest, diagnostics, and gates; compiler lock files; and downstream +calibration/selection/release artifacts when those are in the flip. Compare raw +H5 SHA only where the same deterministic materializer and normalized embedded +metadata are guaranteed. + +For F0, spell out the exact post-run predicate over the existing receipts: +null `deepest_resumed_stage`, primary `resume_status == initialized`, and no ACS +target with resumed/checkpoint source. Then add `--resume-policy=forbid` and +propagate it through stage discovery, primary QRF, and every ACS bank. In forbid +mode, reject any pre-existing manifest/stage/target before loading and emit one +typed `resume_audit` with per-stage and per-target attempted/resumed counts; +equivalence requires every count to be zero. The flag is fail-fast hardening, +not a prerequisite for enforcing the current post-run gate. + +### MAJOR N7 — The wave-1 predictor and closure lane order is internally inconsistent + +**Evidence: predictor lane.** The held `per-family-predictor-sets` branch is not +only a YAML mirror edit. It widens ACS PUMS required inputs with dozens of raw +fields +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/us_runtime/acs_pums.py:72-131`) +and passes the resulting person table into the Frame +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/us_runtime/acs_pums.py:297-341`). +It materializes 15 canonical carried predictor columns +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/us_runtime/cps_carried.py:111-135,617-666`), +and its pool test asserts they remain on the person table +(`per-family-predictor-sets:packages/microcosm-build/tests/test_us_multispine_pool.py:3670-3688`). Final H5 +serialization writes every Frame table/column (`us_runtime/h5_io.py:917-927`). +The lane also installs a new participation target order +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/us_runtime/multispine_pool.py:640-658,805-825`), changes +alternative precedence from sorted to declaration order +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/late_producer_dag.py:111-117`), and bumps the late +registry schema +(`per-family-predictor-sets:packages/microcosm-build/src/microcosm/build/us_late_producer_registry.py:102-130`). +Its byte-equal mirror therefore proves synchronization with code, +not that the behavioral edit is acceptable. Its own lane notes say no f025 OOS +sweep or artifact build was run +(`per-family-predictor-sets:_predictor-sets-LANE-NOTES.md:6-10,29-33`). + +**Evidence: closure lane.** The held `lineage-column-closure-697` branch pins a +392-column f025 inventory +(`lineage-column-closure-697:packages/microcosm-build/tests/test_lineage_column_closure.py:54-79`). A direct set +comparison finds 56 raw/canonical predictor names introduced by the predictor +branch absent from that fixture. Live-H5 comparison is opt-in and normally +skips without `MICROCOSM_LINEAGE_POOL_H5` +(`lineage-column-closure-697:packages/microcosm-build/tests/test_lineage_column_closure.py:265-272`). More fundamentally, the +closure test reads the authored lineage file, dynamically imports the retiring +dashboard emitter, and reasserts the old take-up contract +(`lineage-column-closure-697:packages/microcosm-build/tests/test_lineage_column_closure.py:20-46,81-185,187-237`). +That is the opposite of v2's compiler-derived closure. + +**Rollout failure.** Landing closure at F0 and predictors after F0 either makes +the first value edit immediately fail the frozen profile or silently leaves the +fixture stale. Transplanting the closure branch as-is also preserves the three +authorities v2 says F0 dissolves. + +**Concrete fix.** Land the predictor/CHAMPVA work before freezing the F0 +baseline, with its real OOS/statistical acceptance gate. Then either: + +- declare/catalog its added columns, regenerate a certified inventory, and + treat the change as artifact-profile/materializer work; or +- make predictor features typed transient/virtual resources and prove the + executor strips them before checkpoints/final H5. + +Only `emit_artifact_column_inventory.py` and the versioned inventory fixture +from #697 can land unchanged. Retarget exact closure to compiler-emitted +profiles/segments and compiled take-up after the full compiler exists in F1. + +### MAJOR N8 — The held block-first lane does not contain the owner-ruled ASEC complement + +**Evidence.** The held branch explicitly records that the v3 ASEC checkpoint +has only `household_id`, `state_fips`, and `H_TENURE`, then chooses state-only +fallback (`block-first-geography-696:_696-LANE-NOTES.md:33-40`). Its kernel says +ASEC rows draw within the state because identified county is unavailable +(`block-first-geography-696:packages/microcosm-build/src/microcosm/build/us_runtime/geography_ladder.py:1-7,326-345`) +and receipts `asec: state` / `asec_county_status: absent_from_v3_checkpoint` +(`block-first-geography-696:packages/microcosm-build/src/microcosm/build/us_runtime/geography_ladder.py:494-516,596-602`). +The checkpoint loader actively rejects any county field until a dedicated +schema change +(`block-first-geography-696:packages/microcosm-build/src/microcosm/build/us_runtime/asec_checkpoint.py:179-182,207-229`). +The branch therefore cannot implement v2's +`state_minus_identified_counties` rule (`specs/us/geography.yaml:20-34`). + +It is also not “bundle content,” as `docs/spec-engine.md:738` says: the lane +adds a new block artifact schema/source pin, assignment/validation kernel, +checkpoint identity fields, and source/checkpoint plumbing. + +**Rollout failure.** Merging the held branch at F3 would violate the revised +owner ruling by sampling unidentified ASEC households over the whole state, +including counties they are known not to inhabit. Treating it as config-only +would omit required artifact and checkpoint-schema changes. + +**Concrete fix.** Keep F3, but describe this as a coordinated kernel + ASEC +checkpoint-schema/source + block-artifact + bundle migration. Add a bound +official identified-county source, county-identified and state-complement +universes, refusal for empty complements, leakage/complement tests, new +identity/materializer versions, and cold certification. Rework/rebase the held +branch; do not merge it as the implementation of the final YAML. + +### MAJOR N9 — The F2 deletion checklist lacks zero-reference gates for the new surfaces + +**Evidence.** `docs/spec-engine.md:743-765` gives good global conditions but no +specific tombstones for several authorities introduced or absorbed by v2: + +- **Country composition:** raw `CountrySpec.fingerprint`, the gate-battery + `spec_fingerprint`, bare manifest filenames, and old JSON compatibility + projections need an explicit generation/consumer migration (M10). +- **Lineage/dashboard:** current consumers are + `test_imputation_lineage_spec.py:31-124` and + `tools/emit_lineage_dashboard.py:21-83`; held predictor/closure branches add + more tests against that file. The package comment at + `packages/microcosm-build/pyproject.toml:20-22` remains stale, and the external + dashboard handoff is not verified in this repo. +- **Producer graph:** country-specific `CANONICAL_US_LATE_*`, schedule/ownership + receipt constructors, and their tool/runtime imports survive. For example, + the stacked tool consumes the schedule at + `tools/build_us_multispine_pool.py:1105,2621,3015`; the generic + `ProducerInput`/`ProducerOutput`/DAG validators should remain, but the US + declaration constants must not. +- **Frozen splits:** both greedy splitter implementations and + `DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT = 8` remain live + (`acs_transfer.py:88-91,939-942,2338-2380`; + `us_late_producer_registry.py:1338-1396`). +- **Take-up:** `take_up_contract.json`, its loader/currentness assertions, and + the absorbed `source_stages.json` rows remain direct authorities until every + runtime consumer uses compiled bundle projections. +- **Seed protocol:** v2's summary of 578/0/42 plus ACS/QRF is not an exhaustive + production callsite inventory. Reachable source stages also own SSI training + and model seeds (`ssi_disability_criteria.py:241-243`, duplicated in + `source_stages.json:1331-1337`), vehicle/asset stable-string hash algorithms + (`sipp_vehicles.py:299-314`; `sipp_financial_assets.py:307-322`), the ACS-rent + archived hash (`housing_inputs.py:740-756`), the tips training seed + (`sipp_tips.py:114-120`), and the SCF composite `SeedSequence` + (`scf_wealth.py:830-855`). A central map would become a third copy if those + literals remain. +- **Catalogs/vintages:** existing code/docs pins must be removed or converted to + references after the one-authority model in N5 exists; root `specs/us` + drafting copies must disappear after package-data migration. + +**Rollout failure.** F2 can satisfy the eight broad sentences while leaving old +runtime loaders/constants active. The build then still has two authorities even +though the selector and mirror tests—the mechanisms most likely to expose the +drift—have been deleted. + +**Concrete fix.** Make F2 machine-decidable with generated inventories and +zero-reference/tombstone tests: + +1. every package resource/file belongs to exactly one typed manifest and only + the generation-appropriate composition identity is emitted; +2. repository search finds no nonhistorical reference to the root lineage file, + old emitter, or old dashboard schema; the external dashboard is verified + against the compiled catalog/export; +3. no production import/reference to country declaration constants, old + ownership/schedule receipts, greedy splitters, or the width constant remains; +4. every reachable stochastic/hash-draw callsite consumes a resolved stream + token, with code retaining only versioned algorithm implementations and + golden vectors; no build seed/default literal remains outside the audited + legacy-kernel contract; +5. no direct take-up-contract/source-stage authority survives outside the + compatibility reader for generation 0; and +6. no duplicate engine, period, geography-vintage, catalog-owner, or lineage + literal survives. Generated lock files are regenerated outputs, never a new + authored authority. + +## 3. Rollout-order adjudication + +The current phase labels are not credible as written. + +| Work item | Actual size/home | Required order | +|---|---|---| +| Narrow predictor/model mirror sufficient for CHAMPVA-class edits | Honest F0a | Land the held predictor behavior change first (with OOS/statistical acceptance), or land only the narrow mirror schema/parser and keep constants authoritative. | +| Full schemas + canonical `ResolvedCountrySpec` + US/UK/BE compatibility + one CountrySpec manifest/loader + all-payload equality | F1-sized compiler binding | Must precede global `spec_sha256`, bundle authority mode, and the identity-generation cutover. It cannot be split from compile-back by calling the comparison a mirror. | +| `plan.lock.json`, producer graph, generic executor, plan/schedule/ownership compile-back | F1 | Compile-back fixtures must exist before bundle mode can construct authorities. | +| `legacy-v1` seed contract | F1 unless F0 is renamed compiler-sized | Build the reachable stochastic-callsite inventory and golden legacy-kernel vectors before claiming full equivalence. | +| #697 inventory tool/fixture | F0a, **after** predictor materialization is settled | Rebuild/certify the fixture against the post-predictor artifact profile. | +| #697 derived closure/segments/dashboard | F1 | Retarget to compiler outputs; do not land the authored-class test/emitter as held. | +| Held predictor branch | First behavior lane, but not a byte-equality acceptance proof | Treat loader columns, retained artifact columns, target order, kernel changes, and profile/materializer effects explicitly; run its promised OOS/statistical gate. | +| Held block-first branch | F3 coordinated behavior/artifact/schema change | Rework it to include identified-county/complement semantics; it is not merely a bundle diff. | +| Five-to-one tax-itemization chain | F3 | Correct the frozen baseline to five/37 first, then gate the full donor/seed/chain/bank behavior change. | + +Accordingly, the advertised order “closure at F0; predictor right after F0” is +reversed at the artifact boundary. The inventory baseline cannot precede a lane +that adds persistent columns unless those columns are declared transient and +removed before every checkpoint/publication surface. + +The deletion checklist is **not complete**. N9's zero-reference gates should be +added as explicit F2 conditions, alongside these final cleanup requirements: + +1. one typed country resource manifest; no second `bundle.files` inventory; +2. an explicit generation-0 compatibility reader and generation-1 + `ResolvedCountrySpec`, with every fingerprint/spec-hash consumer assigned; +3. no root drafting bundle after package-data installation; +4. old lineage file/emitter/tests and external dashboard schema migrated; +5. old take-up contract/source rows and direct loaders migrated or retained only + in the generation-0 reader; +6. US producer/schedule/ownership constants, splitters, and max-width constant + absent from production; +7. catalog/vintage/pin literals reduced to one authority plus references; +8. exhaustive seed-callsite inventory at zero unbound callsites; +9. authority selector retained through a certified full release and deleted only + after a dated retention deadline; and +10. generated bundle/plan/ABI locks reproducible from their authorities and + rejected if hand-edited. + +## 4. Final verdict and minimal v3 change list + +**V2 is not sign-off-ready for the owner.** Its architectural direction is now +substantially better than v1, but following the current document can still +produce a false equivalence proof, stale stochastic cache reuse, dropped live +targets, an engine-sensitive take-up plan with no review tripwire, and a +CountrySpec implementation that is a rewrite hidden behind the word +“extension.” + +The minimal v3 change list is: + +1. **Make F0 honest.** Either reduce it to a narrow, non-authoritative predictor + mirror or rename/estimate it as the full compiler-binding phase. Do not attach + whole-build `spec_sha256`, identity generation, or dual authority before the + full compile-back surface exists. +2. **Specify the CountrySpec replacement seam.** Use one typed resource manifest + and one file inventory; define YAML/package tests, `ResolvedCountrySpec` + compatibility projections, and the complete raw-fingerprint → canonical-spec + consumer migration (including gate battery and UK/BE readers). +3. **Put identity on concrete fields.** Add `identity_generation` and + `spec_binding` to configured and base identities and all readers/receipts; + define absent/0/1/unknown behavior. Rewrite the node key to include exact + consumed run-request values/root seed and exclude output-invariant execution + profiles. +4. **Draw the legacy RNG boundary.** The spec owns seed sources/sharing, label + grammars, target order, and algorithm IDs; pinned kernels own internal + consumption. Commit an exhaustive stochastic-callsite ledger and golden + ACS/QRF/source-stage vectors. +5. **Replace the producer-graph sketch with a genuinely lossless schema.** Cover + nested alternatives/value kinds, producing stages, receipt IDs, virtual + resource payloads, scope coverage, transfer groups, the full conditional + ownership matrix, execution receipts, and transition authority. Require a + golden byte-identical compile-back fixture. +6. **Replace the take-up examples with all 13 real programs.** Separate engine + ABI facts, semantic pipeline operations, row-scope ownership, and invocation + grouping; add an exact pinned-engine ABI lock and fail engine bumps before + plan compilation. Do not use `dedicated_stage` as an untyped escape hatch. +7. **Correct split facts to five/37 (19/70 overall).** Freeze the exact current + ordered groups as declarations. Describe F3 reunification as a donor-mask, + family-seed, shared-RNG, chained-feature, owner, and bank-identity change—not + only a consumption-order change. +8. **Remove new dual authorities.** Catalogs are non-normative documentation + overlays; owner/class/entity/dtype/period are compiled. Vintages live on + pinned source records and are referenced by ID; engine compatibility appears + exactly once. +9. **Expand the equivalence contract.** Compare the declared vector through + final published logical H5/manifest/diagnostics and every downstream node in + flip scope. Add an enforced end-to-end `resume-policy=forbid` and one complete + per-stage/per-target resume audit. +10. **Correct the lane map.** Settle/validate predictor behavior before freezing + the inventory; move derived closure to the compiler phase; rework #696 for + the ASEC complement and identify it as coordinated code/artifact/schema + work. +11. **Make F2 deletions machine-decidable.** Add the explicit tombstones and + zero-reference/unbound-callsite gates in N9, plus a dated authority-selector + retention deadline. +12. **Stop calling drafts schema-conforming.** Commit schemas and a complete US + bundle (including all take-up rows and correct split ledger), or label all + incomplete excerpts pseudocode until those artifacts exist. + +With those amendments, the owner could sign off the architecture before the +implementation exists. At `d865ba40`, the document still asks the owner to sign +off several invariants that its shown schemas and rollout cannot satisfy. diff --git a/_698-SOL-REVIEW.md b/_698-SOL-REVIEW.md new file mode 100644 index 00000000..dbf4a801 --- /dev/null +++ b/_698-SOL-REVIEW.md @@ -0,0 +1,791 @@ +# Adversarial review of the spec-engine RFC (PR #698) + +Reviewed locally at `952d5add` against `main` at `f7173120`. I treated +`packages/microcosm-build/src/microcosm/build/us_runtime/` as `us_runtime/` +below. This was a source-and-design review only; I did not run the restricted +build or use the network. + +**Bottom line: request changes.** The authority flip is a sound direction, but +the RFC cannot preserve current behavior or current identity as written. The +two known ordering holes are real, the proposed equivalence assertion is not a +behavioral equivalence assertion, the generic kernel discipline does not yet +exist, and the draft would create a second spec system alongside the packaged +`CountrySpec` system already in this repository. + +## MAJOR findings + +### MAJOR 1 — Rollout steps 2 and 3 cannot both be true, and a loaded bundle cannot currently pass the production authority guard + +- **Claim or gap:** Step 2 joins `spec_sha256` into checkpoint identity while + both authorities are live; step 3 then requires the constants-built and + bundle-built runs to have identical identities (`docs/spec-engine.md:244-258, + 283-290`). The RFC does not say that constants mode loads the same bundle + before the execution paths branch. It also assumes a bundle can reconstruct + today's authority objects. +- **Code evidence:** The stacked base identity is assembled in + `tools/build_us_multispine_pool.py:1043-1147`; a stage identity is exactly + that mapping plus `stage` and `stage_index` at `:1298-1308`, and canonical JSON + is hashed at `:1311-1322`. Discovery requires exact identity equality at + `:1181-1244`, while the configured identity determines the checkpoint + namespace at `:1150-1178`. Separately, production authority is recognized by + Python object identity: `_production_stacked_authority` compares module + constants with `is` at `us_runtime/stacked_spine.py:3089-3147`, + `_authority_receipt` sets `canonical_identity = authority is + _canonical_authority` at `:3349-3376`, and production rejects false at + `:3672-3695`. A content-equivalent object constructed by a loader is therefore + non-canonical today. +- **Rollout failure:** If only bundle mode carries the new hash, the two logical + identities differ and the gate fails before comparing behavior. If neither + path carries it, step 2 has not happened. If the loader creates new plan + objects, production fails even before that contradiction. Adding a top-level + `schema_version` also collides with the existing checkpoint-envelope field of + that name (`tools/build_us_multispine_pool.py:1068-1070`). +- **Concrete fix:** Load, strictly validate, resolve, canonicalize, and hash the + same committed bundle **before** selecting an authority implementation. Put a + namespaced object such as + `spec_binding:{country,schema_id,schema_version,canonicalizer_version,spec_sha256}` + into both `_configured_stacked_identity` and + `_stacked_checkpoint_base_identity`; allow that base binding to flow into the + QRF/transfer bank bindings. Record the binding in outer-stage `run_config` + (`outer_stage_runtime.py:227-282,587-608`), not in structural `FrameIdentity`. + `--config-authority=constants` should use an adapter from the same + `ResolvedSpec`, assert that its resolved payload equals the legacy payload, + and differ only in plan construction. Replace singleton canonicality with + loader provenance plus live component digests, semantic authority version, + and the spec binding. Preserve the existing nested authority receipts during + the equivalence window; do not replace them with the one bundle hash. + +### MAJOR 2 — “Identical checkpoint identities and gate outputs” is not a frozen-behavior proof and can become a resume tautology + +- **Claim or gap:** The RFC calls identity equality plus gate-output equality a + frozen-behavior proof (`docs/spec-engine.md:252-260`). Neither side of that + conjunction is an exhaustive digest of stage output. +- **Code evidence:** For an f004 build, the current logical identity contains: + + - envelope fields: artifact kind, pool-checkpoint schema v1, stacked + materializer v11, pipeline, period 2024, model seed 0, and the installed + `policyengine-us` version (`tools/build_us_multispine_pool.py:1067-1078`); + - the six verified roles `asec_raw_stage`, `acs_household`, `acs_person`, + `acs_rent_donor`, `processed_puf`, and `puf_source_year`, each with actual + SHA-256 and byte size (`:689-746,1079`, `:1011-1019`); + - sampling fraction `0.04`, rung token, literal sample seed, the full stack + manifest and its digest, clone fraction/seed, and the complete stacked + authority receipt (`:1080-1093`); that authority receipt in turn carries + authority id/version plus live/declared component digests and payloads for + the gap plan, post-PUF transfer/producer surfaces, declared surface, + metric/joint registries, support/tail contract, and late schedule + (`us_runtime/stacked_spine.py:3317-3471`); + - `pool_code`: operator/pre-clone/post-clone/derive order, gap-fill and late + schedules, late resource semantics, remaining-stage input manifest, primary + QRF target order/schema, ACS earnings and QBI contracts, the entire take-up + contract identity, capital-gains tail schema/support contract, both estimator + counts, max targets per fit, and simulation batch size (`:1094-1146`); + - the cut point (`assembled`, `transferred`, or `simulated`) and its index + (`:1298-1308`; stage order is `us_runtime/multispine_pool.py:200-201`). + + That is a strong configuration/authority identity, but it does not hash all + transferred or simulated cells. `FrameIdentity` is also structural: it hashes + entity/link schema, ordered IDs, memberships, clone/source provenance, and + their dtypes (`outer_stage_runtime.py:611-651,821-852`); it omits ordinary + columns, weights, strata values, frame metadata, and the mass log. The + checkpoint writer separately stores frame metadata/receipts and computes the + **file** SHA-256 (`tools/build_us_multispine_pool.py:1499-1553`). Its serializer + deliberately makes equivalent frames byte-identical + (`frame_checkpoint.py:1-7,91-150`). Finally, if the harness reuses the same + caller-selected/default checkpoint root, equal configured identity routes + both runs to the same store (`tools/build_us_multispine_pool.py:1169-1178, + 4200-4238`), and `load_deepest` can make the second run load the first run's + result. Release IDs and terminal receipts contain timestamps/UUIDs/build IDs + (`:1274-1295,3569-3570,4067-4086`), so whole publication envelopes are not + naturally byte-equal either. +- **Rollout failure:** A changed imputed value can leave identity unchanged and + can evade aggregate gates. Worse, the bundle run can resume the constants + run and “prove” equality without executing its own path. Conversely, + comparing unnormalized publication manifests yields false failures from + nonce fields. +- **Concrete fix:** Run both modes cold at the same commit/dependency/input pins + in separate, initially empty checkpoint roots, and fail the proof if either + run reports a checkpoint or model-bank resume. Compare the SHA-256 of all + three deterministic checkpoint H5 files. If any operational metadata must + differ, instead compare an exhaustive canonical frame digest covering every + table value, column order/dtype/index, link, weight, stratum, metadata, mass + record, and canonical stage/input receipt. Also compare canonical terminal + gate payloads, with only an enumerated set of timestamps, paths, run IDs, and + authority-mode receipts excluded. Emit per-table/per-column diagnostics on a + mismatch. Keep the two logical identities equal by recording authority mode + only in a separate operational receipt. + +### MAJOR 3 — D5 seed derivation is necessarily a behavior change and cannot participate in the equivalence flip + +- **Claim or gap:** D5 replaces current seeds with + `hash(root_seed, stage_id, family_id)` (`docs/spec-engine.md:244-250`), while + the frozen gate requires unchanged output. +- **Code evidence:** CLI sampling and clone-attachment seeds default to 578 + (`tools/build_us_multispine_pool.py:496-517`). The same sample seed restarts + the ASEC and ACS samplers (`us_runtime/stacked_spine.py:619-631`), which use + `np.random.default_rng(seed)` (`frame_sampling.py:252-268`). Partial PUF clone + attachment has another literal generator (`us_runtime/puf_support.py:710-742`). + Pool imputation/source/take-up uses the separate constant seed 0 + (`us_runtime/multispine_pool.py:238-242,2970-2990`), and that 0 is bound into + checkpoint identity (`tools/build_us_multispine_pool.py:1073,1110-1113`). ACS + transfer already derives family/pattern seeds, but with a precise, different + protocol: SHA-256 over NUL-separated labels, first four bytes little-endian + (`us_runtime/acs_transfer.py:2902-2916`). QRF then uses + `SeedSequence(seed).spawn(2)` and consumes one fit RNG in target order + (`microcosm-fit/qrf.py:1077-1107,1128-1148,1333-1355,1428-1429`). PUF aggregate + disaggregation uses literal seed 42 (`us_runtime/puf_source_agi.py:21-52, + 379-403`). The RFC's word “hash” specifies no digest, encoding, width, + endianness, label grammar, or collision/domain-separation rule. +- **Rollout failure:** A root seed of 578 cannot reproduce the current shared + `578`, `0`, and `42` regimes through the proposed generic rule. It also breaks + today's deliberate ASEC/ACS stream sharing and changes QRF results because + target order controls shared-RNG consumption. The equivalence gate must fail + if D5 is actually wired. +- **Concrete fix:** The equivalence release needs a versioned + `seed_protocol: legacy-v1` with explicit named streams: shared ASEC/ACS survey + sampling `578`, clone attachment `578`, pool/QRF/source/take-up `0`, PUF + aggregate allocation `42`, legacy geography seeds, and the existing ACS/QRF + substream algorithms. Hash and expose that resolved map in lineage. Only + after the authority flip and constant deletion should a separate, + intentionally behavior-changing bundle edit select `derived-v2`. Define + domain-separated, length-prefixed UTF-8 inputs, SHA/HMAC choice, output width, + byte order/range, label normalization, and golden vectors; bump the applicable + authority/materializer identities, invalidate checkpoints, and use + statistical/OOS gates rather than legacy byte equality. + +### MAJOR 4 — `geography.yaml` describes behavior no existing geography kernel or artifact can perform + +- **Claim or gap:** The skeleton requires ACS block draws inside observed PUMA, + ASEC county/complement draws, a tract-to-PUMA assertion, and block-derived + layers before gap fill (`specs/us/geography.yaml:3-32`). Rollout step 3 puts + that #696 content inside the supposedly equivalence-gated flip + (`docs/spec-engine.md:288-290`). +- **Code evidence:** The block ladder contains no PUMA or tract-to-PUMA field + (`us_runtime/geography_ladder.py:77-119`). Its current assignment samples + blocks inside an already assigned congressional district + (`:236-305`) and writes block/tract/county/place/SLD/CBSA, but not PUMA, + state, or CD (`:326-334`). The current PUMA ladder instead preserves observed + ACS PUMA, draws PUMA for state-only rows, and draws CD/county within PUMA; + tract is optional and it never writes block/place/SLD/CBSA + (`us_runtime/puma_ladder.py:293-383`). The ACS path runs transfer before pooled + PUMA geography (`us_runtime/acs_multispine.py:127-165`) and explicitly records + block and tract as unresolved for ACS (`:188-218`). No current artifact or + kernel implements `identified_county_set` or + `state_minus_identified_counties`. +- **Rollout failure:** Reusing the block kernel loses the observed-PUMA + invariant; reusing the PUMA kernel cannot produce the declared block-first + surface. Either choice changes assignment, schema, predictor availability, + RNG consumption, and downstream imputation. This is not a representation-only + flip. +- **Concrete fix:** First encode and equivalence-gate the exact legacy geography + behavior (including any current no-op in the stacked tool). Move block-first + geography and the ASEC complement ruling to the post-flip intentional-change + phase. Before enabling it, produce a versioned block artifact that includes an + exact 2020 PUMA relationship, pin the official CPS identified-county source, + implement explicit ACS and ASEC row scopes, derive every final layer from the + selected block, and assert observed ACS state/PUMA. Name both assignment and + validation kernels in the spec. + +### MAJOR 5 — Canonical construction is underspecified, and the examples already encode identity- and behavior-changing values + +- **Claim or gap:** The loader computes a hash over “canonical concatenation” + (`docs/spec-engine.md:65-68`) but gives no canonicalization contract. It also + assumes a semantically equivalent bundle will construct identity-stable + payloads. +- **Code evidence:** Current build identity converts tuple/list to JSON arrays, + sorts sets by canonical JSON, normalizes NumPy/enum/path scalars, rejects + non-finite floats, and uses compact UTF-8 JSON with `ensure_ascii=False` and + sorted keys (`tools/build_us_multispine_pool.py:1315-1322,3769-3802`). Stacked + authority digests use a different serializer whose default is + `ensure_ascii=True` (`us_runtime/stacked_spine.py:2349-2357`). Some registries + sort explicitly, but plan/family/target sequences retain declared iteration + order (`:2137-2163,2273-2299,3264-3314`). That order is behavior-load-bearing + for QRF (`microcosm-fit/qrf.py:1087-1098,1523-1533`). Channel order is also + structural: clone validation requires exactly two channel-major arms in the + supplied order (`outer_stage_runtime.py:675-733`). Yet `spine.yaml` is shown as + a mapping and says `mass_anchor_channel: acs` (`docs/spec-engine.md:99-108`), + while production defaults to ASEC (`us_runtime/stacked_spine.py:542-552, + 677-703`). The late-transfer example declares donor clone 0 + (`docs/spec-engine.md:172-177`), while production declares PUF tax-detail clone + 1 (`us_runtime/support_provenance.py:31-35`; `stacked_spine.py:3390-3405`). + Finally, eight of the ten files named by `specs/us/bundle.yaml:7-17` do not + exist and there is no `specs/schema/`, so there is no executable unknown-key, + default, or canonical-type policy to review. +- **Rollout failure:** Two reasonable loaders can hash or execute different + bundles because of map order, file boundaries, `1` versus `1.0`, absent versus + null/defaulted fields, Unicode/paths, or set/list treatment. An alphabetical + channel compiler can reverse clone order; the literal RFC mass anchor and + donor clone already change behavior. Merely adding a spec-origin string to a + receipt can make deterministic checkpoint bytes differ. +- **Concrete fix:** Before schema sign-off, commit complete closed-world JSON + Schemas (`additionalProperties:false`), a complete legacy-equivalent US + bundle, compiling UK and Belgian compatibility bundles, invalid fixtures, and + golden canonical bytes/hashes. The + normative algorithm should: + + 1. parse one YAML 1.2 document per file; reject duplicate keys, merge keys, + custom tags, non-string keys, implicit timestamps, and non-finite numbers; + 2. validate types and inject every schema default into one typed + `ResolvedSpec`; reject unknown fields before hashing; + 3. hash a domain-separated envelope containing canonicalizer id/version, + schema id/version, country, and a map from normalized POSIX-relative file + names to typed values—not ambiguous raw concatenation; + 4. sort object keys; normalize tuple/list to arrays; normalize each number to + its schema-declared integer/float type; validate lowercase SHA-256 and + canonical IDs; require NFC identifiers rather than silently trimming or + case-folding arbitrary strings; + 5. preserve ordered arrays exactly for stages, channels, directions, + families, predictors, targets, fallback alternatives, and absence rules; + only fields explicitly declared as mathematical sets may deduplicate/sort; + 6. state whether status/notes/documentation are normative hash input, and + keep authority payloads as structured objects rather than JSON strings. + + During equivalence, reuse today's receipt constructors/serialization exactly. + Unifying their serializers is a later identity-format change. Replace the + examples with ASEC as the legacy mass anchor and the named + `puf_tax_detail` support role (not a raw clone integer). + +### MAJOR 6 — The RFC's generic kernel write discipline does not exist + +- **Claim or gap:** The RFC says kernels receive only declared inputs, write + only declared outputs, and that existing ownership/tail guards already enforce + the write side (`docs/spec-engine.md:231-242`). +- **Code evidence:** `ProducerContract` is metadata only; it has no callable or + parameter schema (`us_runtime/late_producer_dag.py:139-163`). + `run_producer_when_ready` validates readiness counts and invokes an opaque + zero-argument callback without projecting reads or diffing writes + (`:426-518`). The stacked dispatcher closes over and passes the full `Frame` + and hard-codes a `contract.kind` chain (`us_runtime/stacked_spine.py:9331-9383`). + Post-execution it checks declared outputs for type/presence/content evidence, + not mutations to undeclared columns, weights, links, strata, metadata, or mass + history (`:9445-9471`; declared-only evidence at `:5972-6043`). CPS source + operators happen to merge only declared outputs + (`us_runtime/multispine_pool.py:2681-2789`), but their row projection still + contains all columns (`:2110-2130,2362-2392`) and whole-pool operators replace + the frame. Existing overlap guards are specialized to particular education, + retirement, transfer, and tail surfaces (`:2441-2454,2490-2623`; + `stacked_spine.py:10250-10681`). The tail guard is invoked after the late DAG + and again at later stage boundaries, not around every kernel + (`tools/build_us_multispine_pool.py:3117-3120,3198-3201,3214-3217, + 3233-3236,3262`). QBI does have a strong undeclared-surface diff guard, but it + is hard-coded to one fixed QBI output set rather than supplied by the registry + (`us_runtime/qbi_inputs.py:1075-1148`). +- **Rollout failure:** A registered kernel can read an undeclared predictor or + mutate an unrelated existing cell and still satisfy every current check. The + bundle's IO declaration would be false, so “kernels are the only escape + hatch” would merely hide behavior behind a named id. +- **Concrete fix:** Route every build kernel through one generic executor. Pass + an immutable, schema-aware projection containing only declared physical and + virtual resources; require a returned patch/output object rather than an + arbitrary replacement frame; snapshot/diff all tables, links, IDs/order, + weights, strata, metadata, and mass history; and reject every change outside + declared entity/column/**row** scopes. Output contracts need policies such as + `fill_missing`, `overwrite_scope`, `assert_equal_noop`, and + `structural_effect`. Registry records must bind callable id, explicit + implementation/contract version, parameter schema, IO schema, and supported + spec range. Retain tail/overlap checks as extra invariants and add adversarial + tests for undeclared reads/writes and structural mutation. + +### MAJOR 7 — The late-producer migration row is lossy + +- **Claim or gap:** The migration map says the canonical late registry/groups/ + schedule become only imputation families plus `computed_producers` + (`docs/spec-engine.md:268-272`); the example gives bare input/output lists + (`:178-183`). +- **Code evidence:** Current inputs include entity/column, value-kind, + `required_scope`, `producing_stage`, alternative physical column sets, and + tolerated-absence receipt ids (`us_runtime/late_producer_dag.py:55-119`). + Outputs include coverage scope (`:122-137`). Optional inventory rows become + producer-bound absence receipts (`us_runtime/us_late_producer_registry.py: + 1559-1594`). The identity payload carries overlap ownership, execution-receipt + rules, schedule order/waves/edges, transfer groups, and every source/primary/ + ACS/transfer inventory (`:2047-2103`). +- **Rollout failure:** Two compilers cannot reconstruct the same readiness + semantics or current schedule receipt. One may choose first-present, another + any-present, and a third zero-fill. Optional inputs can become fatal or + silently optional; row-scope ownership and non-owner actions disappear. That + changes behavior and checkpoint identity before any intended edit. +- **Concrete fix:** Add a lossless `producer_graph` schema that can represent + every current `ProducerInput`, `ProducerOutput`, virtual resource, alternative, + value-kind, absence receipt, coverage scope, final owner, and non-owner action. + Derive order/waves from it. Before flipping, compile the bundle back into the + current schedule/ownership payload and require byte-identical canonical + payloads and receipts. + +### MAJOR 8 — Whole-column closure is insufficient for mixed ownership and conditional outputs + +- **Claim or gap:** `column_lineage.yaml` gives each artifact column exactly one + class and closure is checked against one committed inventory and the final + artifact (`docs/spec-engine.md:223-229`). The RFC does not define expected + presence by profile or cell-scope ownership. +- **Code evidence:** ACS transfer explicitly preserves every existing non-null + target and fills only null cells (`us_runtime/acs_transfer.py:894-900, + 992-1043,3006-3035`). Generic take-up likewise preserves measured/source-owned + non-null cells and fills only missing cells (`us_runtime/take_up.py:307-386`). + Thus one physical column can contain measured-native and imputed cells under + different owners. Presence can also be conditional: `tract_geoid` is written + only when `assign_tract=True` (`us_runtime/puma_ladder.py:81-82,293-383`), and + its gate changes the expected set under the same flag (`:496-498`). ACS + transfer skips fitting and returns the canonicalized recipient when no target + is active (`acs_transfer.py:943-975`), + and an absent ACS source returns the base frame (`us_runtime/acs_multispine.py: + 98-103`). I found no current `f001`/`f004`/`f010`/`f025`/`f100` branch that + changes column presence: those tokens select sample fractions/release grammar + (`tools/build_us_multispine_pool.py:277-293,496-505`). The concrete current + conditional is configuration/profile-dependent (`assign_tract`), not + rung-dependent. +- **Rollout failure:** A column-level class cannot enforce who may write which + cells. A one-sided “no unclaimed extras” check can miss required absent + outputs, while equality to one fixture rejects legitimate profiles. Two + implementers can disagree about whether a skipped kernel must materialize an + all-null/empty column. +- **Concrete fix:** Identity-bind a closed artifact profile and resolve the + expected set at load. Every output/lineage declaration must be `required`, + `forbidden`, or guarded by a closed spec predicate; runtime closure compares + expected and actual in both directions at each checkpoint and final output. + Conditional skips require a canonical skip receipt and required outputs must + materialize with canonical dtype even for zero-row scopes. Keep a primary + whole-column classification for documentation, but add exhaustive, + non-overlapping lineage/ownership segments over + `(entity,column,row_scope,stage,write_policy)`. + +### MAJOR 9 — `take_up.yaml` is factually false and assumes the wrong decomposition + +- **Claim or gap:** The only concrete row says SNAP is one per-flag + `snap_state_take_up` draw, with an eligibility interaction and + `calibration_status: none` (`specs/us/take_up.yaml:4-12`). The RFC says every + `takes_up_*` flag has that mechanism/source shape and leaves the existing + contract as a parameter source (`docs/spec-engine.md:185-198,274`). +- **Code evidence:** The hashed take-up contract is itself a curated authority + over engine facts, treatments, rates, calibration targets, scope owners, and + debt states (`us/take_up_contract.json:1-27`; + `us_runtime/take_up_contract.py:123-230`). SNAP is marked `out_of_scope` for + generic seeding because a national reported-anchor/rate prior is followed by + a dedicated state count-calibration stage (`take_up_contract.json:19-27`). + The separate source manifest declares the national prior stage and then the + state anchored count-calibration stage, including unmasked assignment and an + eligible-only calibration domain (`us/source_stages.json:1950-2036`). + That stage derives a runtime state prior as target divided by weighted modeled + eligibles (`us_runtime/snap_state_take_up.py:186-210`), runs anchored assignment + plus count calibration, and overwrites the final flag (`:225-305`). Eligibility + is the calibration/engine domain, not the entire assignment universe. The + generic seeder is batched across all `seed` programs and special-cases EITC + (`us_runtime/take_up.py:242-294,307-386`). Across the contract, source + manifests, and runtime modules—not all as literal treatment-enum labels—other + flags are measured, transferred, count-calibrated, unsourced/defaulted, + near-universal, or owned by dedicated stages (`take_up_contract.py:60-68` and + the program inventory). +- **Rollout failure:** An implementation faithful to the YAML either omits SNAP + state calibration, changes the assignment universe, or secretly continues to + read the JSON contract/source manifest. Editing YAML may do nothing, leaving + two authorities. The shown per-flag schema has no invocation/group id, + deduplication rule, ordered pipeline, or final-owner semantics, so two + compilers can invoke a batched kernel once, once per flag, or not at all, and + cannot agree on a prior plus later finalizer. +- **Concrete fix:** Replace `draws` with a discriminated program inventory and + ordered mechanism pipelines. Required treatments include at least measured, + imputed/transferred, seeded-rate, batched-seeded, anchored-count-calibrated, + engine-default-with-debt, near-universal, and out-of-scope/dedicated-stage. + For SNAP declare reported anchor, national prior, stable-source-ID draw + universe, runtime state-prior derivation, eligibility calibration domain, + FNS target source, saturation rule, final-owner stage, and diagnostics/gate. + Separate output ownership from kernel invocation so one kernel may own many + flags and one flag may have multiple stages. Either absorb the curated + contract and relevant `source_stages.json` rows into the bundle or make those + exact resources bundle components; do not duplicate a subset while calling + the old authority a mere parameter source. Retain exact coverage checks + against installed engine metadata. + +### MAJOR 10 — The RFC creates a second country-spec system and a second composition hash + +- **Claim or gap:** The RFC presents root `specs//` plus a new loader as + the shared country-spec mechanism (`docs/spec-engine.md:39-68`) without + migrating the existing one. +- **Code evidence:** `country_spec.py` already defines a spec-only country + package, hashes every declared resource, rejects undeclared/missing files, + type-validates its recognized resource kinds, and separately compiles the + source/geography plan with a no-fallback posture (`country_spec.py:1-10, + 797-920,923-996`). `CountrySpec.fingerprint` hashes the composition of every + resource (`:756-794`) using a defined sorted-hash algorithm + (`trace.py:96-123`). US and + UK already have packaged manifests (`us/country_package.json:1-26`; + `uk/country_package.json:1-20`), including US source, support, PUF, take-up, and + fiscal resources. Belgium—not US—is explicitly described as the first full + consumer and already declares source, geography, target, gate, and release + resources (`country_spec.py:12-16`; `be/country_package.json:1-11`). The + `schema_version` present in those package manifests is not read or validated + by the current loader (`country_spec.py:820-859`), another version seam the RFC + must absorb. The loader finds installed package resources with + `importlib.resources` (`country_spec.py:813-817`). Root `specs/us/*.yaml` is not + under the wheel package path declared by + `packages/microcosm-build/pyproject.toml:55-63`. There is also an existing root + `specs/us_imputation_lineage.yaml` described as source of truth and consumed by + a conformance test/dashboard (`specs/us_imputation_lineage.yaml:1-4`; + `packages/microcosm-build/tests/test_imputation_lineage_spec.py:1-31`). +- **Rollout failure:** Production can have two resource trees, two loaders, and + two fingerprints answering “what spec built this?” The existing fingerprint + composes hashes of raw resource bytes (`country_spec.py:786-794`; + `trace.py:109-123`), whereas the proposed hash canonicalizes parsed YAML, so + they even disagree on whitespace/key-order-only edits. Checkout tests may pass + while an installed wheel cannot find the new YAML. Old US or Belgian resources + can continue steering execution outside `spec_sha256`. +- **Concrete fix:** Make the RFC an explicit extension/replacement migration for + `CountrySpec`, not a parallel loader. Choose one packaged manifest, one + installed-resource lookup, and one public composition binding. Prefer placing + the resolved bundle/schemas under package data and adding its new file kinds + to `CountrySpec`; alternatively explicitly package root `specs/`, but delete + the duplicate package resources in the same staged migration. Version the + transition from raw-byte `CountrySpec.fingerprint` to canonical + `spec_sha256` with an explicit identity-generation boundary; they cannot be + aliases. Begin enforcing/versioning the previously inert package + `schema_version`, test a clean built wheel, and migrate a real Belgian bundle + or explicitly scope/deprecate that system. Explicitly retire/move + `us_imputation_lineage.yaml` and its emitter rather than leaving a third + surface. + +### MAJOR 11 — The migration map is not total enough to support constant deletion + +- **Claim or gap:** The RFC's migration table claims the relevant authority has + a bundle home and then deletes constants/conformance tests + (`docs/spec-engine.md:262-281`). It omits multiple behavior- and + identity-bearing classes. +- **Code evidence:** At minimum, the audit found: + + | Missing class | Current evidence | Required disposition | + |---|---|---| + | Pipeline/runtime | mass shares, operator and checkpoint-stage order, seed, period, source-reuse doctrine, model sizes, and simulation batch size in `us_runtime/multispine_pool.py:183-251`; checkpoint schemas/materializer ledgers, estimator counts, filenames, pipeline id, rung grammar, and release regex in `tools/build_us_multispine_pool.py:216-293` | Normative order/params in bundle; serialization/materializer versions remain code-owned and identity-bound | + | Late DAG semantics | registry/receipt/transition versions, hard-coded stage/scope/entity/virtual-resource names at `us_runtime/us_late_producer_registry.py:126-169`; tolerated absences and alternatives at `late_producer_dag.py:69-119` | Lossless producer graph or explicitly versioned kernel ABI | + | PUF support/clone/tail | support channel and clone index at `us_runtime/support_provenance.py:31-35`; tail thresholds, quantiles, topcode, filing statuses, AGI proxies, support/no-widen doctrine at `puf_capital_gains_tail.py:76-240`; PUF aggregate seed/RECIDs/bounds at `puf_source_agi.py:21-52,280-421`; silently loaded and hashed SOI bands at `puf_interest_components.py:100-205` and aggregate-record spec at `puf_aggregate_records.py:226-232`; primary QRF target order, absence doctrine, and checkpoint layout at `puf_qrf_chain.py:80-129` | Bundle resources/params for normative choices; role names instead of raw clone integers; explicit asset pins; code-owned envelope versions | + | Resume/operations | checkpoint-root and bank layout at `tools/build_us_multispine_pool.py:487-554,572-587`; exact-identity discovery at `:1169-1244`; logbook predecessor env fallback at `:3965-3977`; hard-coded spool/receipt roots at `:3985-4064`; fit worker env/interpreter/CPU bindings at `stacked_spine.py:4620-4699` | Formal normative/operational/external-state classification and receipts | + | Calibration/selection | public solver defaults and mass/loss/warm-start/L0/L1/L2 surface at `microcosm-calibrate/solve.py:1306-1331`; pruning and backend cutoffs at `:111-128,410-414`; exact-k requires `pi_hi`, seed, and optional grouping at `exact_k.py:424-480`; registry artifact version at `registry.py:42-44` | Fully resolved build params; internal algorithm choices bound by kernel contract/materializer version | + +- **Rollout failure:** Deleting a listed subset either breaks construction or + leaves hidden Python/package/environment authorities. An unchanged bundle can + then change output or resume stale checkpoints, defeating D1 and D3. +- **Concrete fix:** Generate an audited inventory from the current full base + identity, authority receipts, packaged resource loads, runtime signatures, + environment reads, and stage manifests. Give every item exactly one owner: + bundle-normative, versioned kernel ABI/implementation, artifact schema/ + materializer, operational receipt, or external mutable chain state. Require a + reviewed static allowlist/denylist test until deletion, and require the + resolved legacy bundle to reproduce the current identity/authority payloads + byte-for-byte. Delete a constant only after its replacement or deliberate + code-owned classification is tested. + +### MAJOR 12 — `schema_version`, `spec_sha256`, authority versions, and materializer versions have no interaction policy + +- **Claim or gap:** The RFC introduces three apparent configuration/version + concepts but says only that the bundle hash becomes the primary answer and + semantic versions remain (`docs/spec-engine.md:30-33,244-247`). It does not say + which changes invalidate compatibility/resume or what a schema bump means. +- **Code evidence:** Current code needs several independent axes: checkpoint + schema/materializer v1/v7 with an explicit semantic invalidation ledger + (`tools/build_us_multispine_pool.py:216-258`), stacked materializer v11 + (`:284-289`), frame checkpoint schema v3 (`frame_checkpoint.py:44-48`), outer + context schema v2 (`outer_stage_runtime.py:42-48`), stacked authority v10 and + component digests (`us_runtime/stacked_spine.py:1690-1702,2338-2365`), late + registry/receipt/transition versions + (`us_runtime/us_late_producer_registry.py:126-140`), + and target-registry artifact version (`microcosm-calibrate/registry.py:42-44`). + The materializer comment explicitly requires a bump when implementation + changes under the same registry name (`tools/build_us_multispine_pool.py: + 245-250`). The repository already has `builder_code_identity`, specifically + because pins and seeds alone can blend old-code and new-code checkpoints; it + hashes packaged sources and numeric dependency versions + (`code_identity.py:1-10,27-75`). US PUF support and UK builders use it + (`tools/build_us_puf_support_base.py:699-736`; + `tools/build_uk_national_dataset.py:1032-1069`), but the f004 stacked base + identity does not; its git pin is written only to Logbook + (`tools/build_us_multispine_pool.py:3919-3934,3991-4004`). A YAML hash cannot + notice such Python/dependency changes. +- **Rollout failure:** If “one hash” replaces those invalidators, unchanged YAML + can resume output from changed kernel code. If every schema bump is treated as + a semantic authority bump, harmless grammar evolution becomes impossible. If + semantic versions override the hash, config edits can share identity. +- **Concrete fix:** Define four orthogonal, jointly required identity classes: + + 1. `schema_id`/`schema_version` plus `canonicalizer_version` define accepted + syntax and typed resolution; unsupported versions fail before execution. + 2. `spec_sha256` identifies the exact fully resolved normative configuration; + any semantic config/default/order/pin edit changes it. + 3. Each kernel/authority exposes an implementation/contract version (or a + `kernel_set_sha256` over id, implementation version, params schema, and IO + schema), backed by a code/dependency artifact digest rather than a manual + version alone; the bundle pins a supported version/range and the loader + checks it. + 4. Artifact/checkpoint schema and materializer versions remain code-owned and + describe serialization/materialization compatibility. + + Resume requires equality of all four; none masks another. A schema migration + produces a new schema version and normally a new spec identity even when an + equivalence fixture proves behavior unchanged. A config-only edit changes the + spec hash without requiring an authority bump. A semantic kernel edit bumps + its contract/materializer even with unchanged YAML. An explicit tested + translator/semantic-IR hash may permit migration, but it must be a separate + mechanism, never implicit precedence. + +### MAJOR 13 — Keeping model defaults as library fallback reintroduces dual authority + +- **Claim or gap:** The migration map says build model settings move into the + bundle while `microcosm.fit` keeps defaults for library users + (`docs/spec-engine.md:277`). It does not forbid build kernels from omitting + those arguments. +- **Code evidence:** QRF defaults remain live at + `microcosm-fit/qrf.py:83-89,1033-1044`. Production APIs also have their own + defaults, e.g. ACS transfer seed/estimators/max-targets + (`us_runtime/acs_transfer.py:880-892`) and primary QRF predictors/outputs/seed/ + estimators/absence doctrine (`us_runtime/puf_qrf_chain.py:118-130`). Calibration + has an even larger default surface (`microcosm-calibrate/solve.py:1306-1331`). + This is live production behavior, not hypothetical: primary QRF and ACS + transfer instantiate QRF with only `n_estimators` and `seed`, so `zero_atol` + and `max_samples_leaf` come from the library defaults + (`us_runtime/puf_qrf_chain.py:219-225`; `acs_transfer.py:1341-1345`). + The current mirror test pins some code defaults precisely because omitted + kwargs otherwise matter (`test_imputation_lineage_spec.py:87-97`). +- **Rollout failure:** A library release can change a default and alter a + bundle-built artifact without changing `spec_sha256`. That is the same + code/spec drift the authority flip is intended to remove. +- **Concrete fix:** Registered production adapters must accept a fully + materialized, schema-complete config and pass every build-facing parameter + explicitly. Production code may not use `dict.get(...library_default)`, omit a + behavior-bearing kwarg, or infer a build default from the library. Standalone + library calls may retain convenience defaults. Add a test that monkeypatches + library defaults and proves the resolved build plan/output is unchanged, plus + a static/contract test that every registered build invocation supplies its + declared parameter set. Internal algorithm changes remain covered by the + kernel/materializer identity from MAJOR 12. + +### MAJOR 14 — Source identity and deployment/operational bindings are conflated + +- **Claim or gap:** `sources.yaml` purportedly replaces launcher paths and pins + every external input (`docs/spec-engine.md:72-95,266`), but the RFC calls the + locator “documentation” and gives no runtime binding protocol. +- **Code evidence:** The tool currently requires six local paths and six pins + (`tools/build_us_multispine_pool.py:406-478`). Verified manifest entries include + resolved absolute paths (`:341-355`), while logical checkpoint identity wisely + includes only role, actual hash, and size (`:1011-1019`). Checkpoint roots are + operational fallbacks (`:487-554`); logbook chain head can come from an env var + (`:3965-3977`); spool/receipt directories are derived operationally + (`:3985-4064`). Worker environment and interpreter/CPU choices are separately + captured in fit bindings (`us_runtime/stacked_spine.py:4620-4699`). +- **Rollout failure:** Embedding host paths, credentials, checkpoint roots, or + logbook chain state in the bundle makes identical semantic builds hash + differently and can leak restricted locations. Leaving all env/CLI state out + without classification lets behavior-bearing inputs escape identity. Calling + a locator documentation does not make the bundle drive the build. +- **Concrete fix:** Define three surfaces: (1) normative and hashed logical + source ids, roles, content pins, loader ids, and semantic options; (2) + operational and receipted bindings from source id to local path/URI, + checkpoint/output/spool roots, credentials, and demonstrably output-invariant + worker counts; (3) external mutable chain state such as logbook predecessor, + bound by the chain protocol rather than spec hash. The launcher supplies an + exact id-to-location mapping; the loader requires a bijection, verifies the + bundled hash/size, and never hashes the host location. Explicitly classify + any worker/backend option that can change bytes as normative or implementation + identity, not merely operational. + +### MAJOR 15 — The proposed “shared schema” is US-shaped and already fails the UK implementation + +- **Claim or gap:** Principle 5 and rollout step 6 say UK instantiates the same + schema after the US flip (`docs/spec-engine.md:39-41,295`). +- **Code evidence:** The examples assume ASEC/ACS channel names, PUF attachment, + numeric clone roles, US census blocks/PUMA/FIPS/CD119, `takes_up_*`, and a US + publication grammar (`docs/spec-engine.md:97-134,162-198,218-220`). Production + codifies PUF role `asec`/`puf_tax_detail` and clone 1 + (`us_runtime/support_provenance.py:31-35`). The UK geography mechanism is a + two-stage constituency-then-output-area draw with UK-specific OA/LSOA/MSOA/ + LAD/ward/ITL layers (`uk_runtime/geography_ladder.py:1-25,96-121`), and that + module explicitly says the existing shared country schema only models + `clone_assign_uniform` and needs a new `anchor_sample_oa_ladder` method plus + resource wiring (`:66-74`). UK uses person/benunit/household entities + (`uk_runtime/national_frame.py:61`), FRS/SPI support with 10,000 synthetic + households and 50% prior mass (`uk_runtime/spi_support.py:29-42`), and + different exported versus in-memory clone-column shapes + (`uk_runtime/rowwise_dataset.py:56-78`). Those conventions are not PUF clone + attachment. Belgium's already-live country package is a third compatibility + obligation, not mentioned by the RFC (`country_spec.py:12-16`). +- **Rollout failure:** The first UK bundle will require discriminators or a + schema break after US code/constants have already been deleted. US names can + leak into supposedly generic loader/compiler APIs. +- **Concrete fix:** Split a small country-neutral core (source/resource binding, + typed stage DAG, kernel contracts, lineage/identity, artifact profile) from + discriminated country extensions. At minimum use support kinds such as + `puf_attachment`, `synthetic_prior_replacement`, and `none`; geography kinds + such as `single_anchor` and `two_stage_anchor`; arbitrary entity/channel/role + ids; a geography layer graph instead of fixed FIPS keys; and optional country + take-up/publication extensions. Compiling a minimal UK bundle and a + compatibility bundle for existing Belgian semantics must validate in rollout + step 1. “Same schema” should mean the + same versioned core plus declared extensions, not the US file shapes verbatim. + +### MAJOR 16 — Legacy-mode lifecycle and pre-spec artifact handling are unspecified + +- **Claim or gap:** The RFC introduces a temporary `--legacy-constants` mode, + immediately joins the hash, later deletes constants, and says nothing about + in-flight artifacts or how long dual authority stays tested + (`docs/spec-engine.md:252-258,283-295`). +- **Code evidence:** Adding any identity field routes QRF/ACS banks under a new + digest (`tools/build_us_multispine_pool.py:572-587`). Stacked discovery requires + exact identity equality (`:1181-1244`): adding the binding only to the base + identity rejects old mappings inside the existing configured namespace, while + the required fix of also adding it to `_configured_stacked_identity` routes to + a new namespace (`:1150-1178`). Either way, pre-spec checkpoints cannot resume + under the new identity. The tool already has a separate whole-pipeline + `--legacy-two-spine` flag (`:528-531`), creating four undefined flag + combinations. PR CI also cannot perform the restricted-input certification + build (`CLAUDE.md:31-40`). +- **Rollout failure:** Step 2 is operationally a cold-cache cutover even if it is + semantically unchanged. Relabeling an old artifact with a hash it never bound + is false provenance. A constants path not exercised on every relevant change + will rot, while accidental interaction with `--legacy-two-spine` can compare + different pipelines. +- **Concrete fix:** Call the selector + `--config-authority={bundle,constants}`, apply it only to the current stacked + pipeline, and reject or precisely define its interaction with + `--legacy-two-spine`. Parameterize fixture integration tests over both + authorities on every change to bundle, loader, registry, identity, or kernels; + require both to load/hash the same bundle. Run the cold f004 proof in the + restricted certification lane and retain the selector through at least one + certified full release and a stated retention deadline. For old artifacts, + make a hard cut: drain old runs on old code, keep old identity namespaces + read-only, add an `identity_generation`, and cold-build the new namespace. + Never retrofit `spec_sha256`; a legacy reader may inspect old artifacts for + comparison but must not promote them. Readers/logbook schemas should accept + historic generation 0 and require the new binding for release promotion after + the cutoff. + +## MINOR findings + +### MINOR 1 — The RFC and committed geography skeleton disagree about the ASEC fallback + +- **Claim or gap:** The RFC says unidentified ASEC rows draw from + `identified_county_else_state` (`docs/spec-engine.md:121-123`); the committed + skeleton says state **minus** identified counties + (`specs/us/geography.yaml:8-15`). +- **Code evidence:** `identified_county_set.preferred` and `.fallback` are prose + tokens, not source ids (`specs/us/geography.yaml:16-19`), despite the source + pin rule at `docs/spec-engine.md:72-95`; neither existing geography kernel + consumes them. +- **Rollout failure:** Implementers can choose different universes, and a + fallback derived from a sampled pooled file can vary by rung. +- **Concrete fix:** Make the complement ruling the sole text, reference a pinned + official source id, and disallow the fallback in production. If a fallback is + unavoidable, define an exact pre-sampling derivation and bind the resolved + county list plus digest into identity/receipt. + +### MINOR 2 — “Unreferenced kernel fails load” has no registry scope + +- **Claim or gap:** Total closure rejects an unreferenced kernel + (`docs/spec-engine.md:22-25`). +- **Code evidence:** The same RFC expects a shared multi-country schema and + country-specific bundles (`:39-41`), while the current late registry already + contains multiple kinds and stages (`us_runtime/us_late_producer_registry.py: + 126-169`). +- **Rollout failure:** If “unreferenced” means every callable in a process-global + registry, the US bundle fails merely because UK, library-only, diagnostic, or + test kernels are installed. +- **Concrete fix:** Require closure over the selected country/profile registry + namespace and instantiated aliases. Unknown referenced ids and duplicate ids + always fail; unused library implementations do not, or are explicitly marked + `library_only`. + +### MINOR 3 — “Every logbook row has spec_sha256” is impossible for spec-load failures + +- **Claim or gap:** D3 says every logbook row carries the validated bundle hash + (`docs/spec-engine.md:30-33,244-247`) while malformed bundles must refuse at + load. +- **Code evidence:** The tool creates a terminal-attempt state before input or + configuration validation and records failure rows using its current identity + digest (`tools/build_us_multispine_pool.py:3985-4015,4138-4165`). A missing, + duplicate-key, or unparsable bundle has no valid canonical `spec_sha256`. +- **Rollout failure:** Implementers must either omit the row, lie with a partial + hash, or violate the “every row” schema. +- **Concrete fix:** Make `spec_sha256` required only after successful spec + resolution. Failure rows carry `spec_binding_status`, schema/canonicalizer + attempt, and—when bytes were readable—a separate raw manifest/file-set digest + plus the validation error. Never call that raw digest `spec_sha256`. + +## NIT findings + +### NIT 1 — The advertised “real excerpts” and sign-off artifacts still contain placeholders + +- **Claim or gap:** The RFC calls each example a real excerpt + (`docs/spec-engine.md:70`) and asks for schema sign-off in rollout step 1 + (`:283-286`). +- **Code evidence:** The excerpts contain literal `"…"`, ``, ``, and an abbreviated release grammar + (`docs/spec-engine.md:88-92,150-151,171,181-182,209-210,218-219`); the committed + take-up skeleton promises future rows rather than declaring them + (`specs/us/take_up.yaml:1-12`). +- **Rollout failure:** Reviewers cannot distinguish schema syntax from prose or + validate whether placeholders are legal values. +- **Concrete fix:** Label snippets pseudocode, or replace every placeholder with + valid schema-conforming data and put explanatory omissions outside the YAML. + +## Deletion checklist + +Do not delete legacy constants or the authority selector until all of the +following are true: + +1. One packaged loader and one canonical composition binding cover all current + `CountrySpec`, root lineage, stage-manifest, contract, and resource authority. +2. Complete schemas, canonicalizer golden vectors, invalid fixtures, a full US + resolved bundle, and compiling a minimal UK bundle plus a compatibility + bundle for existing Belgian semantics pass from a clean wheel. +3. Constants and bundle compile to byte-identical current plan, schedule, + ownership, authority, and identity payloads; both modes load the same bundle. +4. Cold fixture dual-mode tests run on every relevant PR, and a restricted cold + f004 certification compares all stage content plus canonical gates. At least + one full release has been certified before expiry. +5. Production registered kernels use the generic projection/diff executor and + no production path consumes library defaults or undeclared package assets, + env values, stage strings, source pins, or params. +6. Checkpoint/logbook/manifest/dashboard surfaces carry the namespaced spec + binding plus kernel/authority and artifact/materializer identities. +7. Historic identity-generation retention, reader compatibility, hard-cut + rebuild, and cleanup dates are documented; no old artifact is relabeled. +8. Only then remove constants, `--config-authority=constants`, and interim + code-equals-bundle tests. Retain schema/canonicalizer golden tests, generic + kernel-discipline tests, closure tests, and content-equivalence fixtures. + +## Verdict on D1–D5 and rollout order + +Using the RFC's five operative decisions as D1 authority flip, D2 registered +kernels, D3 spec identity binding, D4 frozen-behavior gate, and D5 derived seed +streams: + +| Decision | Verdict | Required amendment | +|---|---|---| +| D1 — bundle is build authority | **Needs amendment; direction stands, wording does not.** | Extend/replace the existing packaged `CountrySpec`; make the fully resolved bundle the only normative build config; use cell-scope lineage; separate runtime source bindings; keep code-owned kernel and artifact contracts explicit. | +| D2 — registered kernels are the escape hatch | **Needs amendment.** | Add the generic projected-input/patch-output/diff executor and a lossless producer graph. A name lookup plus current specialized guards is insufficient. | +| D3 — `spec_sha256` joins identity/logbook | **Needs amendment.** | Both modes load the same bundle before branching; use a namespaced binding; retain kernel/authority/materializer identity; define invalid-spec log rows and the pre-spec artifact cutover. | +| D4 — identical identities and gates prove the flip | **Does not stand as written.** | Use two cold isolated executions, forbid resume, and compare deterministic stage content/exhaustive frame digests plus normalized gates. | +| D5 — derive every stream from one root seed in the flip | **Does not stand as written.** | Equivalence uses explicit `legacy-v1` streams. A fully specified `derived-v2` is a later intentional behavior change with fresh identity/caches and statistical gates. | + +Therefore **none of D1–D5 stands completely as written**, although D1's +high-level direction remains worth pursuing. + +The current rollout order is not sound. A coherent order is: + +1. Reconcile the RFC with `CountrySpec`; land the complete schemas, typed IR, + canonicalizer and golden vectors, full legacy-equivalent US bundle, minimal + real UK bundle, Belgian compatibility fixture, version-interaction policy, + operational binding model, and old-artifact policy. +2. Make both current-stacked authority paths load/hash that same bundle. Add the + namespaced spec binding everywhere in one intentional cold-cache identity + generation; retain explicit legacy seeds and current geography/mass/clone + semantics. +3. Build the generic kernel executor and lossless producer graph behind the + legacy execution path. Require byte-identical compiled plan/authority/ + schedule payloads before comparing data. +4. Run per-PR cold fixture equivalence and restricted cold f004 content + certification, flipping one stage at a time. Geography at this point must be + the exact legacy behavior, not #696's block-first change. +5. After at least one certified full release and the deletion checklist, remove + Python configuration constants and the temporary authority selector. +6. Land intentional behavior edits separately: block-first/ASEC complement, + eligibility and calibration changes, then `derived-v2` seed streams. Each is + a bundle diff with the appropriate authority/materializer bump, cold caches, + and f025/OOS or statistical gates—not a legacy equivalence claim. +7. Expand country extensions only after the real UK and Belgian conformance + bundles have already proved the shared core. diff --git a/docs/spec-engine.md b/docs/spec-engine.md new file mode 100644 index 00000000..504497ca --- /dev/null +++ b/docs/spec-engine.md @@ -0,0 +1,649 @@ +# The spec engine: one declared bundle drives the build + +Status: **v3 — APPROVED by Max 2026-08-16** ("sure i approve") after two +rounds of cross-family review. Sign-off covers the schema shape, the +P→F0→F3 phasing, and the schema set in `specs/schema/`. D6 RULED same day: +**`microcosm-us-2024-*`** is the spec-engine-era release line (Max, +2026-08-16); `populace-us-*` rows stay valid-historical in the chain; the +HF dataset destination is unchanged and outside this ruling. +Nothing here is wired; excerpts below are **non-normative pseudocode until +the committed schemas and the complete US bundle exist** (both reviews +correctly rejected "schema-conforming" as a label for drafts). + +Review provenance: + +- Round 1: sol (code-grounded, 16 MAJOR) + GPT-5.6 Pro (design, 15 + findings) on v1 → v2 (d865ba40). +- Round 2 on v2: **both returned request-changes.** + - sol (`_698-SOL-REVIEW-R2.md`): 11 of 20 r1 findings RESOLVED, 6 + partial, 3 renamed-not-resolved; 9 new MAJORs (N1–N9). Every + spot-checked claim held again (5 batches/37 targets; 13 contract + programs with no wic/social_security; YAML banned by the package + test). + - Pro (`_698-PRO-REVIEW-R2.md`): 3 of 15 adopted-correctly, 11 + mutated, 1 LOST (the versioning triad); fusion adjudication + + 11 fresh MAJORs. + - Convergent blockers (both, independently): F0 as written contains + F1's compiler and manual dual-editing is a dual-authority trap; the + node key over-keys execution profile and under-keys run values; + catalogs/vintages recreate authored drift pairs; the take-up enum + conflates dimensions; the producer graph is not yet lossless; the + equivalence vector stops too early; the deletion checklist is not + machine-decidable. +- v2 factual errors corrected here: `puf_tax_itemization` is **five** + batches / **37** targets (19 bounded groups / 70 targets across the + registry), not 4/32; `wic` and `social_security` are not contract + programs; `takes_up_eitc`/`takes_up_dc_ptc` break the suffix naming + rule (column mapping must come from an engine ABI projection). + +## The ruling this implements + +Max, 2026-08-15: *"It should be immediately obvious which predictors we're +using for each variable … this part should just be a yaml file. Also the +attributes of the ML model. … Let's do it right before making atomic +fixes. Scope out the right schema for everything first."* + +Today the spec (#695/#697 line) **mirrors** code and CI enforces the +mirror. The spec engine **flips the authority**: the build compiles the +spec bundle into an executable plan, constructs its plan objects from it, +and the Python constants are deleted. Custom logic survives as **named +kernels** — code keeps the *how*, the spec owns the *what, from what, +with what*. + +## What exists today (five authority surfaces, one disposition each) + +1. **`CountrySpec`** (`country_spec.py`): packaged spec-only country + system; US/UK/BE packages live; Belgium is its first full consumer. + Its manifest is a bare filename list, every resource parses as a JSON + mapping, typed behavior keys off hard-coded filenames, and the + package test bans YAML (`SPEC_SUFFIXES = {".json", ".jsonld"}`). + **Disposition: replaced-in-place through an explicit seam** — see + "One spec system." Calling this an "extension" without the seam was + v2's mistake; the migration is real work and is named as such. +2. **`specs/us_imputation_lineage.yaml`** (#695): mirror-mode lineage + spec. **Disposition: retired at the flip**, with its named consumers + migrated, not orphaned: `test_imputation_lineage_spec.py`, + `tools/emit_lineage_dashboard.py`, the stale packaging comment, and + the external dashboard handoff. +3. **The take-up contract** (`us/take_up_contract.json` + + `source_stages.json` rows): a *reviewed snapshot* of engine facts + whose currentness assertion is a deliberate tripwire. + **Disposition: absorbed — but the tripwire survives** as a generated, + committed **engine ABI lock** (below); derive-and-assert against the + live engine alone would be circular on an engine bump. +4. **Python constants** across `us_runtime/` and the stacked tool. + **Disposition: bundle-normative per the migration map; deletion gated + by the machine-decidable checklist.** +5. **`builder_code_identity`** (`code_identity.py`): used by UK/PUF + builders, absent from f004 stacked identity. **Disposition: joins the + code identity class.** + +## Principles + +1. **Total, or failing.** Closure over the selected country/profile + registry namespace; unknown referenced ids and duplicates always + fail; library-only implementations don't. +2. **Declarations carry mechanisms, not just names.** Every stochastic + write declares draw universe, conditioning, parameter source, and + seed stream — and the runtime is **forced to obey** (RNG broker, + below), or the declaration is documentation. +3. **Provenance is global; reuse is per-node.** Global identities + describe every run in receipts and logbook rows; node keys decide + cache/checkpoint reuse. Neither substitutes for the other. +4. **Kernels are the only escape hatch — behind a generic executor and + brokers.** No second escape hatch anywhere else in the schema (the + v2 take-up `dedicated_stage` value died for this reason). +5. **Country files instantiate a shared core plus declared extensions.** + UK and Belgian semantics must be *expressible*; Belgium must + **build** (smoke run, not compile-only) before the identity cutover. +6. **The dashboard reads the same bundle the build reads.** +7. **Derive, don't declare — in one direction only.** A field derivable + from another authority is compiled and asserted. Where a *reviewed + snapshot* of an external authority is the point (engine facts), the + snapshot is a **generated lock**, regenerated deliberately, never + silently re-derived at build time. +8. **Single-authored always.** At no phase do humans hand-edit two + representations of the same fact. The fast path is generation, not + parallel editing. + +## One spec system: the CountrySpec seam (explicit) + +- `country_package.json` becomes the **single typed resource manifest**: + `{path, kind, schema_id}` rows replace the bare filename list. The + bundle introduces **no second file inventory** — v2's + `bundle.yaml: files` is dead; the root `bundle.yaml` shrinks to + bundle-level settings (country, seed protocol, generation). +- The loader returns one `ResolvedCountrySpec` carrying (a) + migration-era **compatibility projections** (today's `sources`, + `gates`, take-up contract views — so `load_take_up_contract()` and + UK `load_country_spec("uk").gates` keep working until their consumers + migrate) and (b) the compiled spec-engine IR. +- The spec-only package test is updated **intentionally** to admit YAML + + kernel ids as declared kinds. +- **Generation semantics** (Pro r2): generation-0 artifacts keep the + historical raw `fingerprint`. In generation 1 the raw file-set + fingerprint is a **transport/package-integrity receipt only**; + `spec_sha256` is the semantic authority receipt; node reuse uses + compiled node slices, never either global hash. A compatibility + record may map `{legacy_fingerprint, canonical_spec_sha256}`; old + artifacts are never retro-labeled. +- Named consumer migrations: the gate battery's `spec_fingerprint` + (today composed only from `gates.json` while its contract doc claims + the whole package) gets one owner and one definition; every + fingerprint/spec-hash consumer is assigned a generation. +- Schema files, migration translators, and the composition manifest all + carry **immutable content digests**. + +## Canonicalization, schema migration, and locks + +The v2 canonicalization contract stands (YAML 1.2 restrictions, +closed-world schemas, default injection into a typed `ResolvedSpec`, +domain-separated envelope, ordered-arrays-stay-ordered, golden vectors, +invalid fixtures). Round 2 adds the missing pieces: + +- **`schema_version` selects an immutable migration chain**: migration + ids + implementation digests are recorded in the grammar receipt. A + semantics-preserving migration changes the audit receipt, never node + reuse, because the compiled node slice is unchanged. +- The **compiler itself has an identity**: `compiler_ir_abi` + digest. + Two compiler implementations resolving the same declared schema + version differently is an identity difference, not a silent fork. +- Emitted, never authored: `bundle.lock.json` (file hashes + grammar + receipt), `plan.lock.json` (typed IR: stage DAG, resolved node + params, kernel pins, node keys), `engine_abi.lock.json` (below). + Generated locks are reproducible from their authorities and rejected + if hand-edited. + +## Identity: the triad restored, provenance split from reuse + +Round 2's sharpest convergent finding: v2 bound everything into +everything. v3 separates three questions. + +**1. What is this run? — `run_provenance_identity`** (in every receipt, +manifest, terminal attempt, and logbook row; never a reuse gate): + +```text +run_provenance_identity: + identity_generation: 1 # absent/0 = historic, readable, never + # promotable; 1 = binding required; + # unknown = refuse + source_grammar_receipt # schema_version + canonicalizer + + # migration chain ids/digests + spec_binding: # {country, schema_id, schema_version, + # canonicalizer_version, spec_sha256} + authority_versions: # semantic contract version per named + # authority (stacked authority v10 is + # the live example) — field, bump + # rules, and precedence below + code_inventory_digest # builder_code_identity + kernel set + artifact_protocol_inventory # materializer/serialization versions + run_request # rung, seeds, k, release label + execution_receipt # resolved backend/profile, workers +``` + +**The triad and its precedence** (Pro r1's finding, dropped by v2, +restored): `schema_version` (grammar era; unsupported → refuse before +execution) → `spec_sha256` (exact resolved configuration; any semantic +config edit changes it) → `authority_version` (the *semantic contract +era* of a named authority; bumped when the contract's meaning changes +even under unchanged YAML shape; recorded per authority; a bump +invalidates dependent caches by entering the affected node slices). +Code and serialization digests pin implementations underneath. A +semantic behavior change may never ride a materializer bump unless +representation also changed. + +**2. What may be reused? — `node_reuse_key`** (per executable node, once +the compiled plan drives execution): + +```text +node_reuse_key = H( + identity_generation, + compiler_ir_abi_and_digest, + resolved_transitive_node_slice, # this node's plan + upstream + behavior_relevant_run_inputs, # rung, k where consumed, and + # ACTUAL seed material for this + # node's streams + transitive_input_content_hashes, + per_node_implementation_and_dependency_digest, + rng_protocol_and_seed_material, + input_and_output_artifact_contracts, + per_artifact_materializer_abi, + output_sensitive_backend_abi # ONLY backends proven to +) # affect bytes; nothing else +``` + +Execution profile is **not** in the key. Proven byte-invariant settings +live only in the attempt receipt; settings that can change bits are a +resolved backend/numeric ABI in the *affected* nodes; suspected-but- +unproven settings may serve as a temporary cache-compatibility fence and +are never called output-invariant. `device: auto` is never +output-invariant by declaration — the launcher receipts the resolved +backend/dtype/library/deterministic-mode, and CPU/GPU share a key only +after a per-kernel byte-equality conformance test. Release labels never +invalidate computational nodes. + +**3. Interim reality.** Until `plan.lock` drives execution, the existing +whole-run identity machinery keeps gating resume exactly as today — +conservative over-invalidation, which is safe. `identity_generation` + +`spec_binding` enter `_configured_stacked_identity` and +`_stacked_checkpoint_base_identity` (stage and bank identities inherit), +`run_config`, manifests, terminal attempts, and logbook rows — the +concrete field homes sol specified. Discovery passes both through when +reconstructing expected identity. + +## Configuration surfaces — five compiled objects, zero leaks + +The five surfaces stand (normative / run request / execution profile / +operational bindings / external chain state). Round 2 caught the +examples violating them, so the rule is now mechanical: **the compiler +emits five physically separate typed objects; every schema field +declares its surface; the canonicalizer hashes only the normative +projection.** A value may be a normative default *and* a run-request +override only with declared precedence and a resolved receipt. + +Example-level corrections baked into the excerpts below: `k` is a +run-request knob (the bundle may declare a default with precedence); +`device` is profile/backend, out of calibration.yaml; the logbook store +and HF destination are operational bindings, out of publication.yaml's +normative block; simulation batch size is classified by an invariance +proof, not by assertion. + +Failed spec loads carry `spec_binding_status` + attempted grammar + +(when readable) a raw file-set digest never called `spec_sha256`. + +## Seeds: an enforced protocol, not a map + +`seed_protocol: legacy-v1` ships with the flip and changes nothing; +`derived-v2` (stateless counter streams, root seed in the run request) +is an F3 behavior edit with statistical gates. Round 2's demand — from +both reviewers independently — is enforcement: + +- **The three-way boundary.** *Spec-normative:* stable draw-site ids, + site→stream mapping, literal base seeds, RNG family/version, spawn + count and index assignment, consumed target/program order, + reset/reuse boundaries, entity/clone ordering where draws depend on + it, digest width/endianness/encoding, and the immutable protocol + implementation id + digest. *Kernel-contract-normative:* internal + consumption patterns (QRF's `SeedSequence(seed).spawn(2)` and shared + fit-RNG advance in target order), pinned by the kernel/code digest. + *Receipt-descriptive:* realized seeds, saved states, rationale. +- **The RNG broker.** Production code obtains RNGs only through a + versioned broker; direct `np.random`/`SeedSequence`/`random`/framework + RNG construction outside it fails static and runtime checks; every + stochastic kernel receipts the stream ids it consumed. Without the + broker the map is documentation. +- **The exhaustive draw-site ledger.** v2's 578/0/42+ACS/QRF summary was + not an inventory. The ledger additionally covers (sol N9): SSI + training/model seeds, SIPP vehicle/asset stable-string hash + algorithms, the ACS-rent archived hash, the tips training seed, and + the SCF composite `SeedSequence` — plus anything a repo-wide audit + finds. Golden ACS label/seed vectors and a multi-regime QRF chain + fixture pin the algorithms. + +## Kernels: executor + brokers + orthogonal capabilities + +The generic executor stands (immutable projection in, patch out, full +structural diff, rejection outside declared entity/column/row scopes). +Round 2 additions: + +- **Ambient access is brokered.** The executor cannot stop a Python + callable from reading globals/env/files/network/clock/private RNG — + so pure and seeded kernels run with ambient access prohibited or + instrumented, and file/env/clock/RNG access exists only through + explicit brokers. +- **Capabilities become orthogonal fields**: + +```text +determinism: deterministic | seeded | nondeterministic +numeric_reproducibility: bitwise | tolerance_bound | unspecified +effects: none | declared_source_read | declared_sink_write +structural_delta: none | filter | expand | join | relink | reorder | reweight +retry_safety: idempotent | attempt_scoped | nonretryable +``` + + `structural_effect` is replaced by the specific delta with pre/post + conditions — otherwise it is an unbounded second escape hatch. +- **Row scopes are a closed predicate algebra.** Labels like `acs_rows` + compile to canonical predicates whose overlap, exhaustiveness, and + equality the compiler can decide; segmented lineage is unenforceable + otherwise. + +## The producer graph: lossless means graph-semantic + +v2's sketch was not lossless (sol verified: it dropped +`producing_stage`, receipt ids, OR-of-AND alternatives, kind-specific +virtual-resource payloads, and the **18-row target × origin × clone-role +ownership matrix**). The v3 contract: + +- Field-lossless: every `ProducerInput`/`ProducerOutput` field — + entity/column/`required_scope`/`producing_stage`, ordered + tolerated-absence receipt ids, alternatives as ordered OR-of-AND lists + of `{entity, column, value_kind}` with declared precedence and + disjoint/exhaustive predicates; outputs with `coverage_scope`; typed + virtual resources (manifests, resolved weights, execution configs, + transition/producer receipts, target banks) with each kind's semantic + payload and digest rules; transfer groups; the full conditional + ownership matrix with finalization and every non-owner action; the + execution-receipt and transition-authority contracts. +- Graph-semantic: explicit read-after-write dependency edges; + deterministic total order for incomparable nodes plus the rule that + incomparable nodes commute or occupy disjoint write scopes; + temporary/validation-only outputs; entity-key and cardinality + effects; typed link/membership/order/weight/mass-history mutations; + ownership unique per **cell segment**; retry/idempotence behavior. +- Acceptance: golden byte-identical compile-back of today's + schedule/ownership payloads before the flip. + +**The frozen split ledger, corrected**: at the flip, the entire current +ordered split structure — **19 bounded groups / 70 targets overall; +five batches / 37 targets for `puf_tax_itemization`** — is declared +explicitly with a reason per split, compiled back against both greedy +splitter implementations and every literal batch-name consumer +(ownership constants included), after which the two splitters and +`DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT = 8` die at F2. The F3 +re-unification is scoped honestly: it changes the donor complete-case +population (intersection across 37 targets), family-label-derived +seeds, shared-RNG consumption order, cross-boundary chained predictors, +overlap-owner references, and bank identities — fresh node/bank +identities, cold caches, OOS/statistical gates. + +## Lineage, closure, catalogs, vintages + +- Whole-column closure derives from declared outputs; authored remnants + are the inventory fixture and time-limited waivers. **The catalog + splits three ways** (both reviewers, independently): a *normative + column contract* (stable id, entity, dtype, unit, period, nullability, + domain, public-stability status); a *derived lineage report* + (producer, owner, origin class, scopes, stages — compiler-emitted, + never authored); and a *non-normative documentation overlay* + (descriptions, citations) hashed into a separate documentation + digest, outside `spec_sha256`. +- Cell-scope segments over `(entity, column, row_scope, stage, + write_policy)` stand, now backed by the row-scope predicate algebra. +- **Vintages become typed references, not repeated literals**: + `tax_period_ref`, `survey_period_ref`, `target_period_ref`, + `geography_vintage_ref`, `policy_engine_surface_ref`, + `release_series_ref` — each living on the pinned source/lock record + it describes, referenced by id everywhere else, with compiler-checked + compatibility relationships. The engine pin appears in exactly one + runtime/source-lock record; a CI check rejects duplicate literal + authorities per normalized key. + +## Take-up: orthogonal ownership × typed pipeline steps, 13 real programs + +The v2 eight-value treatment enum conflated ownership, mechanism, +invocation grouping, policy interaction, and an escape hatch. v3: + +```yaml +# take_up schema (pseudocode until schemas land) +ownership: measured | transferred | modeled | engine +pipeline: # ordered, typed step kinds — closed enum of + - kind: probability_seed | count_calibration | delivery_gate | + assignment | measured_map # STEP kinds, not program kinds + kernel: kernel:… + …step-typed params… +dependence: {group: …} # batching/correlation grouping — an execution + # property, never a treatment +final_owner_stage: … +``` + +- **All 13 contract programs get committed rows** (no ellipsis): SNAP + (national prior → state anchored count calibration, eligible-only + domain, final owner); TANF (seeded rate); EITC (seeded rate, + rate-by-approximated-qualifying-children); Medicaid (anchored count + calibration); CHIP / Basic Health Program / DC PTC / Early Head Start + (engine default with named debt); Medicare (measured ASEC map + + support-clone propagation); SSI (target-derived age-band probability + seed + delivered-recipient gate — it never count-matches flags); + Head Start (imputed/transferred: SIPP-trained QRF); housing + assistance (mixed row-scope: measured on ASEC rows, imputed on PUF + support — segments, not one whole-column treatment); ACA (typed + dedicated pipeline steps for Marketplace assignment + calibration). +- **The engine ABI lock**: from the single engine pin, the compiler + generates `engine_abi.lock.json` — `{program → variable, entity, + value_type, default, engine_class, consumers}` — committed and + reviewed. CI compares a fresh derivation to the lock **before** bundle + compilation; engine bumps fail closed until the lock is regenerated + and every bundle-owned treatment re-reviewed. The engine owns its + facts; the bundle owns treatments and pipelines; neither derives from + the other at build time. +- Column/entity mapping comes from the ABI projection with a + **total-and-injective assertion** (the suffix naming rule died on + `takes_up_eitc`/`takes_up_dc_ptc`). + +## Eligibility guards: target-specific concepts + +v2's family-level `required_concepts: [eligibility]` with a generic tag +does **not** make CHAMPVA-class defects impossible (drop `veteran_va`, +keep `own_coverage`, still passes — Pro r2). v3: a concept registry maps +named concepts to predictor columns, and requirements are per-target and +concept-specific: + +```yaml +concepts: + veteran_status: [is_veteran, receives_va_payments] + military_coverage_context: [acs_hins_va] + disability_status: [has_hearing_difficulty, has_vision_difficulty] +families: + - id: gap_fill/asec_survey_to_acs/person/benefit_participation + targets: + - name: has_champva_health_coverage_at_interview + requires_concepts: [veteran_status, military_coverage_context] + - name: has_tricare_health_coverage_at_interview + requires_concepts: [veteran_status] +``` + +A target whose resolved predictor set fails to cover its concept set +fails the **load**. + +## The equivalence gate: four builds, plan-derived vector + +- **Four cold isolated builds**: constants A, constants B, bundle C, + bundle D — require A=B and C=D (within-mode determinism) and A=C + (cross-authority equivalence). Two runs cannot distinguish authority + difference from nondeterminism. +- **The comparison set derives from `plan.lock.json`**, never a + hand-named file list: every sealed stage artifact, trained-model bank + used for resume, the final published logical frame/schema/period, + normalized final manifest + diagnostics + gates, mass history, and + behavior-bearing receipts. A missing output fails certification. + Downstream calibration/selection/release nodes join the vector when + they are in flip scope. +- **Resume-forbidden is a concrete predicate** over existing receipts: + null `deepest_resumed_stage`, primary QRF `resume_status == + initialized`, and no ACS target bank entry with + `load_status: resumed` / `source: checkpoint` — plus + `--resume-policy=forbid` (refuse pre-existing manifests before + loading) and one typed `resume_audit` with per-stage/per-target + counts, all required zero. +- **Adversarial fixtures**, because a generic fixture never exercises + the motivating failures: CHAMPVA-scale donor scarcity, empty and + saturated take-up domains, county complements, crosswalk boundaries, + overlapping producer fallbacks, zero/negative calibration targets, + infeasible exact-k, clone tails, mixed-ownership columns. +- Joining the binding is an explicit `identity_generation` bump — + operationally a cold-cache cutover; pre-spec artifacts stay + generation 0, readable, never promotable, never retro-labeled. + +## Gates and calibration: executable, not named + +Every gate specifies: exact metric formula, input artifact + stage, +population + denominator, slices, reference release/data digest, +minimum support / effective sample size, absolute + relative +thresholds, uncertainty or multi-seed rule, missing-slice treatment, +fail/warn/report-only status, and typed failure-reason mapping. +`report_all_never_widen` names a *policy over that schema*, not a +metric. The calibration spec exposes the full mathematical contract: +target scaling, row-weight construction, cap behavior, zero/negative +target treatment, objective aggregation, mass constraints, +initialization/warm-start, optimizer + schedule + dtype, stopping +conditions, infeasibility policy, target-priority policy, attainment +verdict. Exact-k states post-selection weight semantics explicitly: +selected records carry their calibrated weights unchanged; non-selected +records are absent; no re-normalization unless declared. Gates evaluate +the **final selected artifact**, not only the pool. + +## Publication: attempt events, atomic promotion, two graphs + +- **Attempts are append-only events ending in one immutable terminal + seal** (`landed | failed | expired`); a running attempt is an event + stream, never a "sealed" status. +- Two-phase publication defines: temporary artifact namespace → atomic + manifest seal → output-content verification → idempotency key → + promotion transaction → recovery for seal-ok/append-fail and + append-ok/alias-fail → orphan and expiry reconciliation. +- **The strict-linear logbook chain is the tamper-evident audit + sequence; it is not the release topology.** A separate release + relationship graph carries `derived_from`, `supersedes`, `revokes`; + serial appends to the audit chain imply nothing about release + parentage. +- `latest` promotion stays a human gate; eval artifacts never promote + on a red battery. +- **D6 (release line rename)**: still open — the release line is marked + **provisional/non-normative and excluded from `spec_sha256`** until + Max rules, so golden bundles freeze without baking an unratified + name. HF destination unchanged without an explicit ruling. + +## File excerpts (all pseudocode until schemas + the full US bundle land) + +### bundle.yaml — bundle-level settings only (no file inventory) + +```yaml +country: us +identity_generation: 1 +seed_protocol: legacy-v1 +# The resource inventory lives in country_package.json {path,kind,schema_id}. +# Emitted: bundle.lock.json, plan.lock.json, engine_abi.lock.json. +``` + +### sources.yaml / spine.yaml + +As in v2 (asec mass anchor; support roles with `puf_tax_detail`, +`clone_index: 1`), with paths/URIs strictly operational bindings. + +### geography.yaml — block-first + ASEC complement (**phase F3**) + +As in v2 (complement as sole rule, pinned identified-county source, no +sampled fallback) — with the round-2 correction that the held #696 +branch does **not** implement this: it records ASEC county as absent +from the v3 checkpoint and falls back to state-wide draws, and its +loader rejects county fields. F3 here is a **coordinated kernel + ASEC +checkpoint-schema/source + block-artifact + bundle migration** with +complement/leakage tests and refusal on empty complements — the held +branch gets reworked, not merged as-is. + +### imputation.yaml — chains without the 8; concept guards + +As in v2 (declared order, `splits: declared_only`, +`release_after_draw`, keep-together pairs) with the corrected frozen +ledger (five/37; 19/70) declared explicitly at the flip and the +per-target `requires_concepts` blocks from the concept registry. + +### take_up.yaml — see the orthogonal schema above; 13 committed rows. + +### battery.yaml / calibration.yaml / selection.yaml + +Per "Gates and calibration": each gate a full executable record; the +solver surface fully resolved; `k` a run-request knob with a declared +bundle default and precedence; no `device` in normative content. + +### publication.yaml + +Attempt-event model + promotion protocol + release relationship graph; +audit chain settings; release line marked provisional (D6). + +## Rollout — honest phases + +**P (now, before any freeze): the CHAMPVA lane lands first.** The held +predictor-sets branch is a behavior change (widened loaders, 15 carried +columns, new participation target order, alternatives-precedence and +registry-schema bumps), not a mirror edit. It lands under the existing +#695 mirror plus its **own OOS/statistical acceptance gate**, before any +baseline freezes. The #697 inventory fixture is then **regenerated and +re-certified against the post-predictor artifact profile** (the held +fixture is missing 56 of its columns — the v2 lane order was backwards). +The value fix does not wait for the spec engine at all. + +**F0 — compiler front end, single-authored.** The full front end: +parsing/canonicalization/defaults/migration, cross-reference resolution, +typed entities/artifacts/scopes/columns, stage DAG + producer graph +compilation, seed-protocol resolution, complete normalization of every +domain file, the compile-to-legacy-payload adapter covering every +normative field, a usage/coverage report proving no field is ignored, +and round-trip + mutation tests. **The bundle is authored once; the +compiler generates the legacy payload the constants-era executor +consumes** (`config_authority=constants_adapter`). No manual +dual-editing, ever. `spec_sha256` is labeled a *mirror-attested +configuration identity* until F1. The CountrySpec seam lands here; +UK + BE bundles compile; **a minimal Belgian smoke build runs before +the identity-generation cutover**. No executor yet. + +**F1 — drive.** Generic executor + brokers; producer-graph compile-back +byte-identical; bundle mode constructs the authorities; per-PR cold +dual-mode fixtures; the four-build restricted f004 certification, +flipping stage by stage; geography = exact legacy behavior. Derived +closure/segments/dashboard retarget to compiler outputs here (not the +held authored-class tests). + +**F2 — delete, machine-decidably.** After ≥1 certified full release on +bundle mode, deletion requires generated inventories + zero-reference +tombstone gates: no `is`-guards, constant imports, or alternate +dispatch paths; no production reference to `CANONICAL_US_LATE_*`, +schedule/ownership receipt constructors, either greedy splitter, or the +max-width constant; no nonhistorical reference to the retired lineage +file/emitter/dashboard schema (external dashboard verified against the +compiled catalog); no direct take-up-contract/source-stage authority +outside the generation-0 reader; every reachable stochastic/hash-draw +callsite consuming a broker stream token; no duplicate engine / period / +geography-vintage / catalog-owner literals; one typed resource manifest +and no root drafting copies; a **node-invalidation matrix** proving an +unrelated bundle edit reuses unaffected nodes and invalidates +descendants of the changed node; schema-migration and old-bundle reader +fixtures; four-build within-mode + cross-mode determinism; fault +injection around checkpoint write / manifest write / seal / promotion / +logbook append; cross-profile byte-equivalence for every +claimed-invariant profile; RNG-broker and ambient-read enforcement +tests; a real Belgian build and a UK walking-skeleton **execution**; +and a **dated** authority-selector retention deadline. Generated locks +stay reproducible-or-rejected. + +**F3 — intentional-change train.** Each lands as a bundle diff with +authority-version bumps, fresh node/bank identities, cold caches, and +statistical/OOS gates: block-first + ASEC complement (as the +coordinated migration above), the 37-target chain re-unification (full +scope), `derived-v2` seed streams, then the remaining calibration +stages. + +## Migration map + +The v2 table stands with these round-2 amendments: the late-DAG row +routes through the **lossless producer graph** as specified above; the +take-up row absorbs the contract **via the engine ABI lock**; catalogs +and vintages enter as split contract/derived/documentation and typed +references respectively; the seed row expands to the exhaustive +draw-site ledger; `us_imputation_lineage.yaml`'s row names its three +consumers; and the CountrySpec row is labeled replacement-through-seam. + +## Decisions + +| # | Decision | Round-2 status | v3 resolution | +|---|---|---|---| +| D1 | Bundle is build authority | both: direction stands | Via the explicit CountrySpec seam; single-authored from F0; constants become a generated adapter, then delete at F2 | +| D2 | Kernels are the escape hatch | both: amend | Executor + brokers + orthogonal capabilities + predicate-algebra scopes; no schema-level escape hatches anywhere | +| D3 | spec_sha256 joins identity | both: amend | As `run_provenance_identity` (with the restored triad + `identity_generation` on concrete fields); provenance never gates node reuse | +| D4 | Frozen-behavior gate | both: amend | Four cold builds; plan-derived vector incl. publication + banks; concrete resume predicate; adversarial fixtures | +| D5 | Seed streams | both: split stands | `legacy-v1` enforced by the RNG broker + draw-site ledger; `derived-v2` at F3 | +| D6 | Release line rename | Pro: don't freeze it unresolved | **RULED (Max, 2026-08-16): `microcosm-us-2024-*`** at the flip; the line becomes normative in publication.yaml at F0 (release regex + rung grammar; readers accept both prefixes; populace-* rows valid-historical); HF destination unchanged, separate explicit ruling | + +## Review provenance + +- Round 1: `_698-SOL-REVIEW.md`, Pro conversation (receipt in + `_buildo-runtime/out/stacked-full-LAUNCH.md`). +- Round 2: `_698-SOL-REVIEW-R2.md` (verdict: request changes; 12-item + v3 list — all folded), `_698-PRO-REVIEW-R2.md` (verdict: request + changes; 11-item ranked v3 list — all folded; its "triad LOST" + finding drove the identity restoration). +- Where the reviews pulled differently (four classes vs triad; + whole-run vs node granularity), v3 adopts the composition stated in + "Identity": triad + classes as the provenance vocabulary, node keys + as the reuse mechanism, whole-run binding as the safe interim. diff --git a/specs/schema/README.md b/specs/schema/README.md new file mode 100644 index 00000000..87b52466 --- /dev/null +++ b/specs/schema/README.md @@ -0,0 +1,23 @@ +# Bundle schemas (draft, v3-shaped) + +Closed-world JSON Schemas (draft 2020-12, `additionalProperties: false`) +for every bundle file kind, per docs/spec-engine.md v3. Pulled forward +from F0 so sign-off is on real schemas, not prose. + +- `defs.schema.json` — shared: refs (kernel/source/stream/vintage), the + row-scope predicate algebra, the executable gate record, surfaces. +- `resource_manifest.schema.json` — the typed rows country_package.json + adopts (`{path, kind, schema_id}`); the SINGLE file inventory. +- One schema per authored kind: bundle, sources, spine, geography, + imputation (blocks/models/chaining/concepts/families/producer_graph), + take_up (ownership × typed steps), battery (gate records), calibration + (full math contract), selection (exact-k + post-selection weights), + publication (attempt events, promotion, audit≠release graphs), + vintages (typed refs; engine pin once), catalogs (contract + docs; + lineage NEVER authored). +- `locks.schema.json` — EMITTED artifacts: bundle.lock, plan.lock + (node keys), engine_abi.lock. Reproducible-or-rejected. + +Draft status: these bind at F0 through the CountrySpec seam; until then +they are the review surface. Skeletons in specs/us/ validate against +them (CI wiring lands with the loader). diff --git a/specs/schema/battery.schema.json b/specs/schema/battery.schema.json new file mode 100644 index 00000000..a7ba1252 --- /dev/null +++ b/specs/schema/battery.schema.json @@ -0,0 +1,37 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "battery.schema.json", + "title": "Battery: completeness + executable gates on the FINAL selected artifact", + "type": "object", + "additionalProperties": false, + "required": [ + "completeness", + "gates" + ], + "properties": { + "completeness": { + "type": "object", + "additionalProperties": false, + "required": [ + "targets", + "source" + ], + "properties": { + "targets": { + "type": "integer", + "minimum": 1 + }, + "source": { + "type": "string" + } + } + }, + "gates": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/gate_record" + } + } + } +} diff --git a/specs/schema/bundle.schema.json b/specs/schema/bundle.schema.json new file mode 100644 index 00000000..335ff821 --- /dev/null +++ b/specs/schema/bundle.schema.json @@ -0,0 +1,30 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "bundle.schema.json", + "title": "Bundle-level settings (no file inventory - that is country_package.json)", + "type": "object", + "additionalProperties": false, + "required": [ + "country", + "identity_generation", + "seed_protocol" + ], + "properties": { + "country": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "identity_generation": { + "type": "integer", + "minimum": 1 + }, + "seed_protocol": { + "enum": [ + "legacy-v1", + "derived-v2" + ] + }, + "status": { + "type": "string" + } + } +} diff --git a/specs/schema/calibration.schema.json b/specs/schema/calibration.schema.json new file mode 100644 index 00000000..287130b8 --- /dev/null +++ b/specs/schema/calibration.schema.json @@ -0,0 +1,172 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "calibration.schema.json", + "title": "Calibration: the full mathematical contract (no library defaults)", + "type": "object", + "additionalProperties": false, + "required": [ + "solver", + "targets" + ], + "properties": { + "solver": { + "type": "object", + "additionalProperties": false, + "required": [ + "kernel", + "loss", + "target_scaling", + "row_weighting", + "zero_negative_target_policy", + "objective_aggregation", + "mass_constraints", + "initialization", + "optimizer", + "stopping", + "infeasibility_policy", + "target_priority", + "max_weight_ratio", + "l0" + ], + "properties": { + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "loss": { + "type": "object", + "additionalProperties": false, + "required": [ + "formula_id" + ], + "properties": { + "formula_id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "params": { + "type": "object" + } + } + }, + "target_scaling": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "row_weighting": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "zero_negative_target_policy": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "objective_aggregation": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "mass_constraints": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "initialization": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "optimizer": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "dtype" + ], + "properties": { + "name": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "schedule": { + "type": "object" + }, + "dtype": { + "enum": [ + "float32", + "float64" + ] + } + } + }, + "stopping": { + "type": "object", + "additionalProperties": false, + "required": [ + "max_epochs" + ], + "properties": { + "max_epochs": { + "type": "integer", + "minimum": 1 + }, + "tolerance": { + "type": "number" + }, + "patience": { + "type": "integer" + } + } + }, + "infeasibility_policy": { + "enum": [ + "refuse", + "report_and_continue" + ] + }, + "target_priority": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "max_weight_ratio": { + "type": "number", + "exclusiveMinimum": 1 + }, + "l0": { + "type": "object", + "additionalProperties": false, + "required": [ + "mode" + ], + "properties": { + "mode": { + "enum": [ + "exact_k_projection", + "none" + ] + } + } + } + } + }, + "targets": { + "type": "object", + "additionalProperties": false, + "required": [ + "source", + "geography_layers", + "cd_policy" + ], + "properties": { + "source": { + "const": "chronicle_facts" + }, + "geography_layers": { + "type": "array", + "minItems": 1, + "items": { + "enum": [ + "national", + "state", + "congressional_district", + "county" + ] + } + }, + "cd_policy": { + "const": "always_present_report_attainment" + } + } + } + } +} diff --git a/specs/schema/catalogs.schema.json b/specs/schema/catalogs.schema.json new file mode 100644 index 00000000..b932ba75 --- /dev/null +++ b/specs/schema/catalogs.schema.json @@ -0,0 +1,95 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "catalogs.schema.json", + "title": "Column catalog: normative contract + docs overlay; lineage is COMPILER-EMITTED, never authored", + "type": "object", + "additionalProperties": false, + "required": [ + "columns" + ], + "x-forbidden-fields": [ + "owner", + "lineage_class" + ], + "properties": { + "columns": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "key", + "contract" + ], + "properties": { + "key": { + "type": "string", + "description": "Compiler-derived stable column key; authored rows must match a compiled key or fail." + }, + "contract": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "dtype", + "period", + "nullable", + "stability" + ], + "properties": { + "entity": { + "$ref": "defs.schema.json#/$defs/entity" + }, + "dtype": { + "enum": [ + "float64", + "float32", + "int64", + "int32", + "bool", + "string", + "category" + ] + }, + "unit": { + "type": "string" + }, + "period": { + "$ref": "defs.schema.json#/$defs/vintage_ref" + }, + "nullable": { + "type": "boolean" + }, + "domain": { + "type": "string" + }, + "stability": { + "enum": [ + "public_stable", + "public_experimental", + "internal" + ] + } + } + }, + "docs": { + "type": "object", + "additionalProperties": false, + "description": "Non-normative; hashed into the documentation digest, NOT spec_sha256.", + "properties": { + "description": { + "type": "string" + }, + "citations": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + } + } + } + } +} diff --git a/specs/schema/defs.schema.json b/specs/schema/defs.schema.json new file mode 100644 index 00000000..b4a55683 --- /dev/null +++ b/specs/schema/defs.schema.json @@ -0,0 +1,317 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "defs.schema.json", + "title": "Shared definitions for the microcosm spec bundle", + "$defs": { + "sha256": { + "type": "string", + "pattern": "^[a-f0-9]{64}$" + }, + "identifier": { + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$" + }, + "kernel_ref": { + "type": "string", + "pattern": "^kernel:[a-z][a-z0-9_]*$" + }, + "source_ref": { + "type": "string", + "pattern": "^source:[a-z][a-z0-9_.]*$" + }, + "stream_ref": { + "type": "string", + "pattern": "^stream:[a-z][a-z0-9_.]*$" + }, + "vintage_ref": { + "type": "string", + "pattern": "^vintage:[a-z][a-z0-9_]*$" + }, + "entity": { + "enum": [ + "person", + "tax_unit", + "spm_unit", + "household", + "family", + "benunit" + ] + }, + "surface": { + "enum": [ + "normative", + "run_request", + "execution_profile", + "operational", + "chain_state" + ], + "description": "Only the normative projection is hashed into spec_sha256." + }, + "row_scope": { + "oneOf": [ + { + "$ref": "defs.schema.json#/$defs/identifier" + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "all_of" + ], + "properties": { + "all_of": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/row_scope" + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "any_of" + ], + "properties": { + "any_of": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/row_scope" + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "not" + ], + "properties": { + "not": { + "$ref": "#/$defs/row_scope" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "channel_is" + ], + "properties": { + "channel_is": { + "$ref": "#/$defs/identifier" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "support_role_is" + ], + "properties": { + "support_role_is": { + "$ref": "#/$defs/identifier" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "column_predicate" + ], + "properties": { + "column_predicate": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "column", + "op" + ], + "properties": { + "entity": { + "$ref": "#/$defs/entity" + }, + "column": { + "$ref": "#/$defs/identifier" + }, + "op": { + "enum": [ + "eq", + "ne", + "is_null", + "not_null" + ] + }, + "value": {} + } + } + } + } + ], + "description": "Closed predicate algebra: overlap, exhaustiveness, and equality are compiler-decidable." + }, + "gate_record": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "metric", + "input", + "population", + "reference", + "min_support", + "thresholds", + "missing_slice", + "status", + "reason_map" + ], + "properties": { + "id": { + "$ref": "#/$defs/identifier" + }, + "metric": { + "type": "object", + "additionalProperties": false, + "required": [ + "formula_id" + ], + "properties": { + "formula_id": { + "$ref": "#/$defs/identifier" + }, + "params": { + "type": "object" + } + } + }, + "input": { + "type": "object", + "additionalProperties": false, + "required": [ + "artifact", + "stage" + ], + "properties": { + "artifact": { + "$ref": "#/$defs/identifier" + }, + "stage": { + "$ref": "#/$defs/identifier" + } + } + }, + "population": { + "type": "object", + "additionalProperties": false, + "required": [ + "universe", + "denominator" + ], + "properties": { + "universe": { + "$ref": "#/$defs/row_scope" + }, + "denominator": { + "$ref": "#/$defs/identifier" + } + } + }, + "slices": { + "type": "array", + "items": { + "$ref": "#/$defs/identifier" + } + }, + "reference": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "enum": [ + "release", + "data_digest", + "external_fact" + ] + }, + "release_id": { + "type": "string" + }, + "sha256": { + "$ref": "#/$defs/sha256" + }, + "fact_ref": { + "$ref": "#/$defs/source_ref" + } + } + }, + "min_support": { + "type": "integer", + "minimum": 0 + }, + "thresholds": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "absolute": { + "type": "number" + }, + "relative": { + "type": "number" + } + } + }, + "uncertainty": { + "type": "object", + "additionalProperties": false, + "required": [ + "rule" + ], + "properties": { + "rule": { + "enum": [ + "none", + "multi_seed", + "interval" + ] + }, + "seeds": { + "type": "integer", + "minimum": 2 + } + } + }, + "missing_slice": { + "enum": [ + "fail", + "skip_with_receipt" + ] + }, + "status": { + "enum": [ + "fail", + "warn", + "report_only" + ] + }, + "reason_map": { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/identifier" + } + } + } + } + } +} diff --git a/specs/schema/geography.schema.json b/specs/schema/geography.schema.json new file mode 100644 index 00000000..1ba818e0 --- /dev/null +++ b/specs/schema/geography.schema.json @@ -0,0 +1,134 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "geography.schema.json", + "title": "Geography assignment (block-first form is PHASE F3; the flip encodes legacy)", + "type": "object", + "additionalProperties": false, + "required": [ + "phase", + "assignment" + ], + "properties": { + "phase": { + "enum": [ + "legacy", + "f3" + ] + }, + "status": { + "type": "string" + }, + "assignment": { + "type": "object", + "additionalProperties": false, + "required": [ + "anchor", + "order", + "kernels", + "draw", + "derive", + "assertions", + "ladder_source", + "seed" + ], + "properties": { + "anchor": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "order": { + "enum": [ + "before_gap_fill", + "legacy_post_transfer" + ] + }, + "kernels": { + "type": "object", + "additionalProperties": false, + "required": [ + "assign", + "validate" + ], + "properties": { + "assign": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "validate": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + } + } + }, + "draw": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "universe", + "weight" + ], + "properties": { + "universe": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "weight": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": [ + "universe", + "weight" + ], + "properties": { + "universe": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "weight": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + "minProperties": 1 + } + ] + } + }, + "identified_county_source": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "derive": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "assertions": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "ladder_source": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "seed": { + "$ref": "defs.schema.json#/$defs/stream_ref" + }, + "validation": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + } +} diff --git a/specs/schema/imputation.schema.json b/specs/schema/imputation.schema.json new file mode 100644 index 00000000..c392eee0 --- /dev/null +++ b/specs/schema/imputation.schema.json @@ -0,0 +1,539 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "imputation.schema.json", + "title": "Predictor blocks, models, chaining, concepts, families, producer graph", + "type": "object", + "additionalProperties": false, + "required": [ + "predictor_blocks", + "models", + "chaining", + "concepts", + "families", + "producer_graph" + ], + "properties": { + "predictor_blocks": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": [ + "columns" + ], + "properties": { + "columns": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "tags": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "availability": { + "enum": [ + "observed", + "always" + ] + }, + "provides_concepts": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + }, + "models": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": [ + "kernel", + "params" + ], + "properties": { + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "params": { + "type": "object", + "additionalProperties": false, + "description": "SCHEMA-COMPLETE: every build-facing parameter explicit; library defaults never reach a build.", + "required": [ + "n_estimators", + "max_samples_leaf", + "zero_atol" + ], + "properties": { + "n_estimators": { + "type": "integer", + "minimum": 1 + }, + "max_samples_leaf": { + "type": [ + "integer", + "null" + ], + "minimum": 1 + }, + "zero_atol": { + "type": "number", + "exclusiveMinimum": 0 + } + } + } + } + } + }, + "chaining": { + "type": "object", + "additionalProperties": false, + "required": [ + "order", + "splits", + "memory_policy" + ], + "properties": { + "order": { + "const": "declared" + }, + "splits": { + "const": "declared_only" + }, + "split_after": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "family", + "after_target", + "reason" + ], + "properties": { + "family": { + "type": "string" + }, + "after_target": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "reason": { + "type": "string", + "minLength": 10 + } + } + } + }, + "memory_policy": { + "const": "release_after_draw" + }, + "keep_together": { + "type": "array", + "items": { + "type": "array", + "minItems": 2, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + }, + "concepts": { + "type": "object", + "description": "Concept registry: named concept -> predictor columns.", + "additionalProperties": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + "families": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "stage", + "donor", + "model", + "predictors", + "targets" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9_/]+$" + }, + "stage": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "donor": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "maxProperties": 1, + "properties": { + "channel": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "support_role": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + "recipient": { + "type": "object", + "additionalProperties": false, + "properties": { + "channel": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + "model": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "predictors": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "targets": { + "type": "array", + "minItems": 1, + "description": "Participation targets carry TARGET-SPECIFIC concept sets; coverage failure fails the LOAD.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "properties": { + "name": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "requires_concepts": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "value_kind": { + "enum": [ + "amount", + "flag", + "count", + "category" + ] + } + } + } + } + } + } + }, + "producer_graph": { + "type": "object", + "additionalProperties": false, + "required": [ + "external_stages", + "nodes", + "ownership_matrix" + ], + "properties": { + "external_stages": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "ordering": { + "const": "deterministic_total", + "description": "Incomparable nodes must commute or occupy disjoint write scopes; the compiler emits one total order." + }, + "nodes": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "kernel", + "inputs", + "outputs" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "depends_on": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "inputs": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "column", + "required_scope", + "producing_stage", + "absence" + ], + "properties": { + "entity": { + "$ref": "defs.schema.json#/$defs/entity" + }, + "column": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "value_kind": { + "enum": [ + "amount", + "flag", + "count", + "category", + "weight" + ] + }, + "required_scope": { + "$ref": "defs.schema.json#/$defs/row_scope" + }, + "producing_stage": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "absence": { + "type": "object", + "additionalProperties": false, + "required": [ + "policy" + ], + "properties": { + "policy": { + "enum": [ + "fatal", + "tolerated_with_receipt" + ] + }, + "receipt_ids": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + }, + "alternatives": { + "type": "array", + "description": "Ordered OR-of-AND with declared precedence; predicates disjoint + exhaustive.", + "items": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "column" + ], + "properties": { + "entity": { + "$ref": "defs.schema.json#/$defs/entity" + }, + "column": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "value_kind": { + "enum": [ + "amount", + "flag", + "count", + "category", + "weight" + ] + } + } + } + } + } + } + } + }, + "outputs": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "column", + "coverage_scope", + "presence" + ], + "properties": { + "entity": { + "$ref": "defs.schema.json#/$defs/entity" + }, + "column": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "coverage_scope": { + "$ref": "defs.schema.json#/$defs/row_scope" + }, + "presence": { + "enum": [ + "required", + "forbidden", + "predicated" + ] + }, + "predicate": { + "$ref": "defs.schema.json#/$defs/row_scope" + }, + "temporary": { + "type": "boolean" + }, + "validation_only": { + "type": "boolean" + } + } + } + }, + "virtual_resources": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "kind" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "kind": { + "enum": [ + "manifest", + "resolved_weights", + "execution_config", + "transition_receipt", + "producer_receipt", + "target_bank" + ] + }, + "digest_rule": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + }, + "structural_delta": { + "enum": [ + "none", + "filter", + "expand", + "join", + "relink", + "reorder", + "reweight" + ] + }, + "retry_safety": { + "enum": [ + "idempotent", + "attempt_scoped", + "nonretryable" + ] + } + } + } + }, + "transfer_groups": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "members" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "members": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + }, + "ownership_matrix": { + "type": "array", + "description": "The full conditional ownership matrix (18 rows today), keyed target x origin x clone role.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "target", + "origin", + "clone_role", + "owner", + "non_owner_action" + ], + "properties": { + "target": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "origin": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "clone_role": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "owner": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "finalizes": { + "type": "boolean" + }, + "non_owner_action": { + "enum": [ + "preserve", + "fill_missing_only", + "forbidden_write" + ] + } + } + } + } + } + } + } +} diff --git a/specs/schema/locks.schema.json b/specs/schema/locks.schema.json new file mode 100644 index 00000000..835a36a3 --- /dev/null +++ b/specs/schema/locks.schema.json @@ -0,0 +1,232 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "locks.schema.json", + "title": "EMITTED artifacts (never authored; reproducible-or-rejected)", + "$defs": { + "bundle_lock": { + "type": "object", + "additionalProperties": false, + "required": [ + "grammar_receipt", + "files", + "spec_sha256" + ], + "properties": { + "grammar_receipt": { + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "canonicalizer_version", + "migration_chain" + ], + "properties": { + "schema_version": { + "type": "integer" + }, + "canonicalizer_version": { + "type": "integer" + }, + "migration_chain": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "sha256" + ], + "properties": { + "id": { + "type": "string" + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + } + } + } + } + } + }, + "files": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": [ + "sha256", + "byte_size" + ], + "properties": { + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + }, + "byte_size": { + "type": "integer" + } + } + } + }, + "spec_sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + } + } + }, + "plan_lock": { + "type": "object", + "additionalProperties": false, + "required": [ + "compiler_ir_abi", + "nodes" + ], + "properties": { + "compiler_ir_abi": { + "type": "object", + "additionalProperties": false, + "required": [ + "version", + "sha256" + ], + "properties": { + "version": { + "type": "integer" + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + } + } + }, + "nodes": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "node_key", + "kernel", + "depends_on", + "inputs", + "outputs", + "seed_streams" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "node_key": { + "$ref": "defs.schema.json#/$defs/sha256" + }, + "kernel": { + "type": "object", + "additionalProperties": false, + "required": [ + "ref", + "implementation_sha256" + ], + "properties": { + "ref": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "implementation_sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + } + } + }, + "depends_on": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + }, + "inputs": { + "type": "array", + "items": { + "type": "string" + } + }, + "outputs": { + "type": "array", + "items": { + "type": "string" + } + }, + "seed_streams": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/stream_ref" + } + } + } + } + } + } + }, + "engine_abi_lock": { + "type": "object", + "additionalProperties": false, + "required": [ + "engine", + "programs" + ], + "properties": { + "engine": { + "type": "object", + "additionalProperties": false, + "required": [ + "package", + "version" + ], + "properties": { + "package": { + "type": "string" + }, + "version": { + "type": "string" + } + } + }, + "programs": { + "type": "object", + "description": "Generated from the single engine pin; CI compares a fresh derivation BEFORE bundle compilation; bumps fail closed. Total-and-injective program->variable mapping asserted.", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "required": [ + "variable", + "entity", + "value_type", + "default", + "engine_class" + ], + "properties": { + "variable": { + "type": "string" + }, + "entity": { + "$ref": "defs.schema.json#/$defs/entity" + }, + "value_type": { + "enum": [ + "bool", + "float", + "int" + ] + }, + "default": {}, + "engine_class": { + "type": "string" + }, + "consumers": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + } + } + } + } +} diff --git a/specs/schema/publication.schema.json b/specs/schema/publication.schema.json new file mode 100644 index 00000000..1983658e --- /dev/null +++ b/specs/schema/publication.schema.json @@ -0,0 +1,152 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "publication.schema.json", + "title": "Publication: attempt events, atomic promotion, audit chain != release graph", + "type": "object", + "additionalProperties": false, + "required": [ + "attempts", + "promotion", + "release", + "audit_chain", + "release_graph" + ], + "properties": { + "attempts": { + "type": "object", + "additionalProperties": false, + "required": [ + "model", + "terminal_states" + ], + "properties": { + "model": { + "const": "append_only_events_then_terminal_seal" + }, + "terminal_states": { + "type": "array", + "items": { + "enum": [ + "landed", + "failed", + "expired" + ] + } + } + } + }, + "promotion": { + "type": "object", + "additionalProperties": false, + "required": [ + "latest_flip", + "idempotency", + "recovery" + ], + "properties": { + "latest_flip": { + "const": "human_gate" + }, + "idempotency": { + "const": "required_key" + }, + "recovery": { + "type": "array", + "items": { + "enum": [ + "seal_ok_append_fail", + "append_ok_alias_fail", + "orphan_reconciliation", + "expiry_reconciliation" + ] + } + } + } + }, + "release": { + "type": "object", + "additionalProperties": false, + "required": [ + "line", + "pattern", + "rungs" + ], + "properties": { + "line": { + "type": "object", + "additionalProperties": false, + "required": [ + "value", + "normative" + ], + "description": "D6 RULED (Max 2026-08-16): microcosm-us-2024-* is the spec-engine-era line; NORMATIVE (hashed into spec_sha256). Readers accept both prefixes; populace-* rows stay valid-historical.", + "properties": { + "value": { + "type": "string", + "pattern": "^microcosm-[a-z]{2}-[0-9]{4}$" + }, + "normative": { + "const": true + }, + "note": { + "type": "string" + }, + "legacy_prefixes": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Prefixes readers must still accept (valid-historical rows)." + } + } + }, + "pattern": { + "type": "string" + }, + "rungs": { + "type": "array", + "minItems": 1, + "items": { + "type": "string", + "pattern": "^f[0-9]{3}$" + } + } + } + }, + "audit_chain": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "const": "strict_linear" + }, + "store": { + "type": "string", + "description": "OPERATIONAL binding; never hashed." + } + } + }, + "release_graph": { + "type": "object", + "additionalProperties": false, + "required": [ + "relations" + ], + "properties": { + "relations": { + "type": "array", + "items": { + "enum": [ + "derived_from", + "supersedes", + "revokes" + ] + } + } + } + } + } +} diff --git a/specs/schema/resource_manifest.schema.json b/specs/schema/resource_manifest.schema.json new file mode 100644 index 00000000..0eef1fa9 --- /dev/null +++ b/specs/schema/resource_manifest.schema.json @@ -0,0 +1,61 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "resource_manifest.schema.json", + "title": "Typed resource rows for country_package.json (the SINGLE manifest)", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "country", + "resources" + ], + "properties": { + "schema_version": { + "type": "integer", + "minimum": 1 + }, + "country": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "resources": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "path", + "kind", + "schema_id" + ], + "properties": { + "path": { + "type": "string", + "pattern": "^[a-z0-9_./-]+$" + }, + "kind": { + "enum": [ + "bundle", + "sources", + "spine", + "geography", + "imputation", + "take_up", + "battery", + "calibration", + "selection", + "publication", + "vintages", + "catalogs", + "schema", + "legacy_json" + ] + }, + "schema_id": { + "type": "string" + } + } + } + } + } +} diff --git a/specs/schema/selection.schema.json b/specs/schema/selection.schema.json new file mode 100644 index 00000000..d758100e --- /dev/null +++ b/specs/schema/selection.schema.json @@ -0,0 +1,72 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "selection.schema.json", + "title": "Sparse selection: exact-k (validity gates are KERNEL CONTRACT, not config)", + "type": "object", + "additionalProperties": false, + "required": [ + "exact_k" + ], + "properties": { + "exact_k": { + "type": "object", + "additionalProperties": false, + "required": [ + "kernel", + "k", + "pi_hi", + "group_ids", + "on_infeasible", + "post_selection_weights" + ], + "properties": { + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "k": { + "type": "object", + "additionalProperties": false, + "required": [ + "default", + "surface" + ], + "description": "k is a run-request knob; the bundle declares only a default with explicit precedence.", + "properties": { + "default": { + "type": "integer", + "minimum": 1 + }, + "surface": { + "const": "run_request" + }, + "precedence": { + "const": "run_request_overrides_default" + } + } + }, + "pi_hi": { + "type": "number", + "exclusiveMinimum": 0, + "maximum": 1 + }, + "group_ids": { + "oneOf": [ + { + "const": "none" + }, + { + "$ref": "defs.schema.json#/$defs/identifier" + } + ] + }, + "on_infeasible": { + "const": "refuse" + }, + "post_selection_weights": { + "const": "selected_keep_calibrated", + "description": "Selected records carry calibrated weights unchanged; non-selected absent; no re-normalization unless declared." + } + } + } + } +} diff --git a/specs/schema/sources.schema.json b/specs/schema/sources.schema.json new file mode 100644 index 00000000..5635bb5b --- /dev/null +++ b/specs/schema/sources.schema.json @@ -0,0 +1,50 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "sources.schema.json", + "title": "External inputs: logical ids + content pins. NO paths (operational); NO engine pin (vintages).", + "type": "object", + "additionalProperties": false, + "required": [ + "sources" + ], + "properties": { + "sources": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "role", + "sha256", + "loader" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "role": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + }, + "byte_size": { + "type": "integer", + "minimum": 1 + }, + "loader": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "vintages": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/vintage_ref" + } + } + } + } + } + } +} diff --git a/specs/schema/spine.schema.json b/specs/schema/spine.schema.json new file mode 100644 index 00000000..6ae57f00 --- /dev/null +++ b/specs/schema/spine.schema.json @@ -0,0 +1,108 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "spine.schema.json", + "title": "Channels (ORDERED - structural), assembly, support roles", + "type": "object", + "additionalProperties": false, + "required": [ + "channels", + "assembly", + "support_roles" + ], + "properties": { + "channels": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "source", + "observed_geography" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "source": { + "oneOf": [ + { + "$ref": "defs.schema.json#/$defs/identifier" + }, + { + "type": "array", + "minItems": 1, + "items": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + ] + }, + "observed_geography": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + }, + "assembly": { + "type": "object", + "additionalProperties": false, + "required": [ + "mass_anchor_channel", + "shared_dtype_policy" + ], + "properties": { + "mass_anchor_channel": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "shared_dtype_policy": { + "const": "canonical_string_storage" + } + } + }, + "support_roles": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "kind" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "kind": { + "enum": [ + "puf_attachment", + "synthetic_prior_replacement", + "none" + ] + }, + "clone_index": { + "type": "integer", + "minimum": 0 + }, + "tail_support": { + "type": "object", + "additionalProperties": false, + "required": [ + "strata", + "policy" + ], + "properties": { + "strata": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "policy": { + "const": "declared_min_support_with_receipt" + } + } + } + } + } + } + } +} diff --git a/specs/schema/take_up.schema.json b/specs/schema/take_up.schema.json new file mode 100644 index 00000000..5eca6e33 --- /dev/null +++ b/specs/schema/take_up.schema.json @@ -0,0 +1,315 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "take_up.schema.json", + "title": "Take-up: orthogonal ownership x ordered typed steps; facts bind via engine_abi.lock", + "type": "object", + "additionalProperties": false, + "required": [ + "programs" + ], + "properties": { + "status": { + "type": "string" + }, + "programs": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "ownership" + ], + "oneOf": [ + { + "required": [ + "pipeline" + ] + }, + { + "required": [ + "segments" + ] + } + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "ownership": { + "enum": [ + "measured", + "transferred", + "modeled", + "engine", + "mixed" + ] + }, + "pipeline": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "enum": [ + "probability_seed", + "count_calibration", + "delivery_gate", + "assignment", + "measured_map", + "engine_default", + "imputed_transfer" + ] + }, + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "rate_source": { + "type": "string" + }, + "targets": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "domain": { + "enum": [ + "eligible_only", + "all_rows" + ] + }, + "assignment": { + "enum": [ + "unmasked_stable_source_id", + "masked" + ] + }, + "prior": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "method": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "saturation": { + "enum": [ + "cap_at_universe", + "observed_rate_fallback" + ] + }, + "gate": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "source_column": { + "type": "string" + }, + "propagation": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "training": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "model": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "debt": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + }, + "segments": { + "type": "array", + "minItems": 2, + "description": "Mixed row-scope programs declare disjoint, exhaustive segments.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "row_scope", + "ownership", + "pipeline" + ], + "properties": { + "row_scope": { + "$ref": "defs.schema.json#/$defs/row_scope" + }, + "ownership": { + "enum": [ + "measured", + "transferred", + "modeled", + "engine" + ] + }, + "pipeline": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "enum": [ + "probability_seed", + "count_calibration", + "delivery_gate", + "assignment", + "measured_map", + "engine_default", + "imputed_transfer" + ] + }, + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "rate_source": { + "type": "string" + }, + "targets": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "domain": { + "enum": [ + "eligible_only", + "all_rows" + ] + }, + "assignment": { + "enum": [ + "unmasked_stable_source_id", + "masked" + ] + }, + "prior": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "method": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "saturation": { + "enum": [ + "cap_at_universe", + "observed_rate_fallback" + ] + }, + "gate": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "source_column": { + "type": "string" + }, + "propagation": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "training": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "model": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "debt": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + } + } + }, + "dependence": { + "type": "object", + "additionalProperties": false, + "properties": { + "group": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + }, + "final_owner_stage": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } + }, + "$defs": { + "step": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind" + ], + "properties": { + "kind": { + "enum": [ + "probability_seed", + "count_calibration", + "delivery_gate", + "assignment", + "measured_map", + "engine_default", + "imputed_transfer" + ] + }, + "kernel": { + "$ref": "defs.schema.json#/$defs/kernel_ref" + }, + "rate_source": { + "type": "string" + }, + "targets": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "domain": { + "enum": [ + "eligible_only", + "all_rows" + ] + }, + "assignment": { + "enum": [ + "unmasked_stable_source_id", + "masked" + ] + }, + "prior": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "method": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "saturation": { + "enum": [ + "cap_at_universe", + "observed_rate_fallback" + ] + }, + "gate": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "source_column": { + "type": "string" + }, + "propagation": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "training": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "model": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "debt": { + "$ref": "defs.schema.json#/$defs/identifier" + } + } + } + } +} diff --git a/specs/schema/vintages.schema.json b/specs/schema/vintages.schema.json new file mode 100644 index 00000000..80520dc3 --- /dev/null +++ b/specs/schema/vintages.schema.json @@ -0,0 +1,56 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "vintages.schema.json", + "title": "Typed vintage/reference records; the engine pin appears here EXACTLY ONCE", + "type": "object", + "additionalProperties": false, + "required": [ + "records" + ], + "properties": { + "records": { + "type": "array", + "minItems": 1, + "description": "Other files refer by vintage: ref; CI rejects duplicate literal authorities.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "kind", + "value" + ], + "properties": { + "id": { + "$ref": "defs.schema.json#/$defs/identifier" + }, + "kind": { + "enum": [ + "tax_period", + "survey_period", + "target_period", + "geography_vintage", + "policy_engine_surface", + "release_series" + ] + }, + "value": { + "type": [ + "string", + "integer" + ] + }, + "source": { + "$ref": "defs.schema.json#/$defs/source_ref" + }, + "compatible_with": { + "type": "array", + "items": { + "$ref": "defs.schema.json#/$defs/vintage_ref" + } + } + } + } + } + } +} diff --git a/specs/us/bundle.yaml b/specs/us/bundle.yaml new file mode 100644 index 00000000..3cb0556c --- /dev/null +++ b/specs/us/bundle.yaml @@ -0,0 +1,18 @@ +# DRAFT (v3) — pseudocode until schemas land. +# +# Round 2 (sol M10 / Pro): country_package.json becomes THE single typed +# resource manifest ({path, kind, schema_id} rows replacing bare +# filenames) — this file no longer carries a second file inventory. +# Bundle-level settings only. Root specs/ is a drafting location; the +# bundle ships as package data through the CountrySpec seam. +country: us +identity_generation: 1 +seed_protocol: legacy-v1 # enforced via the RNG broker + draw-site + # ledger; derived-v2 is an F3 edit +status: draft-v3-pseudocode +# Emitted (never authored): bundle.lock.json, plan.lock.json, +# engine_abi.lock.json. Generated locks are reproducible from their +# authorities and rejected if hand-edited. +# Identity: run_provenance_identity (records) vs node_reuse_key (reuse) — +# see docs/spec-engine.md "Identity". The release line in publication is +# PROVISIONAL/non-normative pending D6 (excluded from spec_sha256). diff --git a/specs/us/geography.yaml b/specs/us/geography.yaml new file mode 100644 index 00000000..ef0c5e5b --- /dev/null +++ b/specs/us/geography.yaml @@ -0,0 +1,51 @@ +# DRAFT FOR REVIEW — block-first assignment (Max's 2026-08-15 ruling, #696). +# +# PHASE F3 (sol MAJOR 4): this file is the TARGET state, not the flip. +# No existing kernel performs what it declares (the block ladder lacks +# PUMA and samples inside an already-assigned CD; the PUMA ladder never +# writes block). The equivalence flip (F0–F2) encodes LEGACY geography +# exactly; this lands afterward as an intentional behavior change with a +# new versioned block artifact (exact 2020-PUMA relationship), fresh +# identity, cold caches, and statistical gates. +status: draft-for-review +phase: F3 +assignment: + anchor: census_block_2020 + order: before_gap_fill # so PUMA/county serve as gap-fill predictors + kernels: + assign: kernel:block_first_assignment + validate: kernel:block_geography_validation + draw: + acs: {universe: observed_puma, weight: block_population_2020} + asec: + county_identified: {universe: that_county, weight: block_population_2020} + # County identification is a disclosure property of the county (whole + # counties are on or off the CPS list), so an unidentified household is + # KNOWN not to live in any identified county: its universe is the + # state's complement — the SOLE rule, never the whole state (Max ruling + # 2026-08-16; sol MINOR 1 killed the else_state fallback text). + county_unidentified: {universe: state_minus_identified_counties, + weight: block_population_2020} + identified_county_source: source:census_cps_identified_counties + # A pinned official source id (sources.yaml), sha-bound into identity. + # No derived-from-pooled-file fallback in production: a fallback derived + # from a sampled file varies by rung (sol MINOR 1). If one is ever + # unavoidable it needs an exact pre-sampling derivation + resolved list + # digest in identity — refused until then. + derive: + tract_geoid: structural_prefix + county_fips: structural_prefix + state_fips: structural_prefix + puma_2020: {rule: tract_relationship, assert: equals_observed_for_acs} + congressional_district_geoid: {rule: block_equivalency, vintage: cd119} + place_fips: {rule: block_crosswalk} + sldu: {rule: block_crosswalk} + sldl: {rule: block_crosswalk} + cbsa_code: {rule: block_crosswalk} + assertions: + - observed_acs_state_and_puma_preserved + - tract_to_puma_exact + ladder_source: us_block_ladder_2020 + seed: stream:geography_block_draw # named stream (legacy-v1 map n/a — + # this whole file is post-flip) + validation: [puma_cd_overlap_consistency, vintage_refusal] diff --git a/specs/us/publication.yaml b/specs/us/publication.yaml new file mode 100644 index 00000000..3444438a --- /dev/null +++ b/specs/us/publication.yaml @@ -0,0 +1,22 @@ +# DRAFT (v3) — pseudocode until the loader binds. D6 RULED (Max 2026-08-16). +attempts: + model: append_only_events_then_terminal_seal + terminal_states: [landed, failed, expired] +promotion: + latest_flip: human_gate # unchanged doctrine + idempotency: required_key + recovery: [seal_ok_append_fail, append_ok_alias_fail, + orphan_reconciliation, expiry_reconciliation] +release: + line: + value: microcosm-us-2024 # D6: the spec-engine-era line + normative: true # hashed into spec_sha256 + legacy_prefixes: [populace-us-2024] # readers accept; rows valid-historical + note: "HF dataset destination is NOT part of D6 — unchanged until a separate explicit ruling." + pattern: "{line}-stacked-f{rung}-s{seed}" + rungs: [f001, f004, f010, f025, f100] +audit_chain: + kind: strict_linear + store: supabase:logbook # OPERATIONAL binding; never hashed +release_graph: + relations: [derived_from, supersedes, revokes] diff --git a/specs/us/take_up.yaml b/specs/us/take_up.yaml new file mode 100644 index 00000000..69a1471f --- /dev/null +++ b/specs/us/take_up.yaml @@ -0,0 +1,82 @@ +# DRAFT (v3) — pseudocode until schemas land. Orthogonal ownership × +# ordered typed pipeline steps (round 2 killed the eight-value treatment +# enum: it mixed ownership, mechanism, invocation grouping, policy +# interaction, and an untyped escape hatch). All 13 REAL contract +# programs committed — no ellipsis rows (sol r2 enumerated the actual +# mechanisms; wic/social_security were never contract programs). +# +# Column/entity mapping derives from engine_abi.lock.json (generated from +# the single engine pin, committed, reviewed; engine bumps fail closed) — +# NOT from a suffix naming rule (takes_up_eitc / takes_up_dc_ptc break it). +# The mapping carries a total-and-injective assertion. +status: draft-v3-pseudocode +programs: + - id: snap + ownership: modeled + pipeline: + - {kind: probability_seed, kernel: kernel:take_up_seed_national, + rate_source: source:take_up_reported_anchor} + - {kind: count_calibration, kernel: kernel:snap_state_take_up, + targets: source:fns_state_participation_counts, + domain: eligible_only, assignment: unmasked_stable_source_id, + prior: runtime_target_over_weighted_modeled_eligibles, + saturation: cap_at_universe} + final_owner_stage: count_calibration + - id: tanf + ownership: modeled + pipeline: [{kind: probability_seed, kernel: kernel:take_up_seed_batched, + rate_source: contract:administrative_rate}] + dependence: {group: generic_seeder_batch} + - id: eitc + ownership: modeled + pipeline: [{kind: probability_seed, kernel: kernel:take_up_seed_batched, + rate_source: contract:rate_by_approximated_qualifying_children}] + dependence: {group: generic_seeder_batch} + - id: medicaid + ownership: modeled + pipeline: + - {kind: probability_seed, rate_source: source:reported_anchor} + - {kind: count_calibration, kernel: kernel:medicaid_state_take_up, + targets: source:state_counts, prior: runtime_state_prior, + method: greedy_state_count} + final_owner_stage: count_calibration + - id: chip + ownership: engine + pipeline: [{kind: engine_default, debt: source_follow_up_open}] + - id: basic_health_program + ownership: engine + pipeline: [{kind: engine_default, debt: source_follow_up_open}] + - id: dc_ptc + ownership: engine + pipeline: [{kind: engine_default, debt: source_follow_up_open}] + - id: early_head_start + ownership: engine + pipeline: [{kind: engine_default, debt: source_follow_up_open}] + - id: medicare + ownership: measured + pipeline: [{kind: measured_map, source_column: asec:MCARE, + propagation: support_clones}] + - id: ssi + ownership: modeled + pipeline: # never count-matches flags — typed as + - {kind: probability_seed, # target-derived banded seed + gate + rate_source: derived:age_band_targets_over_modeled_eligibles} + - {kind: delivery_gate, gate: delivered_recipients} + final_owner_stage: delivery_gate + - id: head_start + ownership: transferred + pipeline: [{kind: imputed_transfer, training: source:sipp_direct_response, + model: regime_gated_qrf}] + - id: housing_assistance + ownership: mixed # row-scope segments, not one whole-column + segments: # treatment (measured ASEC rows; imputed + - {row_scope: asec_rows, ownership: measured, # PUF support) + pipeline: [{kind: measured_map, source_column: asec:housing_receipt}]} + - {row_scope: puf_support_rows, ownership: transferred, + pipeline: [{kind: imputed_transfer, model: regime_gated_qrf}]} + - id: aca + ownership: modeled + pipeline: # typed dedicated steps — no untyped + - {kind: assignment, kernel: kernel:marketplace_assignment} # escape + - {kind: count_calibration, kernel: kernel:aca_calibration} # hatch + final_owner_stage: count_calibration