Tracking: live keys and guards blocked on a reference implementation

[florian6973]

This issue exists to stay open. Once the other audit issues close, the repository looks covered — but
the highest-ranked risk in the audit is still entirely unexercised, and nothing on the board would say so.

The blocker

A generated repo ships a stub model.py, so no chain has ever run training or prediction. The rendered
conformance suite gates this whole tier behind skip_if_stub. Closing it needs a reference implementation
living outside the payload — an examples/ tree that a test renders a profile and drops in. That is
the agreed follow-up recorded in CLAUDE.md's "Known gaps", and it is not a reason to put a model back into
template/.

Until it exists, none of the following can be tested, and an agent that picks one up will do one of the
two wrong things: skip_if_stub the test (reproducing the exact gap) or start filling in model.py.

Blocked items

• do_resume end-to-end — that a resumed run actually inherits the checkpoint, and that resuming from
  an unrelated run yields a different model. The path/decision logic is testable now and is covered in
  the torch-free tests issue; only this half is blocked.
• seed — two runs at one seed agree; seed=null skips seed_everything. The settings ship in the
  reproducibility issue; proving them needs two real training runs.
• attach_labels=false — output shape, and that meds_evaluation rejects it as expected.
• _persist_checkpoint's final_weights fallback (train.py:300) — when no ModelCheckpoint
  monitored anything, the last epoch's weights are published instead of a selected best. Recorded in
  the manifest, never asserted.
• load_pretrained_weights's zero-match RuntimeError (train.py:280) — triage first: this may
  need only torch and a two-parameter nn.Module, not a trained artifact. If so it moves to the
  torch-free issue.
• CoverageError (predict.py:175) — reachable in normal use; nothing exercises it.
• InferenceKind.scores / MaterializedPredictCommand — reachable only through
  zero_shot_materialized, which skip_if_stubs.
• test_meds_dev_e2e's final assertions — everything up to the model has been executed for real;
  the assertions never have.

Why one issue and not eight

Eight unactionable issues get re-triaged every time someone reads the list, and each one is an invitation
for an agent to work around the blocker rather than respect it. One issue has a single place to record why
nothing here can move yet.

When the reference implementation lands, this splits into real issues.

Related

Ranked #1 (and, with determinism, #2) in the audit's "how quietly it fails" list — the only silent
failure modes in it. A resume that picks up an unrelated run's checkpoint yields a model that trains,
predicts, and looks entirely plausible.

From the Hydra configuration audit, §2 (docs/CONFIG-AUDIT.md).

[florian6973]

Moving one item here from #4: the load_pretrained_weights zero-match guard (train.py, now the
RuntimeError raised when a warm start matches no parameters).

It was filed in #4 as "triage: determine whether a two-parameter nn.Module and a hand-built state dict
reach it, or whether it needs a trained artifact". That triage belongs here rather than in a torch-free
testing issue, for two reasons:

1. It is not torch-free. Everything else in Cheap torch-free tests for the branches that remain (splits, wrong-type rejection, do_resume) #4 is pure path or dispatch logic; this one needs torch at
   minimum, and plausibly a real checkpoint.
2. docs/PORTING-A-MODEL.md §6 already frames it as a port-time observation — it names this guard
   alongside predict's CoverageError and _persist_checkpoint's final_weights fallback as the
   contract guards a port is the first thing to run against a real model, and asks that anything which
   fires be reported here. That doc now renders into every generated repository, so the instruction
   actually reaches the person doing a port.

It stays worth caring about: it is the load-bearing guard of the finetune profile, and what stops a
renamed encoder from becoming a silent from-scratch run. If the triage shows a synthetic module reaches
it, it can move back out as a cheap test.

[florian6973]

A probe-profile reference implementation now exists — a from-paper port of PORTER
(arXiv:2606.24102) — and it runs the full chain plus meds-dev-model mode=full. Reporting what that
does and does not unblock on this list, since the issue exists to be told.

Exercised, with results:

• seed — two pretrain runs at seed=0 agree bit-for-bit: identical val/loss to 14 digits
  (1.90627920627594 at 6 epochs, 1.906916856765747 at 25) and identical float64 digests of the
  full state_dict. infer → supervised_train → predict from fixed inputs likewise: identical
  probe val/loss, predictions.parquet byte-identical. So seed works — but see infer/predict mis-assign every train-split output: run_predict_step pairs schema_df order with a shuffled dataloader #21: the first
  attempt at this claim failed, and bisecting the disagreement is what found that bug. Worth
  keeping in mind for whatever test lands here: a reproducibility check is a strong bug detector.
• _persist_checkpoint's final_weights fallback — not reached in the probe DAG, and now known
  why: callbacks/default.yaml monitors val/loss, which both BaseLightningModule.validation_step
  and any probe head that logs it satisfy, so every artifact records selection: monitored_best.
  Reaching the fallback needs a run with no validation split or no ModelCheckpoint.
• CoverageError — did not fire; every predictions manifest recorded
  n_expected == n_written per split. Still unexercised in the negative direction.
• load_pretrained_weights' zero-match RuntimeError — your triage note is right that it does
  not need a trained artifact, and separately: the probe DAG never calls it at all, because
  ProbeTrainCommand does not load the pretrained checkpoint. Whatever test lands should use a
  finetune-shaped chain.

Not exercised: attach_labels=false, and do_resume end-to-end.

Two defects fell out of the exercise and are filed separately: #21 (train-split outputs mis-assigned —
this one is silent and changes results) and #22 (cfg.trainer ignored by infer/predict).