Add a custom_featurization data backend (predicate presence columns on MEDS parquet) - #19
Merged
Merged
Conversation
…en precision Ranked #2 in `docs/CONFIG-AUDIT.md`. Seeding was already right — `seed: 0`, `seed_everything(seed, workers=True)`, `num_workers: 0` — but seeding alone does not give the same numbers twice on GPU, and the porting procedure asks a port to produce a metric twice at one seed and treat disagreement as a finding. It could disagree for reasons that had nothing to do with the port. `configs/trainer/default.yaml` now ships `deterministic: warn` and `benchmark: false`. `warn` rather than `true` so an op with no deterministic kernel does not kill a run in the default configuration; `trainer.deterministic=true` is the hard guarantee. That strict mode was not usable before. `torch.use_deterministic_algorithms(True)` raises on CUDA unless `CUBLAS_WORKSPACE_CONFIG` is set, and it raises at the first cuBLAS matmul — minutes in, not at startup. `configure_cublas_workspace()` sets it with `setdefault`, so an exported value still wins. It lives in the dispatcher rather than `train.py` because it must precede any CUDA work and every command reaches torch through that one function; it sets an environment variable and imports nothing, so the torch-free introspection paths are untouched. `torch.set_float32_matmul_precision("medium")` is **deleted** rather than promoted to a config key. Torch's own default is `highest` with `allow_tf32` off, so removing the call is the fix: the contract stops taking a position and the environment matches the one every source implementation ran in. A key would have had to live in four command roots — training *and* inference, since `zero_shot_direct` and `packaged` never train — and be kept agreed across them. `trainer.precision` is the sanctioned speed knob, and a port that specifically needs TF32 sets it in `model.py`, a file the user owns and the ledger already covers. `precision: 32-true` is listed explicitly for a Hydra reason: configs are struct mode, so without the key `trainer.precision=bf16-mixed` needs a `+` prefix that nobody guesses. Two guards, both structural and torch-free: `test_the_contract_takes_no_position_on_matmul_precision` (the pull to re-add it for speed is real) and `test_trainer_config_ships_the_determinism_keys`. The claim is stated exactly, in the generated README and the porting procedure: same seed + same config + same environment -> same metric. Not "reproducible" — changing `num_workers` or `batch_size` legitimately changes results, and cross-hardware reproducibility is not reachable. Proving it needs two real training runs, which needs a model; that half stays in #7. Verified `deterministic: warn` / `benchmark: false` / `precision: 32-true` build a real Lightning Trainer, and that the contract leaves matmul precision at `highest`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n MEDS parquet
A second `data_backend` copier axis (mtd | custom_featurization), specified in
docs/design-featurization.md. `preprocess_data featurization=predicates` parses a
predicates YAML (ACES presence subset: exact / regex / any-of / or(); unsupported
forms skip with warning + manifest record, or error under featurization_strict)
and stamps 0/1 predicate//<name> Int8 columns onto otherwise-untouched
split-sharded MEDS shards, plus features.json as the ordered feature-space
contract. The manifest gains `representation`, digest, n_features, skipped and
match_counts.
The contract ships both representations unconditionally and tolerates either
dependency set: cohort_subjects() dispatches split membership on the manifest,
import guards in lightning/{__init__,modules}.py keep every command importable
without meds-torch-data, and lightning/protocol.py names the datamodule surface
a custom implementation must provide. The copier option gates only the
dependency, rendered configs, the datamodule.py stub and model.yaml's
external_predicates_file={predicates_path} wiring (MEDS-DEV PR#325);
meds-model-add-to-meds-dev ships predicates.yaml alongside model.yaml.
One predicates file: the user-owned predicates.yaml starter is what the tests,
MEDS-DEV runs and production all read — test_property generates its signal
dataset from it, MTD repos build both workspaces so the equivalence guard pins
the two cohort readers to identical label partitions, and the custom CI lane's
venv omits meds-torch-data so the lane itself proves nothing imports it.
Validated: 85 template structural tests (7 profiles x 2 backends); rendered
supervised suites green on both backends (mtd 74 passed / 1 skipped;
custom_featurization without meds-torch-data 72 passed / 3 honest skips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
florian6973
force-pushed
the
feat/data-backend-featurization
branch
from
August 4, 2026 17:57
9049d4b to
fe74148
Compare
3 tasks
florian6973
added a commit
that referenced
this pull request
Aug 4, 2026
…ing agents The agent-facing docs render into every generated repo because that is where a port happens, but an agent only reads them if the task sends it there. Adds the prompt that does, written to close the gaps a port otherwise falls into: choosing profile and data_backend by default rather than by argument, "the tests pass" reported from a run that skipped the slow designed-signal tests and the MEDS-DEV e2e, and source elements dropped because they were awkward for the fixture. Also documents data_backend itself, which landed in #19 without a README entry — a prompt telling an agent to choose a backend is useless if the README never says what the options are.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a second
data_backendcopier axis (mtd|custom_featurization), fully specified indocs/design-featurization.md. Motivation: ports like TECO whose papers use named clinical variables rather than the full code vocabulary — MEDS-DEV's per-dataset ACES predicates become the concept-binding layer.meds_model_base/featurize.py):preprocess_data featurization=predicatesparses a predicates YAML (presence subset: exact / regex / any-of /or()) and stamps 0/1predicate//<name>Int8 columns onto otherwise-untouched split-sharded MEDS shards, writingfeatures.jsonas the ordered feature-space contract. Unsupported forms (value bounds,and()) skip with a warning + manifest record, cascade through dependentor()s, and hard-error underfeaturization_strict=true; zero remaining features is always an error.cohort_subjects()dispatches split membership on the manifest's newrepresentationfield; import guards keep every command importable without meds-torch-data;lightning/protocol.pynames the datamodule surface a custom implementation must provide. The copier option gates only the dependency, rendered configs, thedatamodule.pystub, andmodel.yaml'sexternal_predicates_file={predicates_path}wiring (pairs with MEDS-DEV#325).predicates.yamlstarter is what the tests, MEDS-DEV runs and production all read.test_propertygenerates its designed-signal dataset from it; MTD repos build both workspaces so a new equivalence guard pins the two cohort readers to identical label partitions.supervised × custom_featurizationrendered-smoke lane whose venv omits meds-torch-data — the lane itself proves nothing imports it.Note: this branch was cut from
reproducible-by-default, so it carries those two commits (3f1cc14, 3a421a5) beneath the featurization commit.Test plan
uv run pytest tests/ -q— 85 structural tests, render matrix now 7 profiles × 2 backendssupervised×mtdsuite: 74 passed, 1 skipped (stub e2e), incl. real MTD tensorization and the equivalence guardsupervised×custom_featurizationsuite, installed without meds-torch-data: 72 passed, 3 honest skips; real featurized preprocessing; CLI smoke doubles as the import-guard regression testruff check/ruff format --checkclean at the root and on both rendered payloadstest_meds_dev_e2efor the custom backend (opt-in marker; needs a MEDS-DEV checkout with #325)🤖 Generated with Claude Code