feat: flatten config examples into a single-level shape

data/README.md

@@ -6,14 +6,16 @@ All examples are run from the repository root and write their outputs under `tmp
 
 ## Per-command configs
 
-A per-command config is exactly the command's argument object. Every key has a matching `--flag`, and an explicit flag on the command line still overrides the file (flag > config > default).
+A per-command config is exactly the command's argument object. Its keys map one-to-one to the CLI flags: `--alignment a.fasta` is `alignment: [a.fasta]`, `--clock-rate 0.003` is `clock_rate: 0.003`, `--output-all dir` is `output_all: dir`. An explicit flag on the command line still overrides the file (flag > config > default).
 
 | File                                                                     | Command     | Demonstrates                                                |
 | ------------------------------------------------------------------------ | ----------- | ----------------------------------------------------------- |
 | [`ebola/20/ancestral-parsimony.yaml`](ebola/20/ancestral-parsimony.yaml) | `ancestral` | Fitch parsimony: fast, model-free ancestral states          |
 | [`zika/20/ancestral.yaml`](zika/20/ancestral.yaml)                       | `ancestral` | Marginal reconstruction with an inferred GTR model          |
+| [`sc2/2844/ancestral.yaml`](sc2/2844/ancestral.yaml)                     | `ancestral` | Marginal reconstruction on 2844 SARS-CoV-2 genomes          |
 | [`dengue/100/clock.yaml`](dengue/100/clock.yaml)                         | `clock`     | Root-to-tip regression with a non-default id column         |
 | [`flu/h3n2/20/optimize.yaml`](flu/h3n2/20/optimize.yaml)                 | `optimize`  | Branch-length optimization and min-divergence rerooting     |
+| [`mpox/clade-ii/1000/optimize.yaml`](mpox/clade-ii/1000/optimize.yaml)   | `optimize`  | The mpox `augur refine` replacement (mutations, no indels)  |
 | [`rsv/a/100/prune.yaml`](rsv/a/100/prune.yaml)                           | `prune`     | Dropping short, empty, and shared-mutation branches         |
 | [`zika/20/mugration.yaml`](zika/20/mugration.yaml)                       | `mugration` | Discrete ancestral geography over sampling country          |
 | [`flu/h3n2/200/timetree.yaml`](flu/h3n2/200/timetree.yaml)               | `timetree`  | Dated tree with a fixed clock rate and confidence intervals |
@@ -28,12 +30,14 @@ treetime ancestral --config data/zika/20/ancestral.yaml
 
 A pipeline file lists named steps, each an analysis command with its arguments. Steps run sequentially in one process. A step reads an earlier step's output with `{{ steps.<name>.outputs.<selection> }}`, so intermediate paths are never hand-written. `vars` names shared values once; a top-level `output_all` gives each step its own `<dir>/<step>/` output directory.
 
-| File                                                               | Chain                                   | Demonstrates                                       |
-| ------------------------------------------------------------------ | --------------------------------------- | -------------------------------------------------- |
-| [`mpox/clade-ii/20/pipeline.yaml`](mpox/clade-ii/20/pipeline.yaml) | `optimize` -> `ancestral`               | A Nextstrain-style mpox build in one file          |
-| [`zika/20/pipeline.yaml`](zika/20/pipeline.yaml)                   | `timetree` -> `mugration`               | Phylodynamics and phylogeography end to end        |
-| [`ebola/362/pipeline.yaml`](ebola/362/pipeline.yaml)               | `optimize` -> `ancestral` -> `timetree` | A full three-step build on a real epidemic dataset |
-| [`flu/h3n2/200/pipeline.yaml`](flu/h3n2/200/pipeline.yaml)         | `optimize` -> `timetree`                | A shared, typed clock rate defined once in `vars`  |
+| File                                                                   | Chain                                   | Demonstrates                                                     |
+| ---------------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------- |
+| [`zika/20/pipeline.yaml`](zika/20/pipeline.yaml)                       | `timetree` -> `mugration`               | Phylodynamics and phylogeography end to end                      |
+| [`flu/h3n2/200/pipeline.yaml`](flu/h3n2/200/pipeline.yaml)             | `optimize` -> `timetree`                | A shared, typed clock rate defined once in `vars`                |
+| [`flu/h3n2/500/pipeline.yaml`](flu/h3n2/500/pipeline.yaml)             | `optimize` -> `ancestral` -> `timetree` | A seasonal-influenza build with a fixed clock (500 sequences)    |
+| [`ebola/362/pipeline.yaml`](ebola/362/pipeline.yaml)                   | `optimize` -> `ancestral` -> `timetree` | A three-step build on the 2014 Ebola epidemic dataset            |
+| [`sc2/2844/pipeline.yaml`](sc2/2844/pipeline.yaml)                     | `optimize` -> `ancestral`               | A date-free build on 2844 SARS-CoV-2 genomes                     |
+| [`mpox/clade-ii/1000/pipeline.yaml`](mpox/clade-ii/1000/pipeline.yaml) | `optimize` -> `ancestral` -> `timetree` | The full Nextstrain mpox TreeTime chain (fixed clock, mutations) |
 
 Run a pipeline, preview its plan, or run only part of it:
 
@@ -60,7 +64,7 @@ treetime pipeline --config data/zika/20/pipeline.yaml --steps mugration
 - Validation is static and typed. `--check` and the JSON-schema-backed argument objects catch a bad parameter or a dangling step reference before anything runs; a Snakemake rule only surfaces a bad TreeTime parameter when that rule executes.
 - Reproducible and portable. The config is the exact serialized argument shape, with one dependency, committed beside the data.
 
-This is not a replacement for a full multi-sample workflow manager: incremental rebuilds, cluster scheduling, and non-TreeTime rules (subsampling, alignment, tree building, export) still belong to Snakemake and Augur. The pipeline replaces the TreeTime glue inside such a build. On the Nextstrain mpox `WIP/new_tt` phylogenetic workflow, for instance, `treetime optimize` replaces `augur refine` (branch-length optimization) and `treetime ancestral` folds in `augur translate`; the [`mpox/clade-ii/20/pipeline.yaml`](mpox/clade-ii/20/pipeline.yaml) example runs exactly that pair as one step chain.
+This is not a replacement for a full multi-sample workflow manager: incremental rebuilds, cluster scheduling, and non-TreeTime rules (subsampling, alignment, tree building, export) still belong to Snakemake and Augur. The pipeline replaces the TreeTime glue inside such a build. On the Nextstrain mpox `WIP/new_tt` phylogenetic workflow, for instance, three Snakemake rules drive TreeTime: `treetime optimize` replaces `augur refine` (branch-length optimization), `treetime ancestral` folds in `augur translate`, and `treetime timetree` dates the tree. The [`mpox/clade-ii/1000/pipeline.yaml`](mpox/clade-ii/1000/pipeline.yaml) example runs that chain as three steps in one file.
 
 ## Editor support
 

data/dengue/100/clock.yaml

@@ -10,9 +10,7 @@
 
 tree: "data/dengue/100/tree.nwk"
 metadata: "data/dengue/100/metadata.tsv"
-metadata_id:
-  metadata_id_columns:
-    - "genbank_accession"
+metadata_id_columns:
+  - "genbank_accession"
 
-output:
-  output_all: "tmp/examples/dengue/100/clock"
+output_all: "tmp/examples/dengue/100/clock"

data/ebola/20/ancestral-parsimony.yaml

@@ -9,11 +9,9 @@
 
 tree: "data/ebola/20/tree.nwk"
 alignment:
-  alignment:
-    - "data/ebola/20/aln.fasta.xz"
+  - "data/ebola/20/aln.fasta.xz"
 
 # Fitch parsimony: fast, model-free ancestral states and mutations.
 method_anc: "parsimony"
 
-output:
-  output_all: "tmp/examples/ebola/20/ancestral-parsimony"
+output_all: "tmp/examples/ebola/20/ancestral-parsimony"

data/ebola/362/pipeline.yaml

@@ -23,17 +23,15 @@ steps:
     optimize:
       tree: "{{ vars.data }}/tree.nwk"
       alignment:
-        alignment:
-          - "{{ vars.data }}/aln.fasta.xz"
+        - "{{ vars.data }}/aln.fasta.xz"
       reroot: "min-dev"
 
   # 2. Reconstruct ancestral sequences and mutations on the optimized tree.
   - name: "ancestral"
     ancestral:
       tree: "{{ steps.optimize.outputs.nwk }}"
       alignment:
-        alignment:
-          - "{{ vars.data }}/aln.fasta.xz"
+        - "{{ vars.data }}/aln.fasta.xz"
       method_anc: "marginal"
 
   # 3. Date the optimized tree, keeping the branch lengths from step 1.
@@ -42,7 +40,6 @@ steps:
       tree: "{{ steps.optimize.outputs.nwk }}"
       metadata: "{{ vars.data }}/metadata.tsv"
       alignment:
-        alignment:
-          - "{{ vars.data }}/aln.fasta.xz"
+        - "{{ vars.data }}/aln.fasta.xz"
       coalescent_opt: true
       max_iter: 3

data/flu/h3n2/20/optimize.yaml

@@ -9,17 +9,12 @@
 
 tree: "data/flu/h3n2/20/tree.nwk"
 alignment:
-  alignment:
-    - "data/flu/h3n2/20/aln.fasta.xz"
-
-model_args:
-  model: "infer"
+  - "data/flu/h3n2/20/aln.fasta.xz"
 
 # Per-edge optimizer: Brent's method in sqrt(t) space (the default, matches v0).
 opt_method: "brent-sqrt"
 
 # Reroot by minimizing root-to-tip divergence variance (a date-free method).
 reroot: "min-dev"
 
-output:
-  output_all: "tmp/examples/flu/h3n2/20/optimize"
+output_all: "tmp/examples/flu/h3n2/20/optimize"

data/flu/h3n2/200/pipeline.yaml

@@ -16,24 +16,20 @@ vars:
 output_all: "tmp/examples/flu/h3n2/200/pipeline"
 
 steps:
-  # Optimize branch lengths under an inferred GTR model.
+  # Optimize branch lengths under an inferred GTR model (the default).
   - name: "optimize"
     optimize:
       tree: "{{ vars.data }}/tree.nwk"
       alignment:
-        alignment:
-          - "{{ vars.data }}/aln.fasta.xz"
-      model_args:
-        model: "infer"
+        - "{{ vars.data }}/aln.fasta.xz"
 
   # Date the optimized tree with the fixed clock rate from `vars`.
   - name: "timetree"
     timetree:
       tree: "{{ steps.optimize.outputs.nwk }}"
       metadata: "{{ vars.data }}/metadata.tsv"
       alignment:
-        alignment:
-          - "{{ vars.data }}/aln.fasta.xz"
+        - "{{ vars.data }}/aln.fasta.xz"
       clock_rate: "{{ vars.clock_rate }}"
       clock_std_dev: "{{ vars.clock_std_dev }}"
       confidence: true

data/flu/h3n2/200/timetree.yaml

@@ -12,16 +12,14 @@
 tree: "data/flu/h3n2/200/tree.nwk"
 metadata: "data/flu/h3n2/200/metadata.tsv"
 alignment:
-  alignment:
-    - "data/flu/h3n2/200/aln.fasta.xz"
+  - "data/flu/h3n2/200/aln.fasta.xz"
 
 clock_rate: 0.003
 clock_std_dev: 0.0006
 confidence: true
 coalescent_opt: true
 max_iter: 3
 
-output:
-  output_all: "tmp/examples/flu/h3n2/200/timetree"
+output_all: "tmp/examples/flu/h3n2/200/timetree"
 # Confidence intervals are written to their own TSV when a path is given.
 output_confidence_tsv: "tmp/examples/flu/h3n2/200/timetree/confidence.tsv"

data/flu/h3n2/500/pipeline.yaml

@@ -0,0 +1,49 @@
+# Seasonal influenza H3N2 build: optimize, reconstruct, date (flu/h3n2/500).
+#
+# Run from the repository root:
+#   treetime pipeline --config data/flu/h3n2/500/pipeline.yaml
+#
+# Preview the plan, or resume from a later step after editing it:
+#   treetime pipeline --config data/flu/h3n2/500/pipeline.yaml --check
+#   treetime pipeline --config data/flu/h3n2/500/pipeline.yaml --steps timetree
+#
+# optimize -> ancestral -> timetree on 500 H3N2 hemagglutinin sequences. Seasonal
+# influenza has a well-characterized clock, so the rate is fixed
+# (`clock_rate` 0.003 substitutions/site/year) instead of estimated, defined once
+# in `vars` as a typed number. The optimized tree flows to the later steps through
+# `{{ steps.optimize.outputs.nwk }}`.
+
+vars:
+  data: "data/flu/h3n2/500"
+  clock_rate: 0.003
+  clock_std_dev: 0.0006
+
+output_all: "tmp/examples/flu/h3n2/500/pipeline"
+
+steps:
+  # Optimize branch lengths under an inferred GTR model.
+  - name: "optimize"
+    optimize:
+      tree: "{{ vars.data }}/tree.nwk"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+
+  # Reconstruct ancestral sequences and mutations on the optimized tree.
+  - name: "ancestral"
+    ancestral:
+      tree: "{{ steps.optimize.outputs.nwk }}"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+      method_anc: "marginal"
+
+  # Date the optimized tree with the fixed clock rate from `vars`.
+  - name: "timetree"
+    timetree:
+      tree: "{{ steps.optimize.outputs.nwk }}"
+      metadata: "{{ vars.data }}/metadata.tsv"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+      clock_rate: "{{ vars.clock_rate }}"
+      clock_std_dev: "{{ vars.clock_std_dev }}"
+      confidence: true
+      max_iter: 3

data/mpox/clade-ii/1000/optimize.yaml

@@ -0,0 +1,20 @@
+# Branch-length optimization for a Nextstrain mpox build (mpox/clade-ii/1000).
+#
+# Run from the repository root:
+#   treetime optimize --config data/mpox/clade-ii/1000/optimize.yaml
+#
+# This is the date-free step that replaces `augur refine` in the nextstrain/mpox
+# `WIP/new_tt` workflow. It re-estimates every branch length and reroots on the
+# minimum-divergence branch. mpox divergence is counted in absolute mutations
+# (`divergence_units: mutations`), and indels are excluded from the branch-length
+# likelihood (`no_indels`), matching the pipeline's `treetime optimize` rule.
+
+tree: "data/mpox/clade-ii/1000/tree.nwk"
+alignment:
+  - "data/mpox/clade-ii/1000/aln.fasta.xz"
+
+reroot: "min-dev"
+divergence_units: "mutations"
+no_indels: true
+
+output_all: "tmp/examples/mpox/clade-ii/1000/optimize"

data/mpox/clade-ii/1000/pipeline.yaml

@@ -0,0 +1,65 @@
+# Nextstrain mpox build (clade IIb, hmpxv1), TreeTime part in one file (mpox/clade-ii/1000).
+#
+# Run from the repository root:
+#   treetime pipeline --config data/mpox/clade-ii/1000/pipeline.yaml
+#
+# Preview the whole plan, or resume after a failure, without recomputing earlier steps:
+#   treetime pipeline --config data/mpox/clade-ii/1000/pipeline.yaml --check
+#   treetime pipeline --config data/mpox/clade-ii/1000/pipeline.yaml --steps ancestral,timetree
+#
+# On the nextstrain/mpox `WIP/new_tt` phylogenetic workflow, three Snakemake
+# rules drive TreeTime: `treetime optimize` (replacing `augur refine`), then
+# `treetime ancestral` (folding in `augur translate`), then `treetime timetree`.
+# This file runs that same chain in one process.
+#
+# mpox is a slow-evolving DNA virus: divergence is measured in absolute mutations
+# (`divergence_units: mutations`), the clock rate is fixed rather than estimated
+# (`clock_rate` 5.7e-5 substitutions/site/year, with `clock_std_dev` for date
+# confidence), and indels are excluded from the branch-length likelihood
+# (`no_indels`). The clock constants live in `vars` as typed numbers, defined
+# once and reused. `{{ steps.optimize.outputs.nwk }}` carries the optimized tree
+# to the later steps with no hand-written path.
+
+vars:
+  data: "data/mpox/clade-ii/1000"
+  clock_rate: 5.7e-5
+  clock_std_dev: 2e-5
+
+output_all: "tmp/examples/mpox/clade-ii/1000/pipeline"
+
+steps:
+  # 1. Optimize branch lengths and reroot on the minimum-divergence branch.
+  - name: "optimize"
+    optimize:
+      tree: "{{ vars.data }}/tree.nwk"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+      reroot: "min-dev"
+      divergence_units: "mutations"
+      no_indels: true
+
+  # 2. Reconstruct ancestral sequences and mutations on the optimized tree.
+  - name: "ancestral"
+    ancestral:
+      tree: "{{ steps.optimize.outputs.nwk }}"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+      method_anc: "marginal"
+
+  # 3. Date the optimized tree with the fixed clock, keeping its root and polytomies.
+  - name: "timetree"
+    timetree:
+      tree: "{{ steps.optimize.outputs.nwk }}"
+      metadata: "{{ vars.data }}/metadata.tsv"
+      metadata_id_columns:
+        - "accession"
+      alignment:
+        - "{{ vars.data }}/aln.fasta.xz"
+      clock_rate: "{{ vars.clock_rate }}"
+      clock_std_dev: "{{ vars.clock_std_dev }}"
+      keep_root: true
+      keep_polytomies: true
+      divergence_units: "mutations"
+      no_indels: true
+      confidence: true
+      max_iter: 3

data/rsv/a/100/prune.yaml

@@ -9,12 +9,10 @@
 
 tree: "data/rsv/a/100/tree.nwk"
 alignment:
-  alignment:
-    - "data/rsv/a/100/aln.fasta.xz"
+  - "data/rsv/a/100/aln.fasta.xz"
 
 prune_short: 1.0e-6
 prune_empty: true
 merge_shared_mutations: true
 
-output:
-  output_all: "tmp/examples/rsv/a/100/prune"
+output_all: "tmp/examples/rsv/a/100/prune"

data/sc2/2844/ancestral.yaml

@@ -0,0 +1,18 @@
+# Marginal ancestral reconstruction on 2844 SARS-CoV-2 genomes (sc2/2844).
+#
+# Run from the repository root:
+#   treetime ancestral --config data/sc2/2844/ancestral.yaml
+#
+# A large single-command run: marginal reconstruction treats every site
+# independently, fits a GTR rate matrix from the data (the default
+# `model: infer`), and maps nucleotide mutations onto the tree. The sparse
+# representation (the default) stores only variable sites, which keeps a
+# genome-length SARS-CoV-2 alignment tractable.
+
+tree: "data/sc2/2844/tree.nwk"
+alignment:
+  - "data/sc2/2844/aln.fasta.xz"
+
+method_anc: "marginal"
+
+output_all: "tmp/examples/sc2/2844/ancestral"