Add `keep_all` option for `additional_inputs` and `sequence_source` column

[jbloom]

Add keep_all option for additional_inputs and sequence_source column

Rationale

This is for a use case @CSimonich and I have: we are provided a set of non-public sequences and we want to specify them as additional inputs but keep all of the additional sequences plus subsample Pathoplexus sequences for context.

I made a pull request addressing this in #123

Summary

This PR modifies how additional_inputs work in the RSV build:

1. keep_all: True option: Each entry in additional_inputs can now optionally include keep_all: True to force-include all sequences from that input, bypassing subsampling. Without keep_all (or with keep_all: False), additional inputs are subsampled alongside the default Pathoplexus inputs — preserving the previous default behavior.

2. sequence_source metadata column: When any additional_inputs are defined, a sequence_source column is added to metadata labeling each sequence with its input name or "Pathoplexus". This enables coloring/filtering by origin in Auspice.

Config examples

Default (subsampled)

additional_inputs:
  - name: bloom_lab
    metadata: "data/custom/{a_or_b}_metadata.tsv"
    sequences: "data/custom/{a_or_b}_F.fasta"

Sequences are subsampled normally. A sequence_source column is still added.

Force-include all sequences

additional_inputs:
  - name: bloom_lab
    metadata: "data/custom/{a_or_b}_metadata.tsv"
    sequences: "data/custom/{a_or_b}_F.fasta"
    keep_all: True

All sequences bypass subsampling via augur filter --include. Note that keep_all sequences also bypass quality filters (coverage, missing data threshold, etc.), so they don't need Nextclade QC columns.

Mixed

additional_inputs:
  - name: priority_seqs
    metadata: "data/priority/{a_or_b}_metadata.tsv"
    sequences: "data/priority/{a_or_b}_F.fasta"
    keep_all: True
  - name: other_seqs
    metadata: "data/other/{a_or_b}_metadata.tsv"
    sequences: "data/other/{a_or_b}_F.fasta"

Implementation details

Files changed

File	Change
workflow/snakemake_rules/merge_inputs.smk	Allow keep_all key for additional_inputs; validate separately from inputs
workflow/snakemake_rules/sequence_source.smk	Only force-include accessions from keep_all: True inputs; add sequence_source for all additional inputs; validate keep_all is boolean
workflow/snakemake_rules/core.smk	Rename custom_include → additional_include
config/auspice_config.json	sequence_source coloring and filter (from prior commit, unchanged)
README.md	Document keep_all option, metadata requirements, sequence_source column

How it works

1. merge_inputs.smk merges all inputs (default + additional) into combined metadata/sequences as before
2. sequence_source.smk reads all additional input FASTAs to build an accession→source mapping, adds sequence_source column to metadata
3. sequence_source.smk generates additional_include.txt containing only accessions from keep_all: True inputs (empty file otherwise)
4. core.smk passes additional_include.txt to augur filter --include, which force-includes those accessions during subsampling

Backward compatibility

• Existing configs with additional_inputs but no keep_all behave identically to before (subsampled)
• The only new behavior for such configs is the addition of the sequence_source column
• Configs without additional_inputs are completely unaffected

Note on sequence_source in auspice_config.json

The sequence_source coloring (line 46) and filter (line 127) are always present in config/auspice_config.json, even though the sequence_source metadata column only exists when additional_inputs are defined. This is intentional: Auspice silently ignores colorings and filters that reference columns not present in the metadata. This means:

• With additional_inputs: the "Sequence source" coloring and filter appear in Auspice as expected.
• Without additional_inputs: the entries in auspice_config.json are harmlessly ignored; the user sees no difference.

The alternative would be to dynamically generate auspice_config.json based on whether additional_inputs are present, but that adds complexity for no functional benefit. If the Nextstrain team prefers a cleaner config that doesn't reference potentially-absent columns, this could be revisited.

[victorlin]

Thanks for the issue and PR! +1 for the two features described in the issue summary. As for the PR, I believe there are simpler solutions available that avoid additional complexity in the workflow graph.

1. keep_all: True option can be easily implemented with existing features in Augur. I've pushed a draft to branch victorlin/keep_all. Can you check to see if that works for your use case?

2. sequence_source metadata column is not as straightforward as it seems because the input merging allows the same sequence to be present across multiple inputs, for example to support gathering of metadata columns from various sources. @jameshadfield has proposed some solutions for this in nextstrain/augur#1965 (comment).

[jbloom]

@victorlin, yes, I think your proposed solution here would work well and I agree it is much easier!

And I see your point about the potential for duplicate sequence_source. In practice in my use case this would not be a problem as I'm merging non-public data with public data, but I can see how you would like to have the code be able to hande the possibility.