Influx_Pipeline

This workflow executes barcode cleaning and anchored alignment using Cutadapt, sums per sample barcodes, maps back to metadata, calculates molecular equivalence, models viral titration curves, determines percent neutralization, and visualizes 4 parameter logistic neutralization plots. Designed to work on the JHU RockFish HPC system.

Overview

This repository is organized so that code, configuration, metadata, raw data, processed outputs, and cluster execution settings are kept separate. The workflow is written in Snakemake and is designed to run on RockFish with SLURM using a Conda environment located inside the pipeline directory.

At a high level, the pipeline does the following:

1. Cleans and normalizes the barcode reference file.
2. Creates a 5′-anchored barcode FASTA for Cutadapt.
3. Runs Cutadapt per sample to count barcode matches at the 5′ end.
4. Builds per-sample QC tables and an all-sample QC summary.
5. Aggregates barcode counts across samples.
6. Merges counts back into the annotation sheet.
7. Computes molecular equivalence values using the configured spike-in columns.
8. Fits virus titration line models plate-wise and generates PDF summaries.
9. Computes percent neutralization values from the line fits.
10. Fits 4PL curves and generates final CSV and PDF outputs for pmAb and serum analyses.

---

Directory layout

The layout of the InFlux database looks like this:

/data/hlarman1/InfluxDB/
├── Influx_pipeline/
│   ├── workflow/
│   │   └── Snakefile
│   ├── config/
│   │   └── config_rockfish.yaml
│   ├── envs/
│   │   ├── analysis.yaml
│   │   └── cutadapt.yaml
│   ├── profiles/
│   │   └── rockfish/
│   │       └── config.yaml
│   ├── logs/
│   ├── benchmarks/
│   ├── containers/
│   ├── .conda/
│   │   └── influx_env/
│   └── run_Influx_pipeline.sh
├── Metadata/
│   ├── 073025_influxv2-54plex.txt
│   └── InFlux014-1_Annotation.csv
├── Rawdata/
│   └── <run_folder>/
└── ProcessedData/
    └── <run_folder>_<YYYYMMDD>_out/

What goes where

• Influx_pipeline/workflow/

  • Contains the Snakemake workflow.
  • Main file: Snakefile

• Influx_pipeline/config/

  • Contains workflow configuration files.
  • config_rockfish.yaml is the main cluster config used on RockFish.

• Influx_pipeline/envs/

  • Stores environment recipes and related software notes.
  • The pipeline currently runs from a shared project Conda environment.

• Influx_pipeline/profiles/rockfish/

  • Snakemake profile for SLURM submission on RockFish.
  • Controls job submission behavior, default memory/runtime, and thread overrides.

• Influx_pipeline/logs/

  • Slurm controller logs and child job logs.

• Influx_pipeline/benchmarks/

  • Reserved for future Snakemake benchmark outputs if benchmarking is added.

• Influx_pipeline/containers/

  • Reserved for future container-based deployment if needed.

• Influx_pipeline/.conda/influx_env/

  • Shared project Conda environment used to run the pipeline.

• Influx_pipeline/run_Influx_pipeline.sh

  • Slurm wrapper script that launches the Snakemake controller itself on RockFish.
  • Recommended way to run the pipeline on the cluster.

• Metadata/

  • Stores shared static reference inputs such as:
    • barcode reference file
    • annotation sheet

• Rawdata/

  • Stores input FASTQ folders for each InFlux run.

• ProcessedData/

  • Stores outputs for each run.
  • Output folders are typically named as:
    • <fastq_folder_name>_<YYYYMMDD>_out

---

Main workflow files

workflow/Snakefile

This is the core workflow. The main rules are:

• sanitize_barcodes

  • Reads the barcode reference file.
  • Removes BOM artifacts and formatting issues.
  • Converts sequences to uppercase.
  • Writes barcodes_clean.fa.

• anchor_barcodes

  • Converts cleaned barcode sequences into a 5′-anchored FASTA.
  • Writes barcodes_anchored.fa.

• cutadapt_e

  • Runs Cutadapt per sample for each configured error-rate label.
  • Produces JSON reports and .done sentinel files.

• qc_per_sample_eps

  • Computes per-sample QC statistics using FASTQ read counts and Cutadapt JSON output.

• qc_summary_all

  • Combines all per-sample QC files into a single summary TSV.

• aggregate_counts_e

  • Aggregates barcode counts per sample into a single counts matrix.

• counts_to_annotated_e

  • Merges the count matrix into the annotation sheet.
  • Adds QC metrics and spike_in_fraction.

• annotated_to_me_e

  • Converts raw counts to molecular equivalence values using the configured spike-in columns.

• titration_fit_platewise_e

  • Performs plate-wise virus titration line fitting.
  • Produces CSV summaries and PDF visualizations.

• percent_nt_by_fraction_e

  • Computes percent neutralization values from the line-fit outputs.

• nt_fourpl_fit_e

  • Fits 4-parameter logistic neutralization curves.
  • Produces final CSV and PDF outputs for pmAb and serum.

---

Configuration

config/config_rockfish.yaml

This file defines the pipeline inputs and run parameters.

Key parameters include:

• barcodes_fa

  • Path to the barcode FASTA/TXT reference file.

• fastq_dir

  • Directory containing input FASTQ files.

• fastq_glob

  • File patterns used to discover FASTQ files.

• out_dir

  • Output directory for all results.

• threads_per_job

  • Number of threads used by each Cutadapt job.
  • Snakemake-level parallelism is controlled separately by the Slurm profile.

• error_rate

  • Maximum tolerant error rate for barcode matching.

• error_rates

  • Explicit list of error rates to run.

• overlap

  • Minimum 5′ overlap required for barcode matching.

• no_indels

  • Whether to disallow indels in the match.

• match_read_wildcards

  • Whether to allow N in reads to match any base.

• revcomp

  • Whether to also search the reverse complement.

• annotated_sheet

  • Path to the sample annotation sheet.

• pcr_spike_in_col

  • Column in the annotation sheet that names the spike-in barcode column to use.

• pcr_spike_in_copies_col

  • Column in the annotation sheet containing copies per reaction.

• beta_precision, censor_eps, multistart, profile_steps

  • Parameters used in the final 4PL fitting stage.

---

Software environment

The workflow currently runs from a shared project Conda environment located at:

/data/hlarman1/InfluxDB/Influx_pipeline/.conda/influx_env

This environment was created to avoid system-level dependency issues on RockFish.

Example packages used by the workflow include:

• Python 3.10
• Snakemake 7.6.0
• Cutadapt 5.2
• NumPy
• pandas
• SciPy
• matplotlib

---

Running the pipeline

Option 1: Use the SLURM wrapper (I Recommend this)

Submit the wrapper script:

sbatch /data/hlarman1/InfluxDB/Influx_pipeline/run_Influx_pipeline.sh

The wrapper script behaviour:

• defines the input FASTQ folder
• derives the output folder name from the input folder plus the current date
• activates the shared project environment by using the environment's bin/ path directly
• launches Snakemake with the RockFish profile

---

Option 2: Direct Snakemake command

If you want to run the controller manually from an interactive session, the base command is:

snakemake -p \
  -s workflow/Snakefile \
  --configfile config/config_rockfish.yaml \
  --profile profiles/rockfish

---

Slurm profile

The RockFish profile is stored under:

profiles/rockfish/config.yaml

It controls:

• the maximum number of active Snakemake jobs
• default per-job memory
• default per-job runtime
• thread overrides for specific rules such as cutadapt_e
• the Slurm submission string used for child jobs

This design lets the workflow logic stay mostly unchanged while cluster-specific behavior is controlled outside the Snakefile.

---

Expected outputs

Inside each run-specific output folder, you should expect files such as:

• barcodes_clean.fa
• barcodes_anchored.fa
• json_e/e0/*.json
• counts_e0.tsv
• qc/*.qc.tsv
• qc_summary_all.tsv
• annotated_counts_e0.csv
• Molecular_equivalence_e0.csv
• VirusTitration_LineFits_platewise_e0.csv
• VirusTitration_PlateWise_e0.pdf
• VirusTitration_PlateWise_split_e0.pdf
• percent_NT_by_fraction_e0.csv
• vt100_denominators_e0.csv
• NT_4PL_fits_e0_pmAb.csv
• NT_4PL_fits_e0_serum.csv
• NT_4PL_plots_e0_pmAb.pdf
• NT_4PL_plots_e0_serum.pdf
• NT_4PL_plots_e0_pmAb_1up.pdf
• NT_4PL_plots_e0_serum_1up.pdf

---

Logging

There are two main logging layers:

1. Controller logs

These come from the SLURM job that runs run_Influx_pipeline.sh and contains log info of thr overall Snakemake orchestration. This is the best place to look for an overall log information.

Examples:

• logs/controller-<jobid>.out
• logs/controller-<jobid>.err

2. Child Slurm job logs

Each rule instance submitted by Snakemake can also write Slurm stdout/stderr logs and these logs are useful for debugging failed Cutadapt jobs for example.

---

Resuming after interruption

Snakemake is output-aware. If a run is interrupted, rerunning the same command will usually resume from completed outputs rather than restarting from scratch.

Example:

sbatch /data/hlarman1/InfluxDB/Influx_pipeline/run_Influx_pipeline.sh