add wdl-genomics-on-ray template: WDL Genomics Workflow for bioinformatics - #951
Draft
em-anyscale wants to merge 4 commits into
Draft
add wdl-genomics-on-ray template: WDL Genomics Workflow for bioinformatics#951em-anyscale wants to merge 4 commits into
em-anyscale wants to merge 4 commits into
Conversation
- Introduced the `wdl-genomics-on-ray` template enabling WDL workflows to run on Ray, optimizing resource utilization on autoscaling clusters. - Added Dockerfile to build a custom Ray-compatible image, including necessary tools, libraries, and dependencies for the workflow. - Configured setup for both AWS and GCP environments with corresponding compute configurations (`aws.yaml`, `gce.yaml`). - Added README.ipynb notebook for quick start and detailed documentation on running WDL genomic workflows. - Added example workflows utilizing GIAB Ashkenazi trio data, demonstrating concurrent task execution via Ray’s autoscaling capabilities.
…stry, replacing divergence-derived gating
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs WDL (Workflow Description Language) workflows on Anyscale.
WDL is the language of GATK Best Practices, WARP and Terra, and a large share of
production bioinformatics is written in it. This template adapts the Broad Institute's ONT assembly pipeline [link, pinned to 02089d9] and ships a miniwdl container backend
that runs each WDL task as a Ray task, so tasks bin-pack onto the cluster instead of
each waiting on a VM boot.
wdl-genomics-on-raytemplate: a miniwdl container backend registeredthrough a Python entry point, selected with
[scheduler] container_backend = ray.runtime {}onto Ray resources, including Cromwell/GCE spellings (gpuCount,gpuType,disks), and routes Ray node loss to the WDL's ownruntime.preemptible.podman runfails at
container-init exec:none(cluster image),ray(per-taskimage_uri),miniwdl's own backends, and
native.tools/manifest.tomlpinning the toolchain by version, URL and sha256.aws.yaml,gce.yaml).README.ipynbexample walking the pipeline end to end on the GIAB Ashkenazi trio: three samples in 8m05s against 7m52s for the slowest single sample, on three workers.persist_outputs.py, because an Anyscale Job terminates its cluster on success and/mnt/cluster_storagegoes with it.