lrp_4.0.69

Adding a WDL to run DORADO for basecalling ONT data (#462)

New ONT basecalling workflow on Dorado 1.4.0, plus its container and Dockstore registration.

• wdl/pipelines/ONT/Preprocessing/ONTBasecallDorado.wdl (workflow Dorado): list
  fast5 under gcs_fast5_dir -> partition into shards -> scatter dorado basecaller
  (default model sup@latest); optional --reference alignment; optional barcode
  demux via dorado demux --kit-name (--no-classify when no kit). Emits per-barcode
  pass BAMs, a merged sequencing_summary.txt, and a final run summary; parses the
  basecall model from the BAM @rg header; finalizes outputs (incl. multi-barcode)
  under gcs_out_root_dir/<sample_name>/.
• docker/lr-dorado/{Dockerfile,Makefile}: us.gcr.io/broad-dsp-lrma/lr-dorado:1.4.0
  — Dorado 1.4.0 on nvidia/cuda:11.8.0-base-ubuntu22.04 (GPU) + samtools; image
  tag tracks the Dorado version.
• .dockstore.yml: register ONTBasecallDorado.

Runs on GPU nodes (CUDA base image). Follow-up: workflow meta.description still
reads "Guppy" (copied) and should be updated to Dorado.

---

Co-authored-by: James emeryj@broadinstitute.org
Co-authored-by: Jonn Smith jonn@broadinstitute.org
Co-authored-by: Jonn Smith jonn-smith@users.noreply.github.com
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

lrp_4.0.68

P. falciparum variant-calling: small-genome benchmarking + Pf7 reimplementation pipelines (#517)

Adds small-genome (P. falciparum) variant-calling validation tooling and hardens
the Pf callers against small/low-variance-callset failures found on Terra.

• New BenchmarkVCFsSmall: cost-optimized rtg-vcfeval SNP/indel benchmark for small
  genomes; per-indel-length-bin eval folded into the stratifier scatter; avoids the
  empty-VCF tabix crash via uncompressed intermediates + no index on the indel step.
• New Pf7 pipelines: Pf7JointGenotyping (cohort joint calling) and
  Pf7SingleSampleVariantCalling (VQSR made optional/best-effort so small callsets
  still emit gVCF + genotyped VCF).
• Pf7JointGenotyping fixes: bcftools_docker -> lr-basic (GNU coreutils; BusyBox lacked
  split -d); exclude organellar contigs from MakeIntervals via contigs_to_exclude
  (default API+MIT) so GenomicsDBImport stops rejecting short/padded organellar windows.
• Pf_Niare VQSR: expose snp_max_gaussians / indel_max_gaussians (default 4), previously
  hard-coded, so the SNP model can back off on non-converging per-contig callsets.
• Registers BenchmarkVCFsSmall, Pf7JointGenotyping, Pf7SingleSampleVariantCalling in
  Dockstore. All changed WDLs pass womtool validate + miniwdl check --strict.

lrp_4.0.67

BroadOnPrem pipeline 1: make VQSR best-effort (non-fatal) (#516)

• BroadOnPrem pipeline 1: make VQSR best-effort (non-fatal)

GATK VQSR hard-fails on small callsets (the INDEL negative model: "No data
found"), which failed the whole single-sample align+call workflow. Apply the
"VQSR optional" runbook so VQSR is attempted unchanged but, on failure, the
workflow still succeeds and delivers the gVCF and raw VCF; only the
recalibrated VCF is omitted.

• New task VariantRecalibratorBestEffort: byte-identical GATK commands to
  VariantRecalibrator, but with -e off, each gatk call guarded, partial
  VQSLOD outputs removed on failure (all-or-nothing), status written to
  vqsr_ok.txt, exit 0. Outputs: Boolean vqsr_ok + File? vcf.
• Pipeline 1 calls the best-effort task; the SortCompressIndex consumer of
  the recalibrated VCF is gated on vqsr_ok (inputs via select_first).
• Workflow outputs: recalibrated_vcf / _index are now File?; a Boolean
  vqsr_succeeded is exposed. raw_vcf and gvcf stay required (upstream of VQSR).

The shared VariantRecalibrator task and pipeline 2 are untouched (a new task
avoids changing pipeline 2's required-output contract). womtool + miniwdl clean.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

• Commented out workflows with missing WDLs.

---

Co-authored-by: Claude Opus 4.8 noreply@anthropic.com

lrp_4.0.66

Expose VQSR tuning knobs on VariantRecalibrator (#515)

Surface the VQSR model-fitting parameters as runtime inputs so small
callsets (where the negative model finds no variants below the default
badLodCutoff and fails with "No data found") can be tuned without editing
the WDL:

• snp_max_gaussians (default 8), indel_max_gaussians (default 4)
• min_num_bad_variants (--minNumBadVariants, default 1000)
• bad_lod_cutoff (--badLodCutoff, default -5.0)

Defaults reproduce the prior hardcoded behavior exactly (1000 / -5.0 are
GATK's own defaults). Threaded through
BroadOnPremMalariaPipeline_2_JointVariantCalling.

Co-authored-by: Claude Opus 4.8 noreply@anthropic.com

lrp_4.0.65

CLAUDE.md, Multi-sample VCF decomposition: split/subset workflows Terra table linking (#514)

Adds tooling to decompose joint-called VCFs into per-sample results and wire
them back into Terra data tables, plus supporting utilities and docs.

New workflows and tasks:

• SplitMultiSampleVCF: split a multi-sample VCF into per-sample VCFs. Optional
  sample subset via either an inline list (sample_names) or a file
  (sample_name_list), mutually exclusive with at most one provided; every
  requested sample is validated against bcftools query -l and all absent
  names are reported (stderr) before failing. Each per-sample .vcf.gz and its
  .tbi are delocalized into one directory so the index stays a sibling of its
  VCF (required by downstream GATK/htslib).
• SubsetVCFsToSamples + VariantUtils.SubsetVCFToSamples: subset one or more
  VCFs to a set of samples, emitting a single multi-sample VCF per input (no
  per-sample split) via bcftools view -S. Samples from an inline list or a
  file (exactly one required). error_if_sample_missing (default true) fails on
  any absent sample; when false it warns and continues with --force-samples.
• SplitMultiSampleVCFToTerraTable: split a joint VCF per sample, build a Terra
  entity TSV (vcf, vcf_index, 98_Run_ID, and Terra reference links
  97_Joint_Run_ID and Truth), validate that every link target exists, and
  upload it. joint_run_table/truth_table are auto-derived from the destination
  table name ([TEST_]joint_results_singlesample) or supplied explicitly.
  A ValidateTerraLinks step pulls the joint-run and truth tables and confirms
  the joint-run entity and every per-sample truth entity exist, hard-gating the
  upload. Calls follow the mandatory t_NN aliasing convention.
• TerraUtils.UploadDataTable: upload a Terra entity TSV via firecloud.api
  (flexible model, pet-service-account auth). namespace/workspace auto-detect
  (explicit inputs, then WORKSPACE_* env, then GOOGLE_PROJECT matched against
  the workspace list). require_existing_id and columns_must_exist guards
  (both pull the destination once and report all offending IDs/columns before
  failing).

PfalciparumTypeDrugResistanceMarkers:

• Optionally take a per-contig list of joint-call VCFs (Array[File],
  localization_optional) and stream the analyzed sample's genotypes at the
  drug-resistance loci over gs:// (no localization of the large joint sets),
  using the GVCF to fill no-call evidence. Unchanged when no joint VCFs given.

Also: BenchmarkVCFs.CheckForVariants symlinks the eval VCF's index next to the
VCF before GATK, so it works even when a VCF and index localize to separate
directories.

All WDL validated with womtool and miniwdl.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

lrp_4.0.64

Kvg flare (#513)

• Adopt Hang's FLARE workflow and extend prep for AoU LAI.

Cherry-pick the FlareLocalAncestryInference workflow from hangsu_phasing_new
(Phasing/Flare.wdl, hangsuunc/flare:v1) and replace the Utility/ implementation.
Add VCF prep, chr1 model reuse, GCS finalize, and lr-flare support scripts.

• Option to disable the Flare task itself to allow a run to just do pre-processing

Co-authored-by: Hang Su suhang@broadinstitute.org

lrp_4.0.63

Removed default runtime attr overrides for gnarly genotyper. (#512)

• Pulled out default runtime attrs from call to GnarlyGenotyper to expose them in the Terra UI. This enabled a large joint call with ~30000 Plasmodium falciparum samples from The Gambia and elsewhere.

lrp_4.0.62

Updated short read whole genome pipelines for scalability and speed, bugfixes (#511)

• Added simplified processing WDLs for SRFlowcell, SRWholeGenome, and SRJointCallGVCFsWithGenomicsDB.

• Added new wdls to dockstore.

• Fixed issue in FASTQ output (fastq2 not properly set).

----- Details below -----

• Added SRFlowcell_Simplified.wdl which removes two localization steps
  by combining sorting directly into main operations.

• Added SRWholeGenome_Simplified.wdl which uses a simplified version
  of the HaplotypeCaller workflow to run faster. This was accomplished
  by removing the scatter by interval. This workflow will not scatter
  by interval when performing variant calling. This is OK for small
  genomes / organisms.

• Added HaplotypeCaller_Simplified.wdl which performs germline variant
  calling on an input bam file without any scattering.

• Added SRJointCallGVCFsWithGenomicsDB_simplified.wdl

lrp_4.0.61

Minor bug fixes and updates to make the malaria comparison pipelines run to completion on the new test sets (#510)

• Updated ReblockGVCFs docker image to broadinstitute/gatk-nightly:2025-08-29-4.6.2.0-17-g2a1f41bf3-NIGHTLY-SNAPSHOT to fix exception with certain spanning deletions.
• Updates to malaria workflows for comparisons (BroadOnPremMalariaPipeline_2_JointVariantCalling.wdl, SRJointCallGVCFsWithGenomicsDB_Pf_Niare_VETS.wdl, SRJointCallGVCFsWithGenomicsDB_Pf_Niare_VQSR.wdl, BroadOnPremMalariaPipelineTasks.wdl)
• Fixed bug in SplitContigToIntervals that caused resulting bed files to sort out of order.
• Added SplitMultiSampleVCF.wdl which splits a multi-sample VCF into an array of single-sample VCFs using bcftools +split
• Added SplitMultiSampleVCF.wdl to dockstore.
• Added bcftools plugins to lr-basic docker image.
• Updated lr-basic to v0.1.3

lrp_4.0.60

Various bug fixes for the WDLization of old on-prem malaria variant calling pipelines (#509)

• Fixed read decontamination in BroadOnPremMalariaPipeline_1
• Added ChangeReadGroup task to Utils.wdl
• Fixed small bug in bwa-mem2 read group parsing.
• Fixing gvcf extension in on-prem wdl.
• Added program group hack / fix for on-prem pipeline.