Merge pull request #73 from AyushSemwal/annot_demux_refactor

* Default annotation model switched from 10x3p_sc_ont_013 (Conv+CRF) to 10x3p_sc_ont_016 (CNN-BiLSTM-CRF, 3 conv × 64 filters, 1 BiLSTM × 32 units). Empirically validated to ~525 kb on 4× L40S.
* GPU stability hardening — bounded join→SIGTERM→SIGKILL worker shutdown to prevent NCCL ring corruption across jobs, predict_with_backoff now raises on stale model after K.clear_session(), XLA disabled (tf2crf Viterbi incompatibility on TF 2.15), CUDA_VISIBLE_DEVICES no longer blanked when no GPUs found, model state persisted across length bins.
* QC enhancements — new --gene-body-bed flag accepting RSeQC-style BED12 (replaces auto-extraction from GTF); single-pass BAM scan is significantly faster.
* Training/simulation — new --min-flank/--max-flank for terminal cDNA flanks, separate from interior --min/max-spacer; pre-flight read-length math corrected in assess-model.
* Observability — every pipeline stage reports peak memory + elapsed time; logs and artifacts include the model name.
* Bug fixes — barcode-correct falls back to bundled seq_orders.yaml when only --model-name is given; extract_annotated_seqs no longer crashes on empty Starts.
* Documentation — resource_requirements.qmd re-anchored on _016 (bpt 6,644 → 2,916, k 73.4 → 32.2 KB/bp); new --max-batch-size per-VRAM-tier tuning guidance; XLA references removed; tentative-figures disclaimer added pending per-GPU benchmarking.

Author: AyushSemwal

.github/workflows/test.yml

@@ -37,11 +37,12 @@ jobs:
         shell: bash -l {0}
         run: |
          mkdir -p tests/10x3p/data tests/references
-         curl -L "https://www.dropbox.com/scl/fi/k18rs7icrw7la7ogoi9bh/10x3p_sc_ont_013.h5?rlkey=ssw6ylhpvwe3e96jmwm0j3pzg&st=5byhrx7t&dl=0" -o models/10x3p_sc_ont_013.h5
-         curl -L "https://www.dropbox.com/scl/fi/2plk01thb9q1o8gqgmgks/10x3p_sc_ont_013_params.yaml?rlkey=twtqwmpsyv6ecqeh7a2c5k6tx&dl=0" -o models/10x3p_sc_ont_013_params.yaml
-         curl -L "https://www.dropbox.com/scl/fi/inz070k8k7kfpibnj9lkt/10x3p_sc_ont_013_lbl_bin.pkl?rlkey=pwhonyomb3i4d13yjat8cxpro&dl=0" -o models/10x3p_sc_ont_013_lbl_bin.pkl
+         curl -L "https://www.dropbox.com/scl/fi/gmmifxp964fe66qvv3nfk/10x3p_sc_ont_016.h5?rlkey=vqqob3hxrqsuq8540su1aw985&st=bwk2kblf&dl=0" -o models/10x3p_sc_ont_016.h5
+         curl -L "https://www.dropbox.com/scl/fi/7almukao9vykewf4p73eg/10x3p_sc_ont_016_params.yaml?rlkey=ge8fhouxvvya3umlzryk8mvtq&st=zn01vxqc&dl=0" -o models/10x3p_sc_ont_016_params.yaml
+         curl -L "https://www.dropbox.com/scl/fi/4wvup62v1sgh7v84899ud/10x3p_sc_ont_016_lbl_bin.pkl?rlkey=m0xt4vzdpfc0ztj91zw93ko8p&st=y5skyjsq&dl=0" -o models/10x3p_sc_ont_016_lbl_bin.pkl
          curl -L "https://www.dropbox.com/scl/fi/9203pc2gz45u419kze446/hg38_gencode_chr21.fa?rlkey=k1co80pso7zsvb8oiqqc1o1dy&st=e0e92h5h&dl=1" -o tests/references/hg38_gencode_chr21.fa
          curl -L "https://www.dropbox.com/scl/fi/9zvh26moro7frfho818ko/hg38_gencode_chr21.gtf?rlkey=v3s05m9ap3srpjhvl5yfjfeq7&st=hn6mufoq&dl=0" -o tests/references/hg38_gencode_chr21.gtf
+         curl -L "https://www.dropbox.com/scl/fi/98ykmaori9zrqf0q1qn36/hg38.HouseKeepingGenes.bed.gz?rlkey=dqjabcby9b9owso30njfpauyt&st=zmvxk4pp&dl=0" -o tests/references/hg38.HouseKeepingGenes.bed.gz
 
       - name: Run Pytest with coverage
         shell: bash -l {0}

docs/webpages/model_training/assess_model.qmd

@@ -146,6 +146,8 @@ pannable plots with hover tooltips. This is the primary artifact for reviewing m
 | `--transcriptome` | TEXT | None | Transcriptome FASTA for realistic cDNA | Recommended for realistic assessment |
 | `--min-spacer` | INT | 0 | Minimum spacer length between concatenated fragments | Adjust if your data has inter-fragment cDNA |
 | `--max-spacer` | INT | 50 | Maximum spacer length between concatenated fragments | Adjust if your data has inter-fragment cDNA |
+| `--min-flank` | INT | 0 | Minimum length of random cDNA flank at each terminal end of a read | Rarely changed |
+| `--max-flank` | INT | 50 | Maximum length of random cDNA flank at each terminal end of a read | Raise (e.g. 200–300) to cover real ONT adapter-flank lengths and reduce edge false positives |
 | `--max-trunc-5p` | INT | 0 | Max 5' truncation (bp) | Enable to test truncated read handling |
 | `--max-trunc-3p` | INT | 0 | Max 3' truncation (bp) | Enable to test truncated read handling |
 | `--threads` | INT | 2 | CPU threads | Increase for faster simulation |
@@ -162,7 +164,7 @@ pannable plots with hover tooltips. This is the primary artifact for reviewing m
 
 ```bash
 tranquillyzer assess-model \
-    10x3p_sc_ont_013 \
+    10x3p_sc_ont_016 \
     models/ \
     assessment_output/ \
     --num-reads 500 \

docs/webpages/model_training/simulate_data_cli.qmd

@@ -29,6 +29,10 @@ For background on how reads are simulated and why, see [Read Structure Simulatio
 | `--rc` / `--no-rc` | FLAG | `--rc` | Include reverse complements (doubles the training set) | Disable only if your protocol has a fixed known orientation |
 | `--transcriptome` | TEXT | None | Transcriptome FASTA for realistic cDNA generation | Recommended — improves model accuracy on real data |
 | `--invalid-fraction` | FLOAT | 0.3 | Fraction of reads generated as structurally invalid artifacts | Increase if your data has many malformed reads |
+| `--min-spacer` | INT | 0 | Minimum length of random cDNA spacer between concatenated fragments | Rarely changed |
+| `--max-spacer` | INT | 50 | Maximum length of random cDNA spacer between concatenated fragments | Raise to cover longer real chimeric junctions |
+| `--min-flank` | INT | 0 | Minimum length of random cDNA flank at each terminal end of a read | Rarely changed |
+| `--max-flank` | INT | 50 | Maximum length of random cDNA flank at each terminal end of a read | Raise (e.g. 200–300) to cover real ONT adapter-flank lengths and reduce edge false positives |
 
 # Example Usage
 

docs/webpages/model_training/train_model.qmd

@@ -50,8 +50,9 @@ Reads are converted to integer tokens before training:
 | T | 4 |
 | N | 5 |
 
-This mapping is governed by the `vocab_size` parameter (default: 5, for the five nucleotide symbols). The padding token
-(0) is used only to make reads within a batch the same length and does not represent a nucleotide.
+This mapping is governed by the `vocab_size` parameter (default: 6, covering the four DNA bases, the ambiguity
+symbol `N`, and the padding token). The padding token (0) is used only to make reads within a batch the same length
+and does not represent a nucleotide.
 
 ## Embedding Layer
 
@@ -79,7 +80,7 @@ The key parameters are:
 
 - **`conv_kernel_size`** (default: 25): how wide a motif each layer can detect at once. Larger kernels help detect
   longer protocol motifs and smooth local noise, but values that are too large may blur sharp boundary cues.
-- **`conv_filters`** (default: 128 per layer): the number of distinct motif detectors per layer. More filters means
+- **`conv_filters`** (default: 64 per layer): the number of distinct motif detectors per layer. More filters means
   more pattern capacity at the cost of memory and compute. If boundaries are being missed or adapters vary across
   reads, increasing filters is often more effective than adding more layers.
 - **`conv_layers`** (default: 3): how many convolutional stages are stacked. More layers allow hierarchical motif
@@ -109,7 +110,7 @@ sequence in order and retaining information over long distances.
 _Tranquillyzer_ uses a **bidirectional** LSTM by default, meaning it reads the sequence both left-to-right and
 right-to-left. This allows each position to be interpreted using context from both sides.
 
-- **`lstm_units`** (default: 96): the size of the LSTM's internal memory. Increasing units improves the model's ability
+- **`lstm_units`** (default: 32): the size of the LSTM's internal memory. Increasing units improves the model's ability
   to track long-range structure but increases compute and can overfit.
 - **`lstm_layers`** (default: 1): stacking multiple LSTMs increases modeling power but is often unnecessary for
   standard protocols.

docs/webpages/model_training/train_model_cli.qmd

@@ -55,14 +55,14 @@ parameter values. If you want to search over multiple values for a parameter, pr
 10x3p_sc_ont:
   batch_size: 128
   train_fraction: 0.8
-  vocab_size: 5
+  vocab_size: 6
   embedding_dim: 128
   conv_layers: 3
-  conv_filters: [128, 128, 128]
+  conv_filters: [64, 64, 64]
   conv_kernel_sizes: [25, 25, 25]
-  dilation_rates: [1, 3, 5]
+  dilation_rates: [1, 1, 1]
   lstm_layers: 1
-  lstm_units: [96]
+  lstm_units: [32]
   bidirectional: true
   crf_layer: true
   attention_heads: 0
@@ -100,14 +100,14 @@ other _Tranquillyzer_ commands.
 10x3p_sc_ont:
   batch_size: 128
   train_fraction: 0.8
-  vocab_size: 5
+  vocab_size: 6
   embedding_dim: 128
   conv_layers: 3
-  conv_filters: [128, 128, 128]
+  conv_filters: [64, 64, 64]
   conv_kernel_sizes: [25, 25, 25]
-  dilation_rates: [1, 3, 5]
+  dilation_rates: [1, 1, 1]
   lstm_layers: 1
-  lstm_units: [96]
+  lstm_units: [32]
   bidirectional: true
   crf_layer: true
   attention_heads: 0

docs/webpages/pipeline/annotation.qmd

@@ -26,7 +26,7 @@ whitelist-free workflow):
 
 ```bash
 tranquillyzer annotate-reads \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     --threads 12 \
     OUTPUT_DIR
@@ -38,7 +38,7 @@ Use this when you have a whitelist and want the fastest path to demultiplexed re
 
 ```bash
 tranquillyzer annotate-reads \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     --threads 12 \
     --run-barcode-correction \
@@ -69,7 +69,7 @@ automatically pick up from the last completed chunk. Progress is tracked in a ch
 |---|:--:|---|---|
 | `OUTPUT_DIR` | required | Base output directory | |
 | `WHITELIST_FILE` | optional | Barcode whitelist TSV | Required if using `--run-barcode-correction` |
-| `--model-name` | `10x3p_sc_ont_013` | Model to use for inference | Set to match your protocol |
+| `--model-name` | `10x3p_sc_ont_016` | Model to use for inference | Set to match your protocol |
 | `--seq-order-file` | `utils/seq_orders.yaml` | Library definition file | Only if using a custom file |
 | `--gpu-mem` | 12 GB | GPU memory budget | Always set to your actual VRAM |
 | `--target-tokens` | 1,200,000 | Token budget per GPU | See [Resource Requirements](resource_requirements.qmd) |

docs/webpages/pipeline/barcode_demux.qmd

@@ -73,7 +73,7 @@ tranquillyzer barcode-correct \
 | `INPUT_DIR` | required | Directory containing annotation_metadata/ | |
 | `WHITELIST_FILE` | required | TSV with barcode columns | |
 | `--output-dir` | `INPUT_DIR` | Where to write corrected output | Set if you want separate output |
-| `--model-name` | `10x3p_sc_ont_013` | Model for barcode column resolution | Match your annotation model |
+| `--model-name` | `10x3p_sc_ont_016` | Model for barcode column resolution | Match your annotation model |
 | `--bc-lv-threshold` | 2 | Max Levenshtein distance for fuzzy matching | Increase for noisier barcodes |
 | `--run-demux` | off | Demultiplex concurrently | Enable to get FASTA/FASTQ output |
 | `--output-fmt` | `fasta` | Demux format | Use `fastq` if quality scores available |
@@ -92,7 +92,7 @@ using count-based knee-point detection.
 
 ```bash
 tranquillyzer annotate-reads \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     OUTPUT_DIR
 ```
@@ -101,7 +101,7 @@ tranquillyzer annotate-reads \
 
 ```bash
 tranquillyzer generate-whitelist \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --expected-cells 5000 \
     OUTPUT_DIR
 ```
@@ -126,7 +126,7 @@ tranquillyzer generate-whitelist \
 | Option | Default | Description | When to change |
 |---|:--:|---|---|
 | `OUTPUT_DIR` | required | Annotation output directory | |
-| `--model-name` | `10x3p_sc_ont_013` | Model for barcode column resolution | Match your annotation model |
+| `--model-name` | `10x3p_sc_ont_016` | Model for barcode column resolution | Match your annotation model |
 | `--expected-cells` | None (auto) | Hint for expected number of cells | Provide if known for better knee detection |
 | `--min-cell-ratio` | 0.50 | Knee threshold as fraction of cliff-top count | Lower to include more cells; raise for stricter filtering |
 | `--min-reads-per-barcode` | 3 | Minimum reads for a barcode to be considered | Increase for noisier data |

docs/webpages/pipeline/qc_metrics.qmd

@@ -33,7 +33,7 @@ Providing an aligned BAM file (with CB and UB tags) enables:
 - **Unique UMIs per cell**
 - **Mapping rate per cell**
 - **Duplicate rate per cell**
-- **Gene body coverage** (also requires `--gtf`)
+- **Gene body coverage** (also requires `--gene-body-bed`)
 
 ## Tier 3: Gene Quantification Metrics (Requires `--counts-matrix` and `--gtf`)
 
@@ -66,9 +66,30 @@ tranquillyzer qc-metrics \
     --bam aligned_files/dup_marked.bam \
     --counts-matrix counts_matrix.tsv \
     --gtf gencode.v44.annotation.gtf \
+    --gene-body-bed hg38.HouseKeepingGenes.bed.gz \
     INPUT_DIR
 ```
 
+## Gene Body Coverage
+
+The gene body coverage plot follows the [RSeQC convention](https://rseqc.sourceforge.net/#genebody-coverage-py):
+the user supplies a curated BED12 (one transcript per line) via `--gene-body-bed`, and each line is treated
+independently in the pileup. No transcript is auto-picked from the GTF — supplying a curated BED keeps the
+percentile axis aligned with the canonical mRNA species and avoids artifacts from extended-UTR or retained-intron
+isoforms. Both plain and gzipped BED12 are accepted.
+
+Common sources for the BED:
+
+- **RSeQC housekeeping BEDs** ([SourceForge](https://sourceforge.net/projects/rseqc/files/BED/)) — pre-curated
+  single-isoform-per-gene sets for hg38, hg19, mm10, etc. The fastest path to a clean curve.
+- **MANE_Select export** — for human, the ~19,000 MANE_Select transcripts converted to BED12 (e.g. via UCSC
+  `gtfToGenePred` + `genePredToBed`).
+- **Custom curated set** — any BED12 the user trusts (e.g. APPRIS principal isoforms, project-specific genes
+  of interest).
+
+Without `--gene-body-bed`, the gene body coverage plot is omitted from the report (other QC sections are
+unaffected).
+
 # Command Line Options
 
 | Option | Default | Description | When to change |
@@ -81,7 +102,8 @@ tranquillyzer qc-metrics \
 | `--invalid-file` | auto-detect | Path to invalid annotations parquet | Only if non-standard location |
 | `--bam` | None | Coordinate-sorted BAM with CB/UB tags | Provide for saturation and alignment metrics |
 | `--counts-matrix` | None | featureCounts counts matrix TSV | Provide for gene-level QC |
-| `--gtf` | None | GTF annotation file | Required with `--counts-matrix`; also used for gene body coverage |
+| `--gtf` | None | GTF annotation file | Required with `--counts-matrix` |
+| `--gene-body-bed` | None | BED12 file (plain or `.gz`) for gene body coverage, RSeQC-style | Provide a curated single-transcript-per-gene BED (e.g. RSeQC `HouseKeepingGenes.bed.gz`, MANE_Select export) to enable the gene body coverage plot |
 | `--read-len-bin-width` | 100 | Bin width for read-length histograms | Decrease for finer resolution |
 
 # Output

docs/webpages/pipeline/visualization.qmd

@@ -20,14 +20,14 @@ without annotation.
 # Visualize N random reads
 tranquillyzer visualize \
     --num-reads 10 \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     OUTPUT_DIR
 
 # Visualize specific reads by name
 tranquillyzer visualize \
     --read-names read_001,read_002,read_003 \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     OUTPUT_DIR
 ```
@@ -40,7 +40,7 @@ Either `--num-reads` or `--read-names` must be provided.
 |---|:--:|---|
 | `OUTPUT_DIR` | required | Directory with preprocessed reads (writes to `plots/`) |
 | `--output-file` | `full_read_annots` | PDF filename prefix |
-| `--model-name` | `10x3p_sc_ont_013` | Model for inference |
+| `--model-name` | `10x3p_sc_ont_016` | Model for inference |
 | `--num-reads` | None | Number of random reads to visualize per Parquet file |
 | `--read-names` | None | Comma-separated list of specific read names |
 | `--gpu-mem` | 12 GB | GPU memory budget |

docs/webpages/quick_start.qmd

@@ -20,14 +20,14 @@ _Tranquillyzer_ ships with pre-trained models for common 10x Genomics protocols:
 |:--|:-:|:-:|
 | Batch Size | 128 | 128 |
 | Training Fraction | 0.8 | 0.8 |
-| Vocab Size | 5 | 5 |
+| Vocab Size | 5 | 6 |
 | Embedding Dimension | 128 | 128 |
 | Conv Layers | 4 | 3 |
-| Conv Filters | 128 | 128 |
+| Conv Filters | 128 | 64 |
 | Conv Kernel Size | 25 | 25 |
-| Dilation Rates | [1, 1, 1, 1] | [1, 3, 5] |
+| Dilation Rates | [1, 1, 1, 1] | [1, 1, 1] |
 | LSTM Layers | 1 | 1 |
-| LSTM Units | 96 | 96 |
+| LSTM Units | 96 | 32 |
 | Bidirectional | True | True |
 | CRF Layer | True | True |
 | Attention Heads | 0 | 0 |
@@ -51,7 +51,7 @@ tranquillyzer preprocess \
 
 # 2. Annotate reads + correct barcodes + demultiplex (single pass)
 tranquillyzer annotate-reads \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     --threads 12 \
     --run-barcode-correction \
@@ -92,14 +92,14 @@ tranquillyzer preprocess \
 
 # 2. Annotate reads (no whitelist)
 tranquillyzer annotate-reads \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --gpu-mem 48 \
     --threads 12 \
     /path/to/output
 
 # 3. Discover cell barcodes via knee-point detection
 tranquillyzer generate-whitelist \
-    --model-name 10x3p_sc_ont_013 \
+    --model-name 10x3p_sc_ont_016 \
     --expected-cells 5000 \
     /path/to/output