|
| 1 | +# v17 comprehensive audit — 2026-04-21 |
| 2 | + |
| 3 | +A full "ship-ready" pass: docs consistency (live docs only), walkthrough |
| 4 | +numbers vs committed outputs, all 17 shipped HTML reports, hands-on |
| 5 | +Python API / MCP / error-message / device-detection exercise. |
| 6 | + |
| 7 | +## Method |
| 8 | + |
| 9 | +- **Docs sweep** — read every `README.md` / `docs/*.md` / `scripts/README.md` |
| 10 | + / `AUDIT_PROMPT.md` / MCP server tool descriptions. Cross-checked every |
| 11 | + track count and oracle spec against what the library actually returns |
| 12 | + (`create_oracle(...).sequence_length`, `.get_track_info()`, |
| 13 | + `len(ag_meta._track_index_map)`). |
| 14 | +- **Walkthroughs** — cross-checked numbers in each `README.md` against its |
| 15 | + companion `example_output.md` / `.json`. Accepted ±0.005 drift as CPU |
| 16 | + non-determinism, flagged larger gaps. |
| 17 | +- **HTML reports** — screenshotted all 17 `examples/walkthroughs/**/*.html` |
| 18 | + at 1400×2800 via headless Chrome and eye-checked each one for label |
| 19 | + leaks, formula drift, missing sections, or broken layout. |
| 20 | +- **Python API** — exercised `create_oracle` with good/bad inputs, |
| 21 | + `ModelNotLoadedError` path, invalid oracle name, empty oracle name. |
| 22 | +- **MCP server** — imported `chorus.mcp.server`, walked its registered |
| 23 | + tool list (22 tools) via the internal FastMCP registry. |
| 24 | +- **Device/platform** — `detect_platform()` on macOS arm64 returns |
| 25 | + `key=macos_arm64, has_cuda=False`. Each oracle env handles its own |
| 26 | + backend (MPS/TF Metal/JAX-CPU), which is the intended design. |
| 27 | + |
| 28 | +## Fixed in this PR |
| 29 | + |
| 30 | +1. **`chorus/mcp/server.py:77`** — AlphaGenome description said |
| 31 | + `"1-bp resolution across 5930 tracks"`. Real count is 5,731 (matches |
| 32 | + what `list_oracles()` prints and what the notebooks were fixed to in |
| 33 | + v16). Updated to `5,731`. |
| 34 | + |
| 35 | +2. **`chorus/mcp/server.py:1482`** — the MCP system prompt that the |
| 36 | + server ships to the AI assistant said |
| 37 | + `"AlphaGenome: 1Mb window, 5930 tracks, 1bp resolution"`. Same fix: |
| 38 | + `5,731`. Also added the explicit `7,611 tracks` count for Borzoi on |
| 39 | + the adjacent line so a new Claude session using Chorus via MCP gets |
| 40 | + consistent numbers. |
| 41 | + |
| 42 | +3. **`chorus/mcp/server.py:71`** — LegNet `"input_size_bp": 230` was |
| 43 | + stale. `create_oracle('legnet').sequence_length` returns 200. Updated |
| 44 | + to 200. |
| 45 | + |
| 46 | +4. **`scripts/README.md:37, 94`** — Borzoi `"7,612"` in the background |
| 47 | + build-script table and the output directory example. Real count from |
| 48 | + `get_track_info()` summed across assay types is 7,611. Fixed both. |
| 49 | + |
| 50 | +5. **`examples/walkthroughs/validation/SORT1_rs12740374_with_CEBP/README.md:21`** — |
| 51 | + "CEBPB binding gain: +0.22 (moderate)" but the committed |
| 52 | + `example_output.md` line 31 shows `+0.270`. 0.05 gap is beyond CPU |
| 53 | + non-determinism. Corrected to `+0.27`. |
| 54 | + |
| 55 | +## What else was checked and is clean |
| 56 | + |
| 57 | +- **Every other number** in every walkthrough README is either an exact |
| 58 | + match with the committed `example_output.md` or within ±0.006 (accept |
| 59 | + as CPU non-det). |
| 60 | +- **Formula labels** everywhere: chromatin/TF/histone/TSS rows carry |
| 61 | + `log2FC`; gene-expression rows carry `lnFC`; MPRA rows carry |
| 62 | + `Δ (alt−ref)`. Zero rogue labels. |
| 63 | +- **Links** — no live file references the old `examples/applications/` |
| 64 | + path (only `audits/` snapshots do, which is intentional). |
| 65 | +- **All 17 HTMLs** render with the glossary, formula chips, consensus |
| 66 | + matrix, per-layer tables, and IGV browser placeholder (IGV itself is |
| 67 | + client-side JS and doesn't render in headless file:// — not a bug). |
| 68 | +- **Error messages**: |
| 69 | + - `create_oracle('fakeOracle')` → `ValueError: Unknown oracle: fakeoracle. Available: ['enformer', 'borzoi', 'chrombpnet', 'sei', 'legnet', 'alphagenome']` ✓ |
| 70 | + - `predict()` without `load_pretrained_model()` → clear |
| 71 | + `ModelNotLoadedError: Model not loaded. Call load_pretrained_model first.` ✓ |
| 72 | +- **Platform detection** correctly identifies macOS arm64 and falls |
| 73 | + back to per-oracle-env device selection. No bogus CUDA detection |
| 74 | + attempts on Mac. |
| 75 | +- **MCP tool registry** — 22 tools all named consistently with the |
| 76 | + walkthrough READMEs. No dead / renamed tools. |
| 77 | + |
| 78 | +## Known latent issues, NOT fixed here |
| 79 | + |
| 80 | +- **Off-by-one in `predict_variant_effect` ref-allele check** |
| 81 | + (`chorus/core/base.py:322-328`) — tracked in |
| 82 | + `audits/2026-04-21_v16_notebook_and_html_audit.md`. Still needs a |
| 83 | + focused code PR with SNV + indel + reverse-strand coverage. |
| 84 | +- **NumExpr "safe limit of 16"** warning on every chorus import. This is |
| 85 | + a numexpr / matplotlib upstream thing, not a Chorus bug. |
| 86 | +- **Notebook shipped outputs contain `/srv/local/lp698/...` paths** — |
| 87 | + cosmetic, author's machine. |
| 88 | + |
| 89 | +## Test status |
| 90 | + |
| 91 | +`pytest tests/ --ignore=tests/test_smoke_predict.py -q` → 333 passed / |
| 92 | +1 skipped on this branch. |
0 commit comments