Harden eQTL evidence semantics and refresh site

Author: smkwray

README.md

@@ -10,23 +10,24 @@ chrXatlas is a chromosome X evidence atlas built from publicly available Pan-UK
 
 ---
 
-## Current build (2026-03-18)
+## Current build (2026-03-19)
 
 | Metric | Value |
 |--------|-------|
 | Curated panels | 3 (Broad Atlas, Mind & Risk, Biochemistry Deep Dive) |
 | Curated traits | 79 across panels |
 | Chromosome X loci mapped | 1,109 |
-| eQTL support rate | 97% of curated traits |
+| eQTL lookup-hit coverage | 77 of 79 curated traits |
+| Strict eQTL-supported traits | 62 of 79 curated traits |
 | Biological domains | 26 unique |
 | Discovery pool | 150 traits, 643 loci |
 
 ### Curated panels
 
-- **Broad Atlas** — 48 traits, 751 loci, 98% eQTL-supported. The featured all-purpose panel with broad trait coverage across 21 biological domains.
-- **Mind & Risk** — 10 traits, 26 loci, 90% eQTL-supported. Focused behavior and cognition panel: risk-taking, smoking, alcohol, mood, irritability, reaction time, neuroticism, insomnia, depression, fluid intelligence.
-- **Biochemistry Deep Dive** — 21 traits, 332 loci, 100% eQTL-supported. High-yield blood biochemistry panel covering lipoproteins, kidney markers, endocrine biomarkers, minerals, and liver enzymes.
-- **Discovery Pool** — 150 traits, 643 loci, 48% eQTL-supported. The discovery pool is built from the full Pan-UKB max independent set filtered to `num_pops_pass_qc >= 2`, so the broad build starts from non-redundant traits with at least minimal multi-population QC rather than the noisiest single-population results. It is kept as a curation reservoir, not a featured panel.
+- **Broad Atlas** — 48 traits, 751 loci, 47 lookup-hit and 40 strictly eQTL-supported. The featured all-purpose panel with broad trait coverage across 21 biological domains.
+- **Mind & Risk** — 10 traits, 26 loci, 9 lookup-hit and 4 strictly eQTL-supported. Focused behavior and cognition panel: risk-taking, smoking, alcohol, mood, irritability, reaction time, neuroticism, insomnia, depression, fluid intelligence.
+- **Biochemistry Deep Dive** — 21 traits, 332 loci, 21 lookup-hit and 18 strictly eQTL-supported. High-yield blood biochemistry panel covering lipoproteins, kidney markers, endocrine biomarkers, minerals, and liver enzymes.
+- **Discovery Pool** — 150 traits, 643 loci, 72 lookup-hit and 34 strictly eQTL-supported. The discovery pool is built from the full Pan-UKB max independent set filtered to `num_pops_pass_qc >= 2`, so the broad build starts from non-redundant traits with at least minimal multi-population QC rather than the noisiest single-population results. It is kept as a curation reservoir, not a featured panel.
 
 ---
 
@@ -125,13 +126,13 @@ eQTL evidence is incorporated through targeted queries against the **eQTL Catalo
 3. **Stop if no rsID** — if no rsID can be recovered, mark the locus as not safely assessed (coordinate-based region queries are provisional and explicitly flagged)
 
 For each locus, the pipeline records:
-- Whether the locus has eQTL support
-- Which studies provided evidence
-- Best eQTL p-value
-- Supporting gene IDs
+- Whether the lookup returned any eQTL associations (`eQTL lookup-hit`)
+- Whether any positional candidate gene passes the strict support rule (`eQTL-supported`)
+- Which studies and datasets contributed evidence
+- Best observed eQTL p-value
 - Lookup mode used (`rsid`, `variant_recoder+rsid`, or `region_provisional`)
 
-A locus is **eQTL-supported** when its rsID-based lookup against prioritized eQTL Catalogue datasets returns at least one significant association between the lead variant and gene expression. A trait is **eQTL-supported** when at least one of its chrX loci has eQTL-supported candidate-gene evidence.
+A locus is **eQTL lookup-hit** when its rsID-based follow-up against prioritized eQTL Catalogue datasets returns one or more associations. A locus is **eQTL-supported** only when at least one pre-defined candidate gene at that locus has aggregated eQTL evidence with `best_eqtl_pvalue <= 1e-5` in the current build. A trait is **eQTL-supported** when at least one of its chrX loci has eQTL-supported candidate-gene evidence.
 
 Note: the current pipeline does not perform chrX LD modeling, colocalization analysis, or trait-specific tissue relevance matching. Study selection uses a prioritized dataset list (see `config/eqtl_priority_studies.csv`), not a tissue-relevance model. LD-aware colocalization and tissue-specific weighting are future work.
 
@@ -165,7 +166,7 @@ Each trait receives a **coverage grade** reflecting the quality and completeness
 - **Grade C** — high-quality cohort of at least 1,000
 - **Grade U** — does not meet the above thresholds; coverage is uncertain
 
-The coverage grade reflects **GWAS data quality**, not eQTL follow-up completeness. A trait can have Grade A coverage (strong upstream GWAS) but no eQTL support (no functional follow-up hits).
+The coverage grade reflects **GWAS data quality**, not eQTL follow-up completeness. A trait can have Grade A coverage (strong upstream GWAS) but no strict eQTL support.
 
 ### Gene annotation
 
@@ -192,8 +193,8 @@ The interactive frontend lives in `site/` and reads the JSON data bundle from `s
 ### Pages
 
 - **Homepage** (`index.html`) — Hero, metrics, curated panel cards, domain composition, top traits rail, panel comparison, discovery pool, methods explainer, search
-- **Panel detail** (`panel.html?id=...`) — Full trait table with column sorting, domain filter chips, text search, eQTL-only toggle
-- **Trait detail** (`trait.html?panel=...&trait=...`) — Evidence score, eQTL support explanation card, trait metadata grid, expandable locus cards with candidate gene tables, notes
+- **Panel detail** (`panel.html?id=...`) — Full trait table with column sorting, domain filter chips, text search, eQTL-supported-only toggle
+- **Trait detail** (`trait.html?panel=...&trait=...`) — Evidence score, eQTL lookup-hit versus support explanation card, trait metadata grid, expandable locus cards with candidate gene tables, notes
 
 ### Features
 

docs/FRONTEND_DATA_CONTRACT.md

@@ -0,0 +1,120 @@
+# Frontend Data Contract
+
+This document defines the JSON bundle contract produced by:
+
+`scripts/14_export_frontend_bundle.py`
+
+The bundle is additive. Existing release TSV files remain canonical for scientific auditing.
+
+## Output root
+
+Default output root:
+
+`../data/release/frontend/`
+
+Layout:
+
+- `manifest.json`
+- `panels/<panel_id>/summary.json`
+- `panels/<panel_id>/traits.json`
+- `panels/<panel_id>/traits/<trait_id>.json`
+
+## Schema versioning
+
+- `schema_version` is currently `2`.
+- `schema_version` is included in:
+  - `manifest.json`
+  - each panel `summary.json`
+  - each trait-detail JSON
+- If shape changes are not backward compatible, increment `schema_version`.
+
+## Panels in v1
+
+V1 exports exactly these panel IDs:
+
+- `panel_expanded` (`featured: true`)
+- `panel_mind_risk_core` (`featured: true`)
+- `panel_blood_biochemistry_core` (`featured: true`)
+- `panel_independent_set` (`featured: false`)
+
+`panel_independent_set` is intentionally marked non-featured so frontend can demote it by default.
+
+## Field semantics
+
+## `manifest.json`
+
+- `generated_at`: UTC ISO timestamp
+- `panels`: one entry per exported panel
+- each panel entry includes:
+  - `panel_id`, `slug`, `title`, `featured`, `description`
+  - `trait_count`, `lookup_hit_trait_count`, `supported_trait_count`, `zero_locus_trait_count`, `locus_count`
+  - `files` with relative paths to:
+    - `summary`
+    - `traits`
+    - `trait_details_dir`
+
+## `summary.json`
+
+- `panel`: panel metadata copied from manifest entry
+- `metrics`: aggregate counts
+- `domain_summary`: grouped by `domain`
+- `top_traits`: top compact trait rows
+- `unsupported_traits`: traits without strict eQTL support
+
+## `traits.json`
+
+Compact list for grid/table UIs. Each row includes:
+
+- `trait_id`
+- `query_id`
+- `slug`
+- `description`
+- `domain`
+- `x_evidence_score`
+- `coverage_grade`
+- `n_loci`
+- `eqtl_lookup_hit`
+- `eqtl_lookup_hit_locus_count`
+- `eqtl_lookup_hit_count`
+- `eqtl_supported`
+- `eqtl_supported_locus_count`
+- `eqtl_total_hit_count`
+- `top_candidate_genes` (array)
+- `confidence_notes`
+- `eqtl_lookup_note`
+
+## `traits/<trait_id>.json`
+
+- `trait`: compact trait row plus phenotype metadata (`trait_type`, `phenocode`, `pheno_sex`, `coding`, `modifier`, `category`)
+- `loci`: sorted by strongest signal first
+- each locus includes:
+  - `locus_id`, `x_region`, `locus_start`, `locus_end`
+  - `lead_pos`, `lead_rsid`, `lead_varid`, `lead_neglog10_pvalue`
+  - `top_gene_id`, `top_gene_name`, `best_mapping_relation`
+  - `eqtl_lookup_hit`, `eqtl_lookup_hit_gene_count`
+  - `eqtl_supported`, `eqtl_supported_gene_count`
+  - `eqtl_lookup_status`, `eqtl_lookup_mode`, `eqtl_lookup_n_hits`
+  - `candidate_genes` array
+- each candidate gene includes:
+  - `candidate_rank`, `gene_id`, `gene_name`, `gene_biotype`
+  - `mapping_relation`, `distance_to_lead_bp`, `eqtl_gene_role`
+  - `eqtl_lookup_hit`, `eqtl_lookup_hit_count`
+  - `eqtl_supported`, `eqtl_study_count`, `eqtl_dataset_count`, `best_eqtl_pvalue`
+
+## Evidence semantics
+
+- `eqtl_lookup_hit` means the prioritized rsID-based follow-up returned one or more associations.
+- `eqtl_supported` is stricter and means at least one candidate gene passed the build's current support rule after aggregation across queried datasets.
+- Trait-level `eqtl_supported` does not imply the positional top gene is supported.
+- `eqtl_gene_role = candidate` means the gene came from the positional mapping hierarchy.
+- `eqtl_gene_role = followup` means the gene was added only because it appeared in eQTL lookup results.
+
+## Normalization rules
+
+- Booleans are JSON booleans, not string literals.
+- Missing numeric/string values are `null`.
+- `top_candidate_genes` is always an array (split from comma-delimited TSV field).
+- Sorting:
+  - traits: `x_evidence_score desc`, `n_loci desc`, `eqtl_total_hit_count desc`, then stable string tie-breakers
+  - loci in trait detail: `lead_neglog10_pvalue desc`, `n_sig_variants desc`, then genomic start
+  - candidate genes per locus: `candidate_rank asc`, then lexical tie-breakers

pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "chrXatlas"
 version = "0.1.0"
-description = "Chromosome X evidence atlas built from public Pan-UK Biobank summary statistics with locus mapping and targeted eQTL follow-up"
+description = "Chromosome X discovery atlas built from public Pan-UK Biobank summary statistics with locus mapping and targeted eQTL follow-up"
 readme = "README.md"
 requires-python = ">=3.10"
 dependencies = [

scripts/06_fetch_eqtl_region_hits.py

@@ -177,15 +177,21 @@ def lookup_eqtl_by_rsid(
     api_base_url: str,
     timeout_seconds: float,
     study_specs: list[dict[str, str]],
+    pvalue_threshold: float | None,
 ) -> tuple[pd.DataFrame, str, str, str]:
     failures: list[str] = []
-    saw_explicit_no_hits = False
+    all_hits: list[pd.DataFrame] = []
+    completed_datasets = 0
+    no_hit_datasets = 0
 
     for spec in study_specs:
         dataset_id = str(spec.get("dataset_id", "") or "").strip()
         if not dataset_id:
             continue
+        dataset_failed = True
         params = {"rsid": rsid, "size": 1000}
+        if pvalue_threshold is not None:
+            params["p_upper"] = pvalue_threshold
         for api_version in ("v3", "v2"):
             outcome, hits, detail = try_api_request(
                 session,
@@ -196,18 +202,44 @@ def lookup_eqtl_by_rsid(
             )
             mode = f"rsid_api_dataset:{api_version}:{dataset_id}"
             if outcome == "hits":
-                return hits.drop_duplicates(), "complete", mode, detail
+                dataset_failed = False
+                completed_datasets += 1
+                tagged_hits = hits.drop_duplicates().copy()
+                tagged_hits["eqtl_dataset_id"] = dataset_id
+                tagged_hits["eqtl_api_version"] = api_version
+                all_hits.append(tagged_hits)
+                break
             if outcome == "no_hits":
-                saw_explicit_no_hits = True
-                failures.append(f"{mode}: {detail or 'no hits'}")
+                dataset_failed = False
+                completed_datasets += 1
+                no_hit_datasets += 1
                 break
+            failures.append(f"{mode}: {detail}")
             if detail.startswith("HTTP 404:"):
-                failures.append(f"{mode}: {detail}")
                 continue
-            failures.append(f"{mode}: {detail}")
+            break
+        if dataset_failed:
+            continue
+
+    if all_hits:
+        merged = pd.concat(all_hits, ignore_index=True).drop_duplicates()
+        status = "complete" if not failures else "partial_complete"
+        detail = (
+            f"Aggregated {len(merged):,} associations across {int(merged['eqtl_dataset_id'].nunique())} dataset(s)"
+        )
+        if no_hit_datasets:
+            detail += f"; {no_hit_datasets} dataset(s) returned no hits"
+        if failures:
+            detail += f"; {len(failures)} dataset/version attempt(s) failed"
+        return merged, status, "rsid_api_aggregate", compact_detail(detail)
+
+    if completed_datasets:
+        status = "no_hits" if not failures else "partial_complete"
+        detail = f"No associations returned across {completed_datasets} completed dataset lookup(s)."
+        if failures:
+            detail += f" {len(failures)} dataset/version attempt(s) failed."
+        return pd.DataFrame(), status, "rsid_api_aggregate", compact_detail(detail)
 
-    if saw_explicit_no_hits:
-        return pd.DataFrame(), "no_hits", "rsid_api", "No associations returned for the requested rsID."
     raise RuntimeError(compact_detail("; ".join(failures) or "No successful rsID lookup attempt."))
 
 
@@ -218,15 +250,21 @@ def lookup_eqtl_by_variant(
     api_base_url: str,
     timeout_seconds: float,
     study_specs: list[dict[str, str]],
+    pvalue_threshold: float | None,
 ) -> tuple[pd.DataFrame, str, str, str]:
     failures: list[str] = []
-    saw_explicit_no_hits = False
+    all_hits: list[pd.DataFrame] = []
+    completed_datasets = 0
+    no_hit_datasets = 0
 
     for spec in study_specs:
         dataset_id = str(spec.get("dataset_id", "") or "").strip()
         if not dataset_id:
             continue
+        dataset_failed = True
         params = {"variant": variant, "size": 1000}
+        if pvalue_threshold is not None:
+            params["p_upper"] = pvalue_threshold
         for api_version in ("v3", "v2"):
             outcome, hits, detail = try_api_request(
                 session,
@@ -237,18 +275,44 @@ def lookup_eqtl_by_variant(
             )
             mode = f"variant_api_dataset:{api_version}:{dataset_id}"
             if outcome == "hits":
-                return hits.drop_duplicates(), "complete", mode, detail
+                dataset_failed = False
+                completed_datasets += 1
+                tagged_hits = hits.drop_duplicates().copy()
+                tagged_hits["eqtl_dataset_id"] = dataset_id
+                tagged_hits["eqtl_api_version"] = api_version
+                all_hits.append(tagged_hits)
+                break
             if outcome == "no_hits":
-                saw_explicit_no_hits = True
-                failures.append(f"{mode}: {detail or 'no hits'}")
+                dataset_failed = False
+                completed_datasets += 1
+                no_hit_datasets += 1
                 break
+            failures.append(f"{mode}: {detail}")
             if detail.startswith("HTTP 404:"):
-                failures.append(f"{mode}: {detail}")
                 continue
-            failures.append(f"{mode}: {detail}")
+            break
+        if dataset_failed:
+            continue
+
+    if all_hits:
+        merged = pd.concat(all_hits, ignore_index=True).drop_duplicates()
+        status = "complete" if not failures else "partial_complete"
+        detail = (
+            f"Aggregated {len(merged):,} associations across {int(merged['eqtl_dataset_id'].nunique())} dataset(s)"
+        )
+        if no_hit_datasets:
+            detail += f"; {no_hit_datasets} dataset(s) returned no hits"
+        if failures:
+            detail += f"; {len(failures)} dataset/version attempt(s) failed"
+        return merged, status, "variant_api_aggregate", compact_detail(detail)
+
+    if completed_datasets:
+        status = "no_hits" if not failures else "partial_complete"
+        detail = f"No associations returned across {completed_datasets} completed dataset lookup(s)."
+        if failures:
+            detail += f" {len(failures)} dataset/version attempt(s) failed."
+        return pd.DataFrame(), status, "variant_api_aggregate", compact_detail(detail)
 
-    if saw_explicit_no_hits:
-        return pd.DataFrame(), "no_hits", "variant_api", "No associations returned for the requested variant."
     raise RuntimeError(compact_detail("; ".join(failures) or "No successful variant lookup attempt."))
 
 
@@ -408,6 +472,12 @@ def main() -> None:
     )
     parser.add_argument("--window-bp", type=int, default=1_000_000)
     parser.add_argument("--max-studies-per-locus", type=int, default=5)
+    parser.add_argument(
+        "--eqtl-pvalue-threshold",
+        type=float,
+        default=1e-5,
+        help="Conservative raw p-value cutoff applied to API queries when per-dataset FDR calls are unavailable.",
+    )
     parser.add_argument("--sleep-seconds", type=float, default=2.0)
     parser.add_argument("--out", default=str(INTERIM_DIR / "eqtl_hits.tsv.gz"))
     parser.add_argument("--summary-out", default=str(INTERIM_DIR / "eqtl_lookup_summary.tsv"))
@@ -496,6 +566,7 @@ def main() -> None:
                         api_base_url=args.api_base_url.rstrip("/"),
                         timeout_seconds=args.request_timeout_seconds,
                         study_specs=study_specs,
+                        pvalue_threshold=args.eqtl_pvalue_threshold,
                     )
                     if recovered_via_variant_recoder:
                         query_mode = f"variant_recoder+{query_mode}"
@@ -529,6 +600,7 @@ def main() -> None:
                         api_base_url=args.api_base_url.rstrip("/"),
                         timeout_seconds=args.request_timeout_seconds,
                         study_specs=study_specs,
+                        pvalue_threshold=args.eqtl_pvalue_threshold,
                     )
                     status = build_locus_query_status(
                         locus_id=locus_id,