Part of epic #66 (modernize Polars usage / adopt lazy evaluation).
Problem
io.load_table always eagerly read_parquet / read_csvs the full file, even when the caller needs only a preview or a single column. The codebase already scans in a few spots (gui.py:3688-3690 gene-set first column; fastq.py; a couple in io.py) but there's no shared lazy loader.
Goal
Add a lazy loader (e.g. load_table_lazy(...) -> pl.LazyFrame) mirroring load_table's dialect/cleanup logic, and route callers that only need a subset through scan + projection/predicate pushdown:
- report node thumbnails / previews,
- gene-set extraction,
- any "just need shape / first N rows / one column" path.
Acceptance
- Lazy loader returns a LazyFrame whose
.collect() equals load_table(...) for the same file (reproducibility).
- At least the preview + gene-set paths use it; no full-table materialization where a subset suffices.
- TDD.
Part of epic #66 (modernize Polars usage / adopt lazy evaluation).
Problem
io.load_tablealways eagerlyread_parquet/read_csvs the full file, even when the caller needs only a preview or a single column. The codebase already scans in a few spots (gui.py:3688-3690gene-set first column;fastq.py; a couple inio.py) but there's no shared lazy loader.Goal
Add a lazy loader (e.g.
load_table_lazy(...) -> pl.LazyFrame) mirroringload_table's dialect/cleanup logic, and route callers that only need a subset through scan + projection/predicate pushdown:Acceptance
.collect()equalsload_table(...)for the same file (reproducibility).