Skip to content

Lazy table-loading path (scan_parquet/scan_csv) with pushdown for subset reads #138

Description

@GuyTeichman

Part of epic #66 (modernize Polars usage / adopt lazy evaluation).

Problem

io.load_table always eagerly read_parquet / read_csvs the full file, even when the caller needs only a preview or a single column. The codebase already scans in a few spots (gui.py:3688-3690 gene-set first column; fastq.py; a couple in io.py) but there's no shared lazy loader.

Goal

Add a lazy loader (e.g. load_table_lazy(...) -> pl.LazyFrame) mirroring load_table's dialect/cleanup logic, and route callers that only need a subset through scan + projection/predicate pushdown:

  • report node thumbnails / previews,
  • gene-set extraction,
  • any "just need shape / first N rows / one column" path.

Acceptance

  • Lazy loader returns a LazyFrame whose .collect() equals load_table(...) for the same file (reproducibility).
  • At least the preview + gene-set paths use it; no full-table materialization where a subset suffices.
  • TDD.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions