Skip to content

[Migration] Lance, Video modality, Arrow backend fixes, performance improvements. - #59

Merged
Leon-Leyang merged 102 commits into
galilai-group:mainfrom
sami-bg:pyarrow-migration-shardless
May 19, 2026
Merged

[Migration] Lance, Video modality, Arrow backend fixes, performance improvements.#59
Leon-Leyang merged 102 commits into
galilai-group:mainfrom
sami-bg:pyarrow-migration-shardless

Conversation

@sami-bg

@sami-bg sami-bg commented Mar 20, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

  • Lance as a storage backend.
  • Video as a modality, with SSv2 as a starter dataset.
  • Arrow backend memory leak fixed.
  • Refactors, and test coverage.
  • New core builder classes.
  • Performance improvements that make us competitive with HF datasets (both Lance and Arrow backends).
  • benchmarks/ directory to run 6 backbones across >50 image datasets - includes some artifacts for the paper that will be removed/simplified in a later PR (i.e. analysis/ subdirectory).
  • profiling/ directory to run profiling of arrow, lance backends against HF datasets.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sami-bg
sami-bg force-pushed the pyarrow-migration-shardless branch from 9adacf9 to 38aed9f Compare April 27, 2026 19:02
@sami-bg sami-bg changed the title [WIP] [Feature] Vastly improved, shardless PyArrow performance on large datasets [WIP] [Feature] Lance, Video modality, Arrow backend fixes, performance improvements. Apr 29, 2026
@sami-bg sami-bg changed the title [WIP] [Feature] Lance, Video modality, Arrow backend fixes, performance improvements. [Feature] Lance, Video modality, Arrow backend fixes, performance improvements. Apr 29, 2026
@sami-bg sami-bg changed the title [Feature] Lance, Video modality, Arrow backend fixes, performance improvements. [Migration] Lance, Video modality, Arrow backend fixes, performance improvements. May 15, 2026
@Leon-Leyang
Leon-Leyang merged commit 4ceafa1 into galilai-group:main May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants