Skip to content

Drop pandas dependency #147

@bmschmidt

Description

@bmschmidt

For once #145 is complete.

Things will be faster and cleaner if we simply use pyarrow straight through to pass around intermediate representations.
Passing arrow data through pandas tends to give another chance for typecasting errors to creep in, potentially makes it harder to correct errors involving timezones in date fields, etc.

Nothing especially urgent here, particularly because there are some necessary join operations that can't be handled natively by pyarrow.compute. In nonconsumptive, those are usually handled by polars right now; in this one, it might make more sense to do that relational logic on arrow tables by using duckdb on them, since one nice feature of duck is that it just lets you write SQL on local arrow dataframes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions