Skip to content

Dynamic filters with partition columns do not get pushed down through DataSourceExec #18171

@feniljain

Description

@feniljain

Describe the bug

For queries like this:

select sum(ss_quantity) from store_sales, date_dim where ss_sold_date_sk=d_date_sk and d_year='2001';

where in store_sales table, ss_sold_date_sk is a partition column. For these queries, dynamic filter pushdown does not cross try_pushdown_filters of DataSourceExec because we don't pushdown filters containing filter columns here.

We do another round of file pruning for dynamic filters here in FileOpener, this does skip files.

Although it would be better to update partitions to be read using dynamic filters bounds, but this is enough for filtering them down for now at least.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions