Skip to content

Add a datafusion benchmark for filter_pushdown #16729

@alamb

Description

@alamb

Is your feature request related to a problem or challenge?

In order to enable filter_pushdown by default, we need to ensure it doesn't regress existing performance

However, it has been very hard to make forward progress on improving filter pushdown because all our benchmarks compare filter pushdown to not filter pushdown, so the bar for change is quite high.
Here is the most recent example:

Describe the solution you'd like

I would like a benchmark to measure for improvements to the filter pushdown are comparing when filter pushdown is already on. However, we don't have any such benchmark

There is one existing benchmark called parquet but it seems not to capture real world filter pushdown usecases and is not often run (I have never seen or heard of anyone running it)

Describe alternatives you've considered

  1. Remove the old "parquet" benchmark (that isn't used)
  2. Add a new clickbench_filter_pushdown benchmark that runs the same clickbench queries with filter pushdown enabled

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions