Skip to content

Conversation

2010YOUY01
Copy link
Contributor

Which issue does this PR close?

Part of #18116

Rationale for this change

The below configuration can be used to let EXPLAIN ANALYZE only show important high-level insights.

set datafusion.explain.analyze_level = summary;

This PR sets summary level metrics for the parquet data source:

summary level metrics for DataSourceExec with Parquet source

The remaining metrics are kept in the dev level. I'm not sure if the page level pruning metrics should also be included to the summary level, I'm open to suggestions for this, or any other metrics that should also be included.

While implementing this, I came up with a few ideas to further improve metrics tracking in the Parquet scanner. I’ve documented them in #18195

What changes are included in this PR?

Set the above metrics to summary analyze level

Are these changes tested?

UTs

Are there any user-facing changes?

No

@github-actions github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate labels Oct 21, 2025
@xudong963
Copy link
Member

page level pruning metrics

I'd like to see it under any context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants