Update ClickBench benchmarks with DataFusion 49.0.0

[Omega359]

Is your feature request related to a problem or challenge?

Follow on to

#14587

Requires

#16235

Describe the solution you'd like

Once DataFusion 49.0.0 is released, It would be great to update ClickBench https://benchmark.clickhouse.com/ with the newest version and highlight and improvement in the DF 49 blog post

ClickBench is a benchmark heavy on filtering and aggregation that we have used as an optimization target for the last several releases.

Describe alternatives you've considered

No response

Additional context

No response

[Omega359]

Given the number of improvements that have been made to DF in the last 3 releases I'm expecting that DF should show an improvement in the clickbench benchmark (even with #16444)

[Omega359]

@pmcgleenon since you've done the clickbench updates in the past would you be willing to handle this ticket?

[pmcgleenon]

@Omega359 sure I can pick this up. Looks like 49.0.0 is available

[pmcgleenon]

I'm seeing frequent Out of Memory errors when running Clickbench against datafusion 49.0.0 on the instance type c6a.xlarge (8GB RAM). It does seem to be intermittent, for example if I run Q19 several times it might pass once and OOM twice.

I see this on both single and partitioned tests.

Looking at the current results for Clickbench, there are also null results reported - which happens on OOM

• Q19 partitioned https://github.com/ClickHouse/ClickBench/blob/main/datafusion-partitioned/results/c6a.xlarge.json#L30
• Q33 partitioned https://github.com/ClickHouse/ClickBench/blob/main/datafusion-partitioned/results/c6a.xlarge.json#L44

@Omega359 @alamb is this a known issue?

[Dandandan]

I think the previous runs were using c6a.4xlarge?

[pmcgleenon]

@Dandandan yes that's correct the previous runs were all using c6a.4xlarge.

Recently the clickhouse team have added results for these additional instance types. It looks like this was done as part of a wider change to add them across multiple projects

• c6a.xlarge.json
• c6a.2xlarge.json
• c8g.4xlarge.json

Also the datafusion tests are now separated into datafusion (single) and datafusion-partitioned (partitioned) directories

https://github.com/ClickHouse/ClickBench/tree/main/datafusion-partitioned/results
https://github.com/ClickHouse/ClickBench/tree/main/datafusion/results

The published results for c6a.xlarge and c6a.2xlarge have null values in some of the tests which is a sign that OOM happened during the tests. So it seems like c6a.4xlarge is the minimum instance type to get valid results.

I can think of a couple of options - either datafusion will not report results for c6a.xlarge and c6a.2xlarge or we figure out a way to run successfully on the lower memory systems. Maybe there's some configuration we can try.

What do you think?

[Omega359]

We could try changing the default config for the small type though I am unsure if that is in the spirit of the clickbench benchmark. A few options that may or may not help, others may have other ideas

datafusion.execution.collect_statistics -> false,
datafusion.execution.parquet.schema_force_view_types -> false,
datafusion.execution.max_buffered_batches_per_output_file -> 1,

[adriangb]

Do we set memory limits? DataFusion doesn't really detect memory limits from actual memory available or anything like that AFAIK.

[alamb]

@Omega359 @alamb is this a known issue?

It is not a known issue that I know of

The published results for c6a.xlarge and c6a.2xlarge have null values in some of the tests which is a sign that OOM happened during the tests. So it seems like c6a.4xlarge is the minimum instance type to get valid results.

I can think of a couple of options - either datafusion will not report results for c6a.xlarge and c6a.2xlarge or we figure out a way to run successfully on the lower memory systems. Maybe there's some configuration we can try.

I personally think we should run DataFusion 49 as is and generate results on all platforms (even if some of the queries OOM) and then we can file issues / fix any problems we encounter running on those platforms

In other words, I think we should decouple the tasks

• "figure out why DataFusion seems to be OOMing"
• "run the benchmarks with DataFusion 49.0.0"

[alamb]

BTW as part of something else, I noticed we don't seem to be Compiling ClickBench with the native CPU instructions: https://github.com/ClickHouse/ClickBench/blob/f4b58bdb60e452e22c8d99196f90afe2158f7b3a/datafusion/benchmark.sh#L17

Specifically, I think RUSTFLAGS should be updated from

RUSTFLAGS="-C codegen-units=1"

to

RUSTFLAGS="-C codegen-units=1 -C target-cpu=native"

Is that something you can try @pmcgleenon ?

[alamb]

Do we set memory limits? DataFusion doesn't really detect memory limits from actual memory available or anything like that AFAIK.

that is correct

the scripts do not set any memory limit

[pmcgleenon]

Specifically, I think RUSTFLAGS should be updated from

RUSTFLAGS="-C codegen-units=1"

to

RUSTFLAGS="-C codegen-units=1 -C target-cpu=native"

Is that something you can try @pmcgleenon ?

It's a great suggestion, I tried that out and the addition of target-cpu=native causes OOM for rustc on the c6a.xlarge instance type (8 GB RAM) when building datafusion-cli. Note that swap isn't enabled on the benchmark instance by default. I was thinking to get around this we could enable swap just for the datafusion-cli build in benchmark.sh and disable it again before the benchmark test runs. What do you think?

[adriangb]

the scripts do not set any memory limit

setting memory limit = memory available on machine sounds reasonable to me and would explain OOMs, but I'm not sure if that's in the spirit of clickbench or not. if that is the issue having a default memory limit equal to memory available on machine sounds reasonable for datafusion-cli.