Skip to content

refactor(query): reduce memory usage during aggregate spill #17550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 85 commits into
base: main
Choose a base branch
from
Draft

refactor(query): reduce memory usage during aggregate spill #17550

wants to merge 85 commits into from

Conversation

zhang2014
Copy link
Member

@zhang2014 zhang2014 commented Mar 4, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

refactor(query): reduce memory usage during aggregate spill

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Mar 4, 2025
@BohuTANG BohuTANG mentioned this pull request Mar 4, 2025
Closed
4 tasks
@zhang2014 zhang2014 mentioned this pull request Mar 13, 2025
Closed
11 tasks
zhang2014 added 22 commits March 13, 2025 13:31
@zhang2014
Merge branch 'main' into refactor/aggregate_spill
e57cbe1 
# Conflicts:
#	src/query/service/src/pipelines/processors/transforms/aggregator/serde/transform_aggregate_spill_writer.rs
#	src/query/service/src/pipelines/processors/transforms/aggregator/serde/transform_exchange_aggregate_serializer.rs
@zhang2014
refactor(query): refactor aggreagte spill code
be7c1da
@zhang2014
refactor(query): refactor aggreagte spill code
b7cf072
@zhang2014
refactor(query): refactor aggreagte spill code
bf59696
@zhang2014
refactor(query): refactor aggreagte spill code
b77c041
@zhang2014
refactor(query): refactor aggreagte spill code
78a0199
@zhang2014
refactor(query): refactor aggreagte spill code
22dad6e
@zhang2014
refactor(query): refactor aggreagte spill code
417f64d
@zhang2014
refactor(query): refactor aggreagte spill code
4193c6c
@zhang2014
refactor(query): refactor aggreagte spill code
20ca2a9
@zhang2014
refactor(query): refactor aggreagte spill code
472aa37
@zhang2014
refactor(query): refactor aggreagte spill code
2f62d3d
@zhang2014
refactor(query): refactor aggreagte spill code
3a0652b
@zhang2014
refactor(query): refactor aggreagte spill code
9a56d22
@zhang2014
refactor(query): refactor aggreagte spill code
552bcda
@zhang2014
refactor(query): refactor aggreagte spill code
7d46563
@zhang2014
Merge branch 'main' into refactor/aggregate_spill
f6465f3
@zhang2014
refactor(query): refactor aggreagte spill code
5f1fbbc
@zhang2014
refactor(query): refactor aggreagte spill code
d61d925
@zhang2014
refactor(query): refactor aggreagte spill code
1206211
@zhang2014
refactor(query): refactor aggreagte spill code
0374a7f
@zhang2014
refactor(query): refactor aggreagte spill code
ce6e754
@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17550-45df4dc-1744551215

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17550-6382953-1744567065

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 14, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17550-8d0356a-1744598816

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 14, 2025
@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 14, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17550-742e31a-1744610730

note: this image tag is only available for internal use.

@zhang2014 zhang2014 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Apr 14, 2025
@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17550-0f043ef-1744613446

note: this image tag is only available for internal use.

@github-actions GitHub Actions
Copy link
Contributor

ClickBench Report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sundy-li sundy-li sundy-li left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
ci-benchmark Benchmark: run all test ci-cloud Build docker image for cloud test pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zhang2014 @sundy-li