[SPARK-52798] [SQL] Add function approx_top_k_combine #51505
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cboumalh
reviewed
Jul 24, 2025
| override def serialize(buffer: CombineInternal[Any]): Array[Byte] = { | ||
| val sketchBytes = buffer.getSketch.toByteArray( | ||
| ApproxTopK.genSketchSerDe(buffer.getItemDataType)) | ||
| val maxItemsTrackedByte = buffer.getMaxItemsTracked.toByte |
There was a problem hiding this comment.
I believe this is casting an Int as a Byte. Could this cause any overflow?
| InternalRow.apply(sketchBytes, null, maxItemsTracked, typeCode) | ||
| } | ||
|
|
||
| override def serialize(buffer: CombineInternal[Any]): Array[Byte] = { |
There was a problem hiding this comment.
Serialization and deserialization here are a bit concerning. The current implementation has brittle byte-level manipulations with no robust error checking or flexibility. Hardcoded assumptions about byte sizes and type conversions create significant risks for future compatibility and data integrity. I think it may be worth looking into using a ByteBuffer and adding versioning to the sketch buffer, or any other approaches!
| @@ -177,6 +177,8 @@ object ApproxTopK { | |||
| val DEFAULT_K: Int = 5 | |||
| val DEFAULT_MAX_ITEMS_TRACKED: Int = 10000 | |||
| private val MAX_ITEMS_TRACKED_LIMIT: Int = 1000000 | |||
| val VOID_MAX_ITEMS_TRACKED = -1 | |||
There was a problem hiding this comment.
Can we document what exactly this VOID_MAX_ITEMS_TRACKED and SKETCH_SIZE_PLACEHOLDER mean?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds a SQL function:
approx_top_k_accumulate, an aggregation function that merges multiple sketches into a single sketch.Syntax
Arguments
expr: An expression of sketch structsmaxItemsTracked: An optional INTEGER literal. If maxItemsTracked is specified, use this value for the newly generated combined sketch. If maxItemsTracked is not specified, all input sketches must have the same maxItemsTracked, and the output sketch would use the same value as well.Returns
The return of this function is a STRUCT with four fields: sketch, itemDataType, maxItemsTracked and typeCode. The return is exactly the same as for approx_top_k_accumulate.
Why are the changes needed?
They are useful sibling functions for approx_top_k queries.
Does this PR introduce any user-facing change?
Yes, this PR introduces a new user-facing SQL function. See user examples as below.
How was this patch tested?
Unit tests for end-to-end SQL queries and invalid input for expressions.
Was this patch authored or co-authored using generative AI tooling?