[SPARK-52448][CONNECT] Add simplified Struct Expression.Literal #51561
+520
−250
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
heyihong
force-pushed
the
SPARK-52448
branch
3 times, most recently
from
5d8e764 to
9d68335
Compare
dengziming
reviewed
Jul 19, 2025
| repeated StructField fields = 3; | ||
|
|
||
| message StructField { | ||
| string name = 1; |
There was a problem hiding this comment.
Thank you for this improvement. So we will contain the field name in every element, the space complexity would be O(n) compared to O(1) in the deprecated way? So what's the perf gain from this change?
There was a problem hiding this comment.
My understanding is that the space complexity in the deprecated approach is also O(n), since every element includes the field name in the struct_type field, correct?
There was a problem hiding this comment.
On second thought, it is not necessary to include the field name/data type in every element, since not setting the data_type field should be sufficient for saving space.
heyihong
force-pushed
the
SPARK-52448
branch
6 times, most recently
from
1db4711 to
f38223d
Compare
HyukjinKwon
approved these changes
Jul 21, 2025
|
Merged to master. |
haoyangeng-db
pushed a commit
to haoyangeng-db/apache-spark
that referenced
this pull request
Jul 22, 2025
### What changes were proposed in this pull request? This PR adds a new `data_type_struct` field to the protobuf definition for struct literals in Spark Connect, addressing the ambiguity issues with the existing `struct_type` field. The changes include: 1. **Protobuf Schema Update**: Added a new `data_type_struct` field of type `DataType.Struct` to the `Literal.Struct` message in `expressions.proto`, while marking the existing `struct_type` field as deprecated. 2. **Enhanced Struct Conversion Logic**: Updated `LiteralValueProtoConverter.scala` to: - Use the new `data_type_struct` field when available for more precise struct type definition - Maintain backward compatibility by still supporting the deprecated `struct_type` field - Add proper field metadata handling in struct conversions - Improve type inference for struct fields when data types can be inferred from literal values ### Why are the changes needed? The current Expression.Struct literal is somewhat overcomplicated since it duplicates most of the information its fields already have. This is bulky to send over the wire, and it can be ambiguous. ### Does this PR introduce _any_ user-facing change? No. This PR maintains backward compatibility with existing struct literal implementations. Existing code using the deprecated `struct_type` field will continue to work without modification. ### How was this patch tested? `build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor 1.2.4 Closes apache#51561 from heyihong/SPARK-52448. Authored-by: Yihong He <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds a new
data_type_structfield to the protobuf definition for struct literals in Spark Connect, addressing the ambiguity issues with the existingstruct_typefield. The changes include:Protobuf Schema Update: Added a new
data_type_structfield of typeDataType.Structto theLiteral.Structmessage inexpressions.proto, while marking the existingstruct_typefield as deprecated.Enhanced Struct Conversion Logic: Updated
LiteralValueProtoConverter.scalato:data_type_structfield when available for more precise struct type definitionstruct_typefieldWhy are the changes needed?
The current Expression.Struct literal is somewhat overcomplicated since it duplicates most of the information its fields already have. This is bulky to send over the wire, and it can be ambiguous.
Does this PR introduce any user-facing change?
No. This PR maintains backward compatibility with existing struct literal implementations. Existing code using the deprecated
struct_typefield will continue to work without modification.How was this patch tested?
build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor 1.2.4