Skip to content

[Variant] Preserve Field throughout Variant shredding flow #8673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+166 −99
Open

[Variant] Preserve Field throughout Variant shredding flow #8673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3567f6c
Drill field ref through the variant flow
friendlymatthew Oct 21, 2025
e21dc1b
Show example where we check for UUid extension type
friendlymatthew Oct 21, 2025
7144a08
Remove example for now
friendlymatthew Oct 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion parquet-variant-compute/src/shred_variant.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ impl<'a> VariantToShreddedObjectVariantRowBuilder<'a> {
let (value, typed_value, nulls) = typed_value_builder.finish()?;
let array =
ShreddedVariantFieldArray::from_parts(Some(value), Some(typed_value), nulls);
builder = builder.with_field(field_name, ArrayRef::from(array), false);
builder = builder.with_column(field_name, ArrayRef::from(array), false);
}
if let Some(nulls) = self.typed_value_nulls.finish() {
builder = builder.with_nulls(nulls);
Expand Down Expand Up @@ -400,6 +400,7 @@ mod tests {
let typed_value_field = result
.typed_value_field()
.unwrap()
.1
Comment on lines 401 to +403
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just glancing at this code in isolation, I would guess that .0 is the field and .1 is the array?
But that requires knowing context (ie of this PR)

If this usage will show up often, is it worth returning a newtype instead of a tuple?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes , please, let's return a type with real field names and documentation

.as_any()
.downcast_ref::<Int64Array>()
.unwrap();
Expand Down Expand Up @@ -459,6 +460,7 @@ mod tests {
let typed_value_int32 = result_int32
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<arrow::array::Int32Array>()
.unwrap();
Expand All @@ -471,6 +473,7 @@ mod tests {
let typed_value_float64 = result_float64
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
Expand Down Expand Up @@ -550,6 +553,7 @@ mod tests {
let typed_value = result
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<arrow::array::StructArray>()
.unwrap();
Expand All @@ -570,6 +574,7 @@ mod tests {
let score_typed_value = score_field
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
Expand All @@ -582,6 +587,7 @@ mod tests {
let age_typed_value = age_field
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<Int64Array>()
.unwrap();
Expand Down Expand Up @@ -912,6 +918,7 @@ mod tests {
let typed_value_field = result
.typed_value_field()
.unwrap()
.1
.as_any()
.downcast_ref::<Int64Array>()
.unwrap();
Expand Down
2 changes: 1 addition & 1 deletion parquet-variant-compute/src/unshred_variant.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ impl<'a> UnshredVariantRowBuilder<'a> {
fn try_new_opt(shredding_state: BorrowedShreddingState<'a>) -> Result<Option<Self>> {
let value = shredding_state.value_field();
let typed_value = shredding_state.typed_value_field();
let Some(typed_value) = typed_value else {
let Some((_, typed_value)) = typed_value else {
// Copy the value across directly, if present. Else caller decides what to do.
return Ok(value.map(|v| Self::ValueOnly(ValueOnlyUnshredVariantBuilder::new(v))));
};
Expand Down
Loading
Loading