Merge arrow-rs/main into shredding-variant-part1 #5

scovich · 2025-09-04T22:56:14Z

Merging apache#8166 with upstream main was a bit hairy because of strong logical conflicts with apache#8179.

Hopefully this helps unblock the PR.

…iant kernel (apache#8201) # Which issue does this PR close? - Closes apache#8060. # Rationale for this change Need to implement `List`, `LargeList` types support for `cast_to_variant` kernel # What changes are included in this PR? Added support for `List`, `LargeList` in `cast_to_variant` kernel # Are these changes tested? Yes, added unit tests # Are there any user-facing changes? Yes, added changes to the `cast_to_variant` kernel --------- Co-authored-by: Konstantin.Tarasov <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

# Which issue does this PR close? - Part of apache#4886 # Rationale for this change This PR introduces benchmark tests for the `AvroWriter` in the `arrow-avro` crate. Adding these benchmarks is essential for tracking the performance of the writer, identifying potential regressions, and guiding future optimizations. # What changes are included in this PR? A new benchmark file, `benches/avro_writer.rs`, is added to the project. This file contains a suite of benchmarks that measure the performance of writing `RecordBatch`es to the Avro format. The benchmarks cover a variety of Arrow data types: - `Boolean` - `Int32` and `Int64` - `Float32` and `Float64` - `Binary` - `Timestamp` (Microsecond precision) - A schema with a mix of the above types These benchmarks are run with varying numbers of rows (100, 10,000, and 1,000,000) to assess performance across different data scales. # Are these changes tested? Yes, this pull request consists entirely of new benchmark tests. Therefore, no separate tests are needed. # Are there any user-facing changes? NA

This method was removed in apache#7824, which introduced an optimized code path for writing bloom filters on little-endian architectures. The method was however still used in the big-endian code-path. Due to the use of `#[cfg(target_endian)]` this went unnoticed in CI. Fixes apache#8207

…che#8177) # Which issue does this PR close? - Closes apache#8063 # Rationale for this change Maps are now cast to `Variant::Object`s # What changes are included in this PR? # Are these changes tested? Yes # Are there any user-facing changes? --------- Co-authored-by: Andrew Lamb <[email protected]>

…he#8105) # Which issue does this PR close? We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. - Closes apache#8091 . # Rationale for this change Implement `VariantArray::value` for some more shredded variants(eg. primitive_conversion/generic_conversion/non_generic_conversion). # What changes are included in this PR? - Extract all `macroRules` to a separate module `type_conversion.rs` - Add a macro for `variant value` # Are these changes tested? Covered by the existing test # Are there any user-facing changes? No

…kernel (apache#8196) # Which issue does this PR close? - Closes apache#8195. # Rationale for this change # What changes are included in this PR? Implement `DataType::Union` for `cast_to_variant` # Are these changes tested? Yes # Are there any user-facing changes? New cast type supported --------- Co-authored-by: Andrew Lamb <[email protected]>

…pache#8206) # Which issue does this PR close? - Closes apache#8205 # Rationale for this change `VariantArrayBuilder` had a very complex choreography with the `VariantBuilder` API, that required lots of manual drop glue to deal with ownership transfers between it and the `VariantArrayVariantBuilder` it delegates the actual work to. Rework the whole thing to use a (now-reusable) `MetadataBuilder` and `ValueBuilder`, with rollbacks largely handled by `ParentState` -- just like the other builders in the parquet-variant crate. # What changes are included in this PR? Five changes (curated as five commits that reviewers may want to examine individually): 1. Make a bunch of parquet-variant builder infrastructure public, so that `VariantArrayBuilder` can access it from the parquet-variant-compute crate. 2. Make `MetadataBuilder` reusable. Its `finish` method appends the bytes of a new serialized metadata dictionary to the underlying buffer and resets the remaining builder state. The builder is thus ready to create a brand new metadata dictionary whose serialized bytes will also be appended to the underlying buffer once finished. 3. Rework `VariantArrayBuilder` to use `MetadataBuilder` and `ValueBuilder`, coordinated via `ParentState`. This is the main feature of the PR and also the most complicated/subtle. 4. Delete now-unused code that had been added previously in order to support the old implementation of `VariantArrayBuilder`. 5. Add missing doc comments for now-public types and methods # Are these changes tested? Existing variant array builder tests cover the change. # Are there any user-facing changes? A lot of builder-related types and methods from the parquet-variant crate are now public.

…g fails (apache#8213) # Which issue does this PR close? - Closes apache#8212 # Rationale for this change In the original code, the bitmap was modified before decoding. Even if decoding fails, the null buffer was modified, leading to bitmap corruption, eventually causing flush to fail. # What changes are included in this PR? This PR fixes the bug where the bitmap was modified before decoding. If there is decoding failure, the bitmap should not be modified but the decode method should be exited gracefully without any side effect. # Are these changes tested? - Added a unit test # Are there any user-facing changes? No.

# Which issue does this PR close? - Closes apache#8152 # Rationale for this change When manipulating existing variant values (unshredding, removing fields, etc), the metadata column is already defined and already contains all necessary field ids. In fact, defining new/different field ids would require rewriting the bytes of those already-encoded variant values. We need a way to build variant values that rely on an existing metadata dictionary. # What changes are included in this PR? * `MetadataBuilder` is now a trait, and most methods that work with metadata builders now take `&mut dyn MetadataBuilder` instead of `&mut MetadataBuilder`. * The old `MetadataBuilder` struct is now `BasicMetadataBuilder` that implements `MetadataBuilder` * Define a `ReadOnlyMetadataBuilder` that wraps a `VariantMetadata` and which also implements `MetadataBuilder` * Update the `try_binary_search_range_by` helper method to be more general, so we can define an efficient `VariantMetadata::get_entry` that returns the field id for a given field name. # Are these changes tested? Existing tests cover the basic metadata builder. New tests added to cover the read-only metadata builder. # Are there any user-facing changes? The renamed `BasicMetadataBuilder` (breaking), the new `MetadataBuilder` trait (breaking), and the new `ReadOnlyMetadataBuilder`.

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/upload-pages-artifact](https://github.com/actions/upload-pages-artifact) from 3 to 4. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…he#8210) # Which issue does this PR close? Closes apache#8209 # Rationale for this change In the Field struct definition ``` /// A field within a [`Record`] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct Field<'a> { /// Name of the field within the record #[serde(borrow)] pub name: &'a str, /// Optional documentation for this field #[serde(borrow, default)] pub doc: Option<&'a str>, /// The field's type definition #[serde(borrow)] pub r#type: Schema<'a>, /// Optional default value for this field #[serde(borrow, default)] pub default: Option<&'a str>, } ``` type is of type `Schema` whereas default is of type `str`. The default should be supported for all types (e.g. int, array, map, nested record), so we should make it more lenient. More details on reproduction is mentioned in the Github Issue. # What changes are included in this PR? Relaxation of default type of avro scheam Field. # Are these changes tested? Added a unit test. # Are there any user-facing changes? It affects `pub struct Field` of `arrow-avro` package, but the impact should be minimal as the `default` attribute is not being used.

…_type` (apache#8216) # Which issue does this PR close? None. # Rationale for this change I noticed an error in the doc comment about error conditions of `Field::try_canonical_extension_type`. # What changes are included in this PR? Fixed the doc comment. # Are these changes tested? No. # Are there any user-facing changes? No.

…t` kernel (apache#8215) # Which issue does this PR close? - Closes apache#8194. # Rationale for this change # What changes are included in this PR? Implement `duration` the same as `interval` # Are these changes tested? Yes # Are there any user-facing changes?

…pache#8141) # Which issue does this PR close? - Closes apache#8217 # Rationale for this change When working with shredded variants, we need the ability to copy nested object fields and array elements of one variant to a destination. This is a cheap byte-wise copy that relies on the fact that the new variant being built uses the same metadata dictionary as the source variant it is derived from. # What changes are included in this PR? Define a helper macro that encapsulates the logic for variant appends, now that we have three very similar methods (differing only in their handling of list/object values and their return type). Add new methods: `ValueBuilder::append_variant_bytes`, which is called by new methods `VariantBuilder::append_value_bytes`, `ListBuilder::append_value_bytes`, and `ObjectBuilder::[try_]insert_bytes`. # Are these changes tested? New unit tests # Are there any user-facing changes? The new methods are public. --------- Co-authored-by: Andrew Lamb <[email protected]>

…e#8214) # Which issue does this PR close? - Closes apache#8184 # Rationale for this change # What changes are included in this PR? There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. # Are these changes tested? Yes # Are there any user-facing changes? `Object::finish` doesn't return `Result` anymore --------- Co-authored-by: Andrew Lamb <[email protected]>

Reverts: - apache#8183 Because the related issue was closed: - apache#8181

# Which issue does this PR close? - Closes apache#8243 . # What changes are included in this PR? pin `comfy-table` to release prior to 7.2.0's MSRV bump to 1.85 - included a TODO to unpin after arrow bumps to 1.85 (context FWIW: caught in delta_kernel [MSRV CI](https://github.com/delta-io/delta-kernel-rs/actions/runs/17310376492/job/49143119497)) # Are these changes tested? validated MSRV with cargo-msrv: ```bash # now passes cargo msrv --path arrow-cast/ verify --rust-version 1.84 --all-features ```

# Which issue does this PR close? - Closes apache#8228. # What changes are included in this PR? Add `Variant::as_f16` # Are these changes tested? Added doc tests # Are there any user-facing changes? Added doc for the function --------- Co-authored-by: Matthijs Brobbel <[email protected]>

Updates the requirements on [hashbrown](https://github.com/rust-lang/hashbrown) to permit the latest version. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

# Which issue does this PR close? The doc for lexsort says it's stable. However, it's an unstable sort. # Rationale for this change Fix the document. # What changes are included in this PR? Fix the document. # Are these changes tested? No need # Are there any user-facing changes? Doc change --------- Co-authored-by: Matthijs Brobbel <[email protected]>

# Which issue does this PR close? \- # Rationale for this change Some services support gRPC compression. Expose this to the CLI client for: - testing - more efficient data transfer over slow internet connections # What changes are included in this PR? CLI argument wiring. # Are these changes tested? No automated tests. I think we can assume that the libraries we use do what they promise to do. But I also verified that this works by inspecting the traffic using Wireshark. # Are there any user-facing changes? They now have more options.

@jecsand838

# Which issue does this PR close? - Part of apache#4886 - Extends work initiated in apache#8006 # Rationale for this change This introduces support for Confluent schema registry ID handling in the arrow-avro crate, adding compatibility with Confluent's wire format. These improvements enable streaming Apache Kafka, Redpanda, and Pulsar messages with Avro schemas directly into arrow-rs. # What changes are included in this PR? - Adds Confluent support - Adds initial support for SHA256 and MD5 algorithm types. Rabin remains the default. # Are these changes tested? Yes, existing tests are all passing, and tests for ID handling have been added. Benchmark results show no appreciable changes. # Are there any user-facing changes? - Confluent users need to provide the ID fingerprint when using the `set` method, unlike the `register` method which generates it from the schema on the fly. Existing API behavior has been maintained. - SchemaStore TryFrom now accepts a `&HashMap<Fingerprint, AvroSchema>`, rather than a `&[AvroSchema]` Huge shout out to @jecsand838 for his collaboration on this! --------- Co-authored-by: Connor Sanders <[email protected]>

…he#8179) # Which issue does this PR close? - Closes apache#8178 # Are these changes tested? Yes # Are there any user-facing changes? Can use `variant_get` for shredded numeric types --------- Co-authored-by: Andrew Lamb <[email protected]>

# Which issue does this PR close? - Part of apache#4886 - Follows up on apache#8047 # Rationale for this change When reading Avro into Arrow with a projection or a reader schema that omits some writer fields, we were still decoding those writer‑only fields item‑by‑item. This is unnecessary work and can dominate CPU time for large arrays/maps or deeply nested records. Avro’s binary format explicitly allows fast skipping for arrays/maps by encoding data in blocks: when the count is negative, the next `long` gives the byte size of the block, enabling O(1) skipping of that block without decoding each item. This PR teaches the record reader to recognize and leverage that, and to avoid constructing decoders for fields we will skip altogether. # What changes are included in this PR? **Reader / decoding architecture** - **Skip-aware record decoding**: - At construction time, we now precompute per-record **skip decoders** for writer fields that the reader will ignore. - Introduced a resolved-record path (`RecordResolved`) that carries: - `writer_to_reader` mapping for field alignment, - a prebuilt list of **skip decoders** for fields not present in the reader, - the set of active per-field decoders for the projected fields. - **Codec builder enhancements**: In `arrow-avro/src/codec.rs`, record construction now: - Builds Arrow `Field`s and their decoders only for fields that are read, - Builds `skip_decoders` (via `build_skip_decoders`) for fields to ignore. - **Error handling and consistency**: Kept existing strict-mode behavior; improved internal branching to avoid inconsistent states during partial decodes. **Tests** - **Unit tests (in `arrow-avro/src/reader/record.rs`)** - Added focused tests that exercise the new skip logic: - Skipping writer‑only fields inside **arrays** and **maps** (including negative‑count block skipping and mixed multi‑block payloads). - Skipping nested structures within records to ensure offsets and lengths remain correct for the fields that are read. - Ensured nullability and union handling remain correct when adjacent fields are skipped. - **Integration tests (in `arrow-avro/src/reader/mod.rs`)** - Added end‑to‑end test using `avro/alltypes_plain.avro` to validate that projecting a subset of fields (reader schema omits some writer fields) both: - Produces the correct Arrow arrays for the selected fields, and - Avoids decoding skipped fields (validated indirectly via behavior and block boundaries). - The test covers compressed and uncompressed variants already present in the suite to ensure behavior is consistent across codecs. # Are these changes tested? - **New unit tests** cover: - Fast skipping for arrays/maps using negative block counts and block sizes (per Avro spec). - Nested and nullable scenarios to ensure correct offsets, validity bitmaps, and flush behavior when adjacent fields are skipped. - **New integration test** in `reader/mod.rs`: - Reads `avro/alltypes_plain.avro` with a reader schema that omits several writer fields and asserts the resulting `RecordBatch` matches the expected arrays while exercising the skip path. - Existing promotion, enum, decimal, fixed, and union tests continue to pass, ensuring no regressions in unrelated areas. # Are there any user-facing changes? N/A since `arrow-avro` is not public yet.

Resolves conflicts between PR 8166 (shredding support) and PR 8179 (multi-type support): - Preserves PR 8179's comprehensive multi-type support for all numeric primitives - Keeps PR 8166's superior row builder architecture and shredding support - Integrates both test suites for complete coverage - Maintains enhanced path parsing from PR 8166 The merge successfully combines: - Multi-type variant_get support (Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64, Float16, Float32, Float64) - Advanced shredding capabilities with row builder approach - Comprehensive test coverage from both PRs

scovich · 2025-09-04T22:58:40Z

Hmm, the diff is a real mess. I'm not sure it would actually make sense to merge this directly into your branch, because it probably wouldn't register as a proper merge commit.

# Which issue does this PR close? - Closes apache#7173. # Rationale for this change Ability to round-trip timezone information. # What changes are included in this PR? Impl `Display` for `Tz` # Are these changes tested? A simple test that strings round trip. # Are there any user-facing changes? New API

# Which issue does this PR close? - Closes apache#8273 . # Rationale for this change When working with the library using encryption, we have sometimes found it necessary to modify an existing set of `WriterProperties` on a per-file basis to set specific encryption properties. More generally, others may need to use an existing set of `WriterProperties` as a template and modify the properties. I have implemented this feature by adding an `into_builder` method, which appears to be the standard approach in other parts of the library. # Are these changes tested? Yes, `test_writer_properties_builder` has been updated to add a round-trip test for `into_builder`. # Are there any user-facing changes? Yes. `WriterProperties` now has a new `into_builder` method. --------- Co-authored-by: Andrew Lamb <[email protected]>

# Which issue does this PR close? - Part of apache#5854. # Rationale for this change Backport changes to allow apples-to-apples comparison of thrift decoding # What changes are included in this PR? Adds a page header benchmark and updates bench names to match those in feature branch. # Are these changes tested? No tests needed...only changes to benchmark # Are there any user-facing changes? No

Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/labeler](https://github.com/actions/labeler) from 6.0.0 to 6.0.1. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

# Which issue does this PR close? - Closes apache#8261. # Rationale for this change Add same API between sync and async API # What changes are included in this PR? There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. # Are these changes tested? Add test_async_arrow_group_writer # Are there any user-facing changes? Yes, add two public function get_column_writers, append_row_group for AsyncArrowWrite

# Which issue does this PR close? - Closes apache#8283. # Rationale for this change Add the `Variant::as_u*` functions` # Are these changes tested? Added doc tests # Are there any user-facing changes? No

# Which issue does this PR close? - Closes apache#8234. # Rationale for this change # What changes are included in this PR? - Grouping related data types together (e.g., numeric types, temporal types). - Extracting large code snippets from match branches into helper functions. - Reordering tests to align with the data type order. # Are these changes tested? Covered by existing tests # Are there any user-facing changes? N/A

) # Which issue does this PR close? - Part of apache#4886 - Follows up on apache#8047 # Rationale for this change Avro `enum` values are **encoded by index** but are **semantically identified by symbol name**. During schema evolution it is legal for the writer and reader to use different enum symbol *orders* so long as the **symbol set is compatible**. The Avro specification requires that, when resolving a writer enum against a reader enum, the value be mapped **by symbol name**, not by the writer’s numeric index. If the writer’s symbol is not present in the reader’s enum and the reader defines a default, the default is used; otherwise it is an error. # What changes are included in this PR? **Core changes** - Implement **writer to reader enum symbol remapping**: - Build a fast lookup table at schema resolution time from **writer enum index to reader enum index** using symbol **names**. - Apply this mapping during decode so the produced Arrow dictionary keys always reference the **reader’s** symbol order. - If a writer symbol is not found in the reader enum, surface a clear error. # Are these changes tested? Yes. This PR adds comprehensive **unit tests** for enum mapping in `reader/record.rs` and a **real‑file integration test** in `reader/mod.rs` using `avro/simple_enum.avro`. # Are there any user-facing changes? N/A due to `arrow-avro` not being public yet.

# Which issue does this PR close? \- # Rationale for this change This is apache#4875 now that the upstream changes are available. Allows analysis of TLS traffic with an external tool like Wireshark. See https://wiki.wireshark.org/TLS#using-the-pre-master-secret # What changes are included in this PR? New flag that opts into into the standard `SSLKEYLOGFILE` handling that other libraries and browsers support. # Are these changes tested? Not automatic test, but I did validate that setting the flag AND the env variable emits a log file that is successfully used by Wireshark to decrypt the traffic. # Are there any user-facing changes? Mostly none for normal users, but might be helpful for developers.

# Rationale for this change Update the docstring from function write() in struct Writer to reflect that we write only one RecordBatch at a time as opposed to a vector of record batches. # What changes are included in this PR? Just the comment doc string as above # Are these changes tested? yes # Are there any user-facing changes? No --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Matthijs Brobbel <[email protected]>

# Which issue does this PR close? - Closes apache#8294. # Rationale for this change The .NET implementation is extracted to apache/arrow-dotnet from apache/arrow. apache/arrow will remove `csharp/` eventually. So we should use apache/arrow-dotnet for integration test. # What changes are included in this PR? * Set `ARCHERY_INTEGRATION_WITH_DOTNET=1` to use the .NET implementation * Checkout apache/arrow-dotnet # Are these changes tested? Yes. # Are there any user-facing changes? No.

…pache#8257) # Which issue does this PR close? - Closes apache#8256 . # Rationale for this change Do not compress v2 data page when compress is bad quality ( compressed size is greater or equal to uncompressed_size ) # What changes are included in this PR? Discard compression when it's too large # Are these changes tested? Covered by existing # Are there any user-facing changes? No

# Which issue does this PR close? - Part of apache#4886 # Rationale for this change This refactor streamlines the `arrow-avro` writer by introducing a single, schema‑driven `RecordEncoder` that plans writes up front and encodes rows using consistent, explicit rules for nullability and type dispatch. It reduces duplication in nested/struct/list handling, makes the order of Avro union branches (null‑first vs null‑second) an explicit choice, and aligns header schema generation with value encoding. This should improve correctness (especially for nested optionals), make behavior easier to reason about, and pave the way for future optimizations. # What changes are included in this PR? **High‑level:** * Introduces a unified, schema‑driven `RecordEncoder` with a builder that walks the Avro record in Avro order and maps each field to its Arrow column, producing a reusable write plan. The encoder covers scalars and nested types (struct, (large) lists, maps, strings/binaries). * Applies a single model of **nullability** throughout encoding, including nested sites (list items, fixed‑size list items, map values), and uses explicit union‑branch indices according to the chosen order. **API and implementation details:** * **Writer / encoder refactor** * Replaces the previous per‑column/child encoding paths with a **`FieldPlan`** tree (variants for `Scalar`, `Struct { … }`, and `List { … }`) and per‑site `nullability` carried from the Avro schema. * Adds encoder variants for `LargeBinary`, `Utf8`, `Utf8Large`, `List`, `LargeList`, and `Struct`. * Encodes union branch indices with `write_optional_index` (writes `0x00/0x02` according to Null‑First/Null‑Second), replacing the old branch write. * **Schema generation & metadata** * Moves the **`Nullability`** enum to `schema.rs` and threads it through schema generation and writer logic. * Adds `AvroSchema::from_arrow_with_options(schema, Option<Nullability>)` to either reuse embedded Avro JSON or build new Avro JSON that **honors the requested null‑union order at all nullable sites**. * Adds `extend_with_passthrough_metadata` so Arrow schema metadata is copied into Avro JSON while skipping Avro‑reserved and internal Arrow keys. * Introduces helpers like `wrap_nullable` and `arrow_field_to_avro_with_order` to apply ordering consistently for arrays, fixed‑size lists, maps, structs, and unions. * **Format and glue** * Simplifies `writer/format.rs` by removing the `EncoderOptions` plumbing from the OCF format; `write_long` remains exported for header writing. # Are these changes tested? Yes. * Adds focused unit tests in `writer/encoder.rs` that verify scalar and string/binary encodings (e.g., Binary/LargeBinary, Utf8/LargeUtf8) and validate length/branch encoding primitives used by the writer. * Round trip integration tests that validate List and Struct decoding in `writer/mod.rs`. * Adjusts existing schema tests (e.g., decimal metadata expectations) to align with the new schema/metadata handling. # Are there any user-facing changes? N/A because arrow-avro is not public yet. --------- Co-authored-by: Ryan Johnson <[email protected]> Co-authored-by: Matthijs Brobbel <[email protected]>

# Which issue does this PR close? - Part of apache#4886 # Rationale for this change Apache Avro’s `decimal` logical type annotates either `bytes` or `fixed` and carries `precision` and `scale`. Implementations should reject invalid combinations such as `scale > precision`, and the underlying bytes are the two’s‑complement big‑endian representation of the unscaled integer. On the Arrow side, Rust now exposes first‑class `Decimal32`, `Decimal64`, `Decimal128`, and `Decimal256` data types with documented maximum precisions (9, 18, 38, 76 respectively). Until now, `arrow-avro` decoded all Avro decimals to 128/256‑bit Arrow decimals, even when a narrower type would suffice. # What changes are included in this PR? **`arrow-avro/src/codec.rs`** * Map `Codec::Decimal(precision, scale, _size)` to Arrow’s `Decimal32`/`64`/`128`/`256` **by precision**, preferring the narrowest type (≤9→32, ≤18→64, ≤38→128, otherwise 256). * Strengthen decimal attribute parsing: * Error if `scale > precision`. * Error if `precision` exceeds Arrow’s maximum (Decimal256). * If Avro uses `fixed`, check that declared `precision` fits the byte width (≤4→max 9, ≤8→18, ≤16→38, ≤32→76). * Update docstring of `Codec::Decimal` to mention `Decimal32`/`64`. **`arrow-avro/src/reader/record.rs`** * Add `Decoder::Decimal32` and `Decoder::Decimal64` variants with corresponding builders (`Decimal32Builder`, `Decimal64Builder`). * Builder selection: * If Avro uses **fixed**: choose by size (≤4→Decimal32, ≤8→Decimal64, ≤16→Decimal128, ≤32→Decimal256). * If Avro uses **bytes**: choose by declared precision (≤9/≤18/≤38/≤76). * Implement decode paths that sign‑extend Avro’s two’s‑complement payload to 4/8 bytes and append values to the new builders; update `append_null`/`flush` for 32/64‑bit decimals. **`arrow-avro/src/reader/mod.rs` (tests)** * Expand `test_decimal` to assert that: * bytes‑backed decimals with precision 4 map to `Decimal32`; precision 10 map to `Decimal64`; * legacy fixed\[8] decimals map to `Decimal64`; * fixed\[16] decimals map to `Decimal128`. * Add a nulls path test for bytes‑backed `Decimal32`. # Are these changes tested? Yes. Unit tests under `arrow-avro/src/reader/mod.rs` construct expected `Decimal32Array`/`Decimal64Array`/`Decimal128Array` with `with_precision_and_scale`, and compare against batches decoded from Avro files (including legacy fixed and bytes‑backed cases). The tests also exercise small batch sizes to cover buffering paths; a new Avro data file is added for higher‑width decimals. New Avro test file details: - test/data/int256_decimal.avro # bytes logicalType: decimal(precision=76, scale=10) - test/data/fixed256_decimal.avro # fixed[32] logicalType: decimal(precision=76, scale=10) - test/data/fixed_length_decimal_legacy_32.avro # fixed[4] logicalType: decimal(precision=9, scale=2) - test/data/int128_decimal.avro # bytes logicalType: decimal(precision=38, scale=2) These new Avro test files were created using this script: https://gist.github.com/jecsand838/3890349bdb33082a3e8fdcae3257eef7 There is also an arrow-testing PR for these new files: apache/arrow-testing#112 # Are there any user-facing changes? N/A due to `arrow-avro` not being public.

…t1-merge-main

sdf-jkl and others added 30 commits August 22, 2025 17:18

Update apache-avro requirement from 0.14.0 to 0.20.0 (apache#8226)

d880a01

Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Unpin nightly rust version (MIRI job) (apache#8229)

1dacecb

Reverts: - apache#8183 Because the related issue was closed: - apache#8181

checkpoint

a3e1934

scovich added 5 commits September 4, 2025 14:44

fmt

f365a23

checkpoint

59341f6

checkpoint

60179ce

checkpoint

89b3c75

manual cleanups

f6fd915

github-actions bot added parquet-variant arrow arrow-avro arrow-flight parquet labels Sep 4, 2025

move definition to a better place

a32ccdb

scovich mentioned this pull request Sep 4, 2025

[Variant] Support Shredded Objects in variant_get apache/arrow-rs#8166

Closed

kylebarron and others added 17 commits September 5, 2025 14:37

Bump actions/github-script from 7 to 8 (apache#8287)

ad1f86e

Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bump actions/labeler from 6.0.0 to 6.0.1 (apache#8288)

3ee1d2c

Bumps [actions/labeler](https://github.com/actions/labeler) from 6.0.0 to 6.0.1. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

[Variant] Add as_u* for Variant (apache#8284)

cd676cd

# Which issue does this PR close? - Closes apache#8283. # Rationale for this change Add the `Variant::as_u*` functions` # Are these changes tested? Added doc tests # Are there any user-facing changes? No

remove uneeded allow(unused)

7a66cfb

Merge remote-tracking branch 'apache/main' into shredding-variant-par…

3e4fafc

…t1-merge-main

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Merge arrow-rs/main into shredding-variant-part1 #5

Merge arrow-rs/main into shredding-variant-part1 #5

Uh oh!

scovich commented Sep 4, 2025 •

edited

Loading

Uh oh!

scovich commented Sep 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

Merge arrow-rs/main into shredding-variant-part1 #5

Are you sure you want to change the base?

Merge arrow-rs/main into shredding-variant-part1 #5

Uh oh!

Conversation

scovich commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scovich commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

scovich commented Sep 4, 2025 •

edited

Loading

scovich commented Sep 4, 2025 •

edited

Loading