Repository navigation
Releases
v0.10.0
Adds configurable write compression and pipelines Rust appends for faster bulk uploads.
- Python and Rust writers support
compression="gzip"for gRPC request compression. - Rust writers also support
compression="lz4"andcompression="zstd"for Arrow buffer compression, matching the read APIs. Compression defaults toNone. - Each Rust stream allows up to four outstanding appends, with bounded queues and ordered acknowledgement checks before commit.
- Unsupported codec/engine combinations fail before table creation or overwrite.
- Documentation includes a codec compatibility table and bulk-upload tuning examples. A standalone Rust example probes actual gRPC read-compression negotiation on public datasets.
- Source distributions exclude Rust build outputs, and Python/native package versions are aligned.
options = dict(engine="rust", compression="zstd")
bq.write_table(table, destination, worker_count=8, batch_size=50_000, **options)
result = bq.read_table(destination, **options)Validation includes 177 Python unit tests, 30 Rust tests, live BigQuery round trips for all write codecs and nested/nullable values, and generated 1 GiB and 10 GiB uploads with row-count and ID-sum verification.
The full test suite passed on Python 3.10–3.15: https://github.com/xando/pyarrow-bigquery/actions/runs/42066836675
Known limitation: Rust append-connection failures are reported to the caller and prevent commit; automatic reconnect/replay is not implemented. One 10 GiB LZ4 attempt required a whole-upload retry after its connection closed.
v0.9.0
Adds safe Arrow casting across all four read APIs and both read engines:
cast_column={"field_name": pa.int32()}converts named columns.cast_type={pa.int64(): pa.int32(), pa.list_(pa.int64()): pa.list_(pa.int32())}converts columns with matching original types.- Column rules override type rules. Rules apply once to top-level column types, including explicitly mapped lists.
- Casts preserve metadata and empty-result schemas. Unsupported conversions, overflow, and precision loss raise errors.
This release also includes the Rust write engine introduced since the last PyPI release, 0.7.3.
Validation: the full CI suite passed on Python 3.10–3.15, including 147 unit tests and 28 live BigQuery casting tests. Lint and local type checks also passed.