Skip to content

Releases

v0.10.0

Choose a tag to compare

@xando xando released this 05 Nov 16:55

Adds configurable write compression and pipelines Rust appends for faster bulk uploads.

  • Python and Rust writers support compression="gzip" for gRPC request compression.
  • Rust writers also support compression="lz4" and compression="zstd" for Arrow buffer compression, matching the read APIs. Compression defaults to None.
  • Each Rust stream allows up to four outstanding appends, with bounded queues and ordered acknowledgement checks before commit.
  • Unsupported codec/engine combinations fail before table creation or overwrite.
  • Documentation includes a codec compatibility table and bulk-upload tuning examples. A standalone Rust example probes actual gRPC read-compression negotiation on public datasets.
  • Source distributions exclude Rust build outputs, and Python/native package versions are aligned.
options = dict(engine="rust", compression="zstd")
bq.write_table(table, destination, worker_count=8, batch_size=50_000, **options)
result = bq.read_table(destination, **options)

Validation includes 177 Python unit tests, 30 Rust tests, live BigQuery round trips for all write codecs and nested/nullable values, and generated 1 GiB and 10 GiB uploads with row-count and ID-sum verification.

The full test suite passed on Python 3.10–3.15: https://github.com/xando/pyarrow-bigquery/actions/runs/42066836675

Known limitation: Rust append-connection failures are reported to the caller and prevent commit; automatic reconnect/replay is not implemented. One 10 GiB LZ4 attempt required a whole-upload retry after its connection closed.

v0.9.0

Choose a tag to compare

@xando xando released this 30 Oct 08:44

Adds safe Arrow casting across all four read APIs and both read engines:

  • cast_column={"field_name": pa.int32()} converts named columns.
  • cast_type={pa.int64(): pa.int32(), pa.list_(pa.int64()): pa.list_(pa.int32())} converts columns with matching original types.
  • Column rules override type rules. Rules apply once to top-level column types, including explicitly mapped lists.
  • Casts preserve metadata and empty-result schemas. Unsupported conversions, overflow, and precision loss raise errors.

This release also includes the Rust write engine introduced since the last PyPI release, 0.7.3.

Validation: the full CI suite passed on Python 3.10–3.15, including 147 unit tests and 28 live BigQuery casting tests. Lint and local type checks also passed.