CHANGES.md

Changelog

Unreleased

• Testing: CrateDBContainer builds on the CrateDB module of Testcontainers for Python. The testing extra requires testcontainers[cratedb]>=4.15,<4.16. CrateDBContainer.CMD_OPTS holds key-value pairs instead of a dict, and the cmd_opts argument still accepts both. Waiting for the HTTP interface is bounded by CrateDBContainer.STARTUP_TIMEOUT, 60 seconds.
• Fixed ctk cfr sys-export stalling on large clusters. It exports the 1000 most recent entries of sys.jobs_log and sys.operations_log, a page at a time, and --log-limit sets another number.
• ctk cfr sys-export waits at most 120 seconds for a response. A log table that fails mid-way keeps the entries already read, listed in manifest.json under data_partial.
• ctk cfr sys-export names the table it is reading in its progress bar, and logs each read's row count, size and duration at debug level.
• Fixed ctk cfr sys-import losing the rows of sys.segments and sys.sessions, of nodes with a dot in an attribute name, and of users with the memory.operation_limit session setting, while reporting success. Thanks, @hammerhead.
• ctk cfr sys-export declares object columns with dotted keys as OBJECT(IGNORED), and text columns that can outgrow an index entry without an index, so CrateDB accepts their values on import. Bundles written by earlier releases lack these declarations: export the bundle again, or correct the .sql file named in the error message.
• Breaking change: ctk cfr sys-import exits non-zero when a table is not fully restored. For each such table, it logs how many rows arrived and the error messages CrateDB returned.
• ctk cfr sys-import drops and recreates each table from the bundle's definition, so a table exported without rows is emptied instead of keeping the rows of an earlier import.
• ctk --debug cfr sys-import reports failures with a traceback.

2026/08/17 v0.1.0

• Fixed ctk cfr jobstats bugs related anonymization, views, ui, report and collect arguments.
• Breaking change: ctk info jobs reported the 99th percentile of the query duration as p90. The field of the top100_count element is called p99 now.
• Documentation: Explained ctk info jobs and ctk cfr jobstats, and how they differ
• Breaking change: Dropped support for Python 3.8 and 3.9, which have reached end-of-life. The minimum supported Python version is now 3.10.
• Fixed a failing SQL statement during ctk info cluster
• Dependency: Update sqlalchemy-crate version to 0.43.1
• Removed support for Rockset
• Fixed a bug in cfr jobstats ui where slider will pick an initial value too large
• Breaking change: Dropped the ingestr dependency and, with it, the generic SQL-source ingest path. The purpose-built I/O adapters: Kinesis (kinesis+dms, kinesis+dynamodb), DynamoDB, InfluxDB, MongoDB, DeltaLake, and Iceberg are unaffected. The io-ingest extra no longer pins sqlalchemy<2, so it no longer needs to be mutually exclusive with io/io-curated for that reason.
• OCI: The cratedb-toolkit-ingest image no longer bundles the ODBC and Oracle Instant Client drivers.

2026/06/17 v0.0.49

• Stopped leaking password to log output in ctk cfr jobstats collect. Thanks, @hammerhead.
• Stopped failing ctk cfr jobstats ui when job statistics recordings are empty. Thanks, @hammerhead.
• Fixed ctk cfr info record re. Document contains at least one immense term in field. Thanks, @hammerhead.
• Fixed ctk info cluster to report correct shard rebalancing progress. Thanks, @hammerhead.
• Attempted to improve ctk cfr sys-export on very active/large clusters by increasing infer_schema_length from 1_000 to 100_000. Thanks, @hammerhead.

2026/05/12 v0.0.48

• OCI: Installed curl, for Compose health checks
• OCI: Installed Supercronic, for scheduling periodic tasks
• Added session_id attribute to SysJobsLog, to accompany newer CrateDB versions
• I/O: Added option to submit workloads to remote Dask clusters

2026/04/27 v0.0.47

• Kinesis: Added ctk kinesis CLI group with list-checkpoints and prune-checkpoints commands for checkpoint table maintenance. Thanks, @hampsterx.
• Dependencies: Permitted installation of click 8.3
• DataQuery: Help agents turn natural language into SQL queries

2026/03/16 v0.0.46

• I/O: API improvements: ctk {load,save} table became ctk {load,save} Also, the --cluster-url option becomes optional. That's a much more concise interface now.

2026/03/13 v0.0.45

• I/O: Refactored ctk.io subsystem
• I/O: Refurbished documentation of the I/O subsystem
• I/O: Started using ingestr for loading data from Amazon Kinesis
• I/O: Started propagating start_date parameter to ingestr's interval_start
• I/O: Started propagating batch_size parameter to ingestr's page_size
• Packaging: Renamed extras all → full, io-ingestr → io-ingest, and io-all → io-curated

2026/03/04 v0.0.44

• I/O: Added adapter for Apache Iceberg tables
• I/O: Added adapter for Delta Lake tables

2026/02/18 v0.0.43

• Retention: Validated integration with Azure blob storage using Azurite
• Fixed missing pandas package that prevented installation/execution without extras

2026/02/11 v0.0.42

• Settings: Stop flagging gateway.recover_after_time as a difference when both gateway.expected_nodes and gateway.expected_data_nodes are unset (-1).
• PyMongo adapter: Adapted to CrateDB 6.2+ which disallows writes to _id columns
• Dependencies: Permitted installation of pyarrow 23
• Dependencies: Permitted installation of duckdb 1.x
• Dependencies: Permitted installation of pandas 3.0
• CI: Validated against Python 3.14
• CI: Validated against InfluxDB 2.8
• CI: Validated against MongoDB 8

2025/08/19 v0.0.41

• I/O: Updated to influxio-0.6.0. Thanks, @ZillKhan.
  • The target table is now the measurement name when importing ILP files.
  • The InfluxDB source URL accepts a timeout query parameter (seconds) to configure the network timeout when talking to the InfluxDB API.
  • For ILP imports, the CrateDB URL no longer needs a table component; you can point it at the schema only (the measurement determines the table).

2025/08/19 v0.0.40

• I/O: Fixed MongoDB CDC invocation. Thanks, Mỹ Duyên.

2025/08/19 v0.0.39

• OCI: Started producing image ghcr.io/crate/cratedb-toolkit-ingest
• I/O: Added drivers for ODBC and Oracle to cratedb-toolkit-ingest
• I/O: Updated BSON library to support ARM64

2025/08/14 v0.0.38

• I/O: Updated to ingestr>=0.13.61
• CFR: Improved log output
• CFR: Fixed double quoting of table name. Thanks, @karynzv.
• CFR: When importing, started using replace strategy instead of append
• CFR: Improved importing data re. type mapping without NumPy
• CFR: Truncated target table before importing, using append strategy again, because replace doesn't do the right DDL.
• I/O: Tuned down ingestr, it masked native I/O adapters

2025/07/01 v0.0.37

• Settings: Fixed comparison of 0s vs 0ms. Thanks, @hlcianfagna.
• DMS: Provided a recipe file to relay primary key and column type map information
• DMS: Provided a recipe option to ignore processing DMS control DDL events
• DMS: Started using the "direct" column mapping by default, retaining the "universal" column mapping optionally.
• Dependencies: Updated to commons-codec>=0.0.23
• I/O: Adapter for PostgreSQL full-load using ingestr
• I/O: Added documentation about ingestr adapter

2025/06/23 v0.0.36

• Dependencies: Migrated from zyp to tikray. It's effectively the same, but provided using a dedicated package now
• Dependencies: Updated to croud-1.14
• Dependencies: Updated to async-kinesis-2.0.0. Thanks, @hampsterx.
• CDC: Added canonical SQL example for PostgreSQL from Ibis
• CDC: Enabled loading DMS events from Kinesis streams and stream-dump files
• CDC: Added subcommand ctk dms table-mappings

2025/05/13 v0.0.35

• Added lost pytest dependencies to cratedb-toolkit[testing]

2025/05/13 v0.0.34

• Downgraded to sqlalchemy-cratedb 0.41, version 0.42 is not GA yet

2025/05/12 v0.0.33

• CFR: Enhanced job statistics with optional reporting database support. Thanks, @WalBeh.
• Settings: Added settings comparison utility. Thanks, @WalBeh.
• Meta: Added parser for https://cratedb.com/releases.json file. Thanks, @WalBeh.
• CFR: Added the ability to anonymize queries recorded by collect
• Cloud API: SDK and CLI for CrateDB Cloud Cluster and Import APIs. Supports headless/unattended operations on CrateDB Cloud clusters, covering deploy/start/resume and data import procedures using fluent API and CLI.
• Cloud API: Added JWT authentication to client API and ctk shell.
• Cloud API: Added health and ping subcommands to ctk cluster
• CLI: Downgraded to Click 8.1, as the code is not compatible with 8.2 yet

Breaking changes

Naming things for CLI options and environment variables:

• Converged --cratedb-sqlalchemy-url vs. --cratedb-http-url options into single --cluster-url
• Converged CRATEDB_SQLALCHEMY_URL vs. CRATEDB_HTTP_URL env vars into single CRATEDB_CLUSTER_URL

2025/04/23 v0.0.32

• MCP: Add subsystem providing a few server and client utilities through the ctk query mcp {list,inquire,launch} subcommands.
• Docs API: Added extractors for CrateDB functions and settings
• Connect: Respect sslmode URI parameter when converting SQLAlchemy connection URLs to http(s)://

2025/01/31 v0.0.31

• Fixed connectivity for jobstats collect
• Refactored code and improved CLI interface of ctk info vs. ctk cfr
• Dependencies: Updated to crate-2.0.0, which uses orjson for JSON marshalling
• CFR: Job statistics and slow-query exploration per Marimo notebook. Thanks, @WalBeh.

2025/01/13 v0.0.30

• Dependencies: Minimize dependencies of core installation, defer polars to cratedb-toolkit[io].
• Fixed ctk cfr info record about too large values of ulimit_hard
• Improved ctk shell to also talk to CrateDB standalone databases
• Added basic utility command ctk tail, for tailing a database table, and optionally following the tail
• Table Loader: Added capability to load InfluxDB Line Protocol (ILP) files
• Query Collector: Now respects CRATEDB_CLUSTER_URL environment variable

2024/10/13 v0.0.29

• MongoDB: Added Zyp transformations to the CDC subsystem, making it more symmetric to the full-load procedure.
• Query Converter: Added very basic expression converter utility with CLI interface
• DynamoDB: Added query expression converter for relocating object references, to support query migrations after the breaking change with the SQL DDL schema, by v0.0.27.

2024/10/09 v0.0.28

• IO: Improved BulkProcessor when running per-record operations by also checking rowcount for handling INSERT OK, 0 rows responses
• MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps by updating to commons-codec 0.0.21.
• Testcontainers: Don't always pull the OCI image before starting. It is unfortunate in disconnected situations.

2024/10/01 v0.0.27

• MongoDB: Updated to pymongo 4.9
• DynamoDB: Change CrateDB data model to use (pk, data, aux) columns Attention: This is a breaking change.

2024/09/26 v0.0.26

• MongoDB: Configure MongoDBCrateDBConverter after updating to commons-codec 0.0.18
• DynamoDB CDC: Fix MODIFY operation to also propagate deleted attributes

2024/09/22 v0.0.25

• Table Loader: Improved conditional handling of "transformation" parameter
• Table Loader: Improved status reporting and error logging in BulkProcessor
• MongoDB: Improve error reporting
• MongoDB Full: Polars' read_ndjson doesn't load MongoDB JSON data well, use fsspec and orjson instead
• MongoDB Full: Improved initialization of transformation subsystem
• MongoDB Adapter: Improved performance of when computing collection cardinality by using collection.estimated_document_count()
• MongoDB Full: Optionally use limit parameter as number of total records
• MongoDB Adapter: Evaluate _id filter field by upcasting to bson.ObjectId, to convey a filter that makes ctk load table process a single document, identified by its OID
• MongoDB Dependencies: Update to commons-codec 0.0.17

2024/09/19 v0.0.24

• MongoDB Full: Refactor transformation subsystem to commons-codec
• MongoDB: Update to commons-codec v0.0.16

2024/09/16 v0.0.23

• MongoDB: Unlock processing multiple collections, either from server database, or from filesystem directory
• MongoDB: Unlock processing JSON files from HTTP resource, using https+bson://
• MongoDB: Optionally filter server collection using MongoDB query expression
• MongoDB: Improve error handling wrt. bulk operations vs. usability
• DynamoDB CDC: Add ctk load table interface for processing CDC events
• DynamoDB CDC: Accept a few more options for the Kinesis Stream: batch-size, create, create-shards, start, seqno, idle-sleep, buffer-time
• DynamoDB Full: Improve error handling wrt. bulk operations vs. usability

2024/09/10 v0.0.22

• MongoDB: Rename columns with leading underscores to use double leading underscores
• MongoDB: Add support for UUID types
• MongoDB: Improve reading timestamps in previous BSON formats
• MongoDB: Fix processing empty arrays/lists. By default, assume TEXT as inner type.
• MongoDB: For ctk load table, use "partial" scan for inferring the collection schema, based on the first 10,000 documents.
• MongoDB: Skip leaking UNKNOWN fields into SQL DDL. This means relevant column definitions will not be included into the SQL DDL.
• MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.
• MongoDB: Sanitize lists of varying objects
• MongoDB: Add treatment option for applying special treatments to certain items on real-world data
• MongoDB: Use pagination on source collection, for creating batches towards CrateDB
• MongoDB: Unlock importing MongoDB Extended JSON files using file+bson://...

2024/09/02 v0.0.21

• DynamoDB: Add special decoding for varied lists. Store them into a separate OBJECT(IGNORED) column in CrateDB.
• DynamoDB: Add pagination support for full-load table loader

2024/08/27 v0.0.20

• DMS/DynamoDB: Fix table name quoting within CDC processor handler

2024/08/26 v0.0.19

• MongoDB: Fix and verify Zyp transformations
• DMS/DynamoDB/MongoDB I/O: Use SQL with parameters instead of inlining values

2024/08/21 v0.0.18

• Dependencies: Unpin commons-codec, to always use the latest version
• Dependencies: Unpin lorrystream, to always use the latest version
• MongoDB: Improve type mapper by discriminating between INTEGER and BIGINT
• MongoDB: Improve type mapper by supporting BSON DatetimeMS, Decimal128, and Int64 types

2024/08/19 v0.0.17

• Processor: Updated Kinesis Lambda processor to understand AWS DMS
• MongoDB: Fix missing output on STDOUT for migr8 export
• MongoDB: Improve timestamp parsing by using python-dateutil
• MongoDB: Converge _id input field to id column instead of dropping it
• MongoDB: Make user interface use stderr, so stdout is for data only
• MongoDB: Make migr8 extract write to stdout by default
• MongoDB: Make migr8 translate read from stdin by default
• MongoDB: Improve user interface messages
• MongoDB: Strip single leading underscore character from all top-level fields
• MongoDB: Map OID types to CrateDB TEXT columns
• MongoDB: Make migr8 extract and migr8 export accept the --limit option
• MongoDB: Fix indentation in prettified SQL output of migr8 translate
• MongoDB: Add capability to give type hints and add transformations
• Dependencies: Adjust code for lorrystream version 0.0.3
• Dependencies: Update to lorrystream 0.0.4 and commons-codec 0.0.7
• DynamoDB: Add table loader for full-load operations

2024/07/25 v0.0.16

• ctk load table: Added support for MongoDB Change Streams
• Fix dependency with the kaggle package, downgrade to kaggle==1.6.14
• DynamoDB CDC: Add demo to support reading DynamoDB change data capture

2024/07/08 v0.0.15

• IO: Added the if-exists query parameter by updating to influxio 0.4.0.
• Rockset: Added CrateDB Rockset Adapter, a HTTP API emulation layer
• MongoDB: Added adapter amalgamating PyMongo to use CrateDB as backend
• SQLAlchemy: Clean up and refactor SQLAlchemy polyfills to cratedb_toolkit.util.sqlalchemy
• CFR: Build as a self-contained program using PyInstaller
• CFR: Publish self-contained application bundle to GitHub Workflow Artifacts

2024/06/18 v0.0.14

• Add ctk info and ctk cfr diagnostics programs
• Remove support for Python 3.7
• SQLAlchemy dialect: Use sqlalchemy-cratedb>=0.37.0 This includes the fix to the get_table_names() reflection method.

2024/06/11 v0.0.13

• Dependencies: Migrate from crate[sqlalchemy] to sqlalchemy-cratedb

2024/05/30 v0.0.12

• Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by using ssl=true query argument also for influxdb2:// source URLs.

2024/05/30 v0.0.11

• Fix InfluxDB Cloud <-> CrateDB Cloud connectivity by propagating ssl=true query argument. Update dependencies to influxio>=0.2.1,<1.

2024/04/10 v0.0.10

• Dependencies: Unpin upper version bound of dask. Otherwise, compatibility issues can not be resolved quickly, like with Python 3.11.9. dask/dask#11038

2024/03/22 v0.0.9

• Dependencies: Use dask[dataframe]

2024/03/11 v0.0.8

• datasets: Fix compatibility with Python 3.7

2024/03/07 v0.0.7

• datasets: Fix dataset loader

2024/03/07 v0.0.6

• Added cratedb_toolkit.datasets subsystem, for acquiring datasets from cratedb-datasets and Kaggle.

2024/02/12 v0.0.5

• Do not always activate pytest11 entrypoint to pytest fixture cratedb_service, as it depends on the testcontainers package, which is not always installed.

2024/02/10 v0.0.4

• Packaging: Use cloud extra to install relevant packages
• Dependencies: Add testing extra, which installs testcontainers only
• Testing: Export cratedb_service fixture as pytest11 entrypoint
• Sandbox: Reduce number of extras by just using all

2024/01/18 v0.0.3

• Add SQL runner utility primitives to io.sql namespace
• Add import_csv_pandas and import_csv_dask utility primitives
• data: Add subsystem for "loading" data.
• Add SDK and CLI for CrateDB Cloud Data Import APIs ctk load table ...
• Add migr8 program from previous repository
• InfluxDB: Add adapter for influxio
• MongoDB: Add migr8 program from previous repository
• MongoDB: Improve UX by using ctk load table mongodb://...
• load table: Refactor to use more OO
• Add examples/cloud_import.py
• Adapt testcontainers to be agnostic of the testing framework. Thanks, @pilosus.

2023/11/06 v0.0.2

• CLI: Upgrade to click-aliases>=1.0.2, fixing erroring out when no group aliases are specified.

• Add support for Python 3.12

• SQLAlchemy: Improve UNIQUE constraints polyfill to accept multiple column names, for emulating unique composite keys.

2023/10/10 v0.0.1

• SQLAlchemy: Add a few patches and polyfills, which do not fit well into the vanilla Python driver / SQLAlchemy dialect.

• Retention: Refactor strategies delete, reallocate, and snapshot, to standalone variants.

• Retention: Bundle configuration and runtime settings into Settings entity, and use more OO instead of weak dictionaries: Add RetentionStrategy, TableAddress, and Settings entities, to improve information passing throughout the application and the SQL templates.

• Retention: Add --schema option, and CRATEDB_EXT_SCHEMA environment variable, to configure the database schema used to store the retention policy table. The default value is ext.

• Retention: Use full-qualified table names everywhere.

• Retention: Fix: Compensate for DROP REPOSITORY now returning RepositoryMissingException when the repository does not exist. With previous versions of CrateDB, it was RepositoryUnknownException.

2023/06/27 v0.0.0

• Import "data retention" implementation from https://github.com/crate/crate-airflow-tutorial. Thanks, @hammerhead.