Skip to content

Commit 02ec652

Browse files
committed
Add iceberg v2 consumer for flattened data consuming, schema will be inferred by arrow
1 parent fe804bf commit 02ec652

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,8 @@ The Iceberg handlers are designed to stream CDC events directly into Apache Iceb
3030
* `IcebergChangeHandler`: A straightforward handler that appends change data to a source-equivalent Iceberg table using a predefined schema.
3131
* **Use Case**: Best for creating a "bronze" layer where you want to capture the raw Debezium event. The `before` and `after` payloads are stored as complete JSON strings.
3232
* **Schema**: Uses a fixed schema where complex nested fields (`source`, `before`, `after`) are stored as `StringType`. It also includes helpful metadata columns (`_consumed_at`, `_dbz_event_key`, `_dbz_event_key_hash`) for traceability.
33-
* With consuming data as json, all source syste schema changes will be absorbed automatically.
33+
* With consuming data as json, all source syste schema changes will be absorbed automatically.
34+
* **Automatic Table Creation & Partitioning**: **It automatically creates a new Iceberg table for each source table** and partitions it by day on the `_consumed_at` timestamp for efficient time-series queries.
3435

3536
* `IcebergChangeHandlerV2`: A more advanced handler that automatically infers the schema from the Debezium events and creates a well-structured Iceberg table accordingly.
3637
* **Use Case**: Ideal for scenarios where you want the pipeline to automatically create tables with native data types that mirror the source. This allows for direct querying of the data without needing to parse JSON.

0 commit comments

Comments
 (0)