Feature: Real-time ETL

[ChenpiDog]

Real-time Data Warehouse and real-time ETL can be realized only with SQL.

[cdmikechen]

This seems like a great idea, but I'd actually like to focus more on how the databend allows the next level to sense changes in the data on the upper level.

[ChenpiDog]

A mechanism like Change Data Capture (CDC)? It may involve technologies such as database logs and materialized views.

[cdmikechen]

@ChenpiDog
My personal understanding of parquet is that it is a columnar storage format and probably won't handle CDC as well as rdbms.

[bohutang]

@cdmikechen
We will handle the CDC in fuse engine(like the Merge statement), now it supports insert/select/delete, and the next plan is to support updates, all in one.