supabase · iambriccardo · Aug 14, 2025 · Aug 13, 2025 · Aug 13, 2025 · Aug 13, 2025
@@ -2,24 +2,26 @@
 <p align="center">
   <a href="https://supabase.io">
         <picture>
-      <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/supabase-logo-wordmark--dark.svg">
-      <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/supabase-logo-wordmark--light.svg">
-      <img alt="Supabase Logo" width="300" src="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/logo-preview.jpg">
+      <img alt="Supabase Logo" width="100%" src="res/etl-logo-extended.png">
     </picture>
   </a>
 
-  <h1 align="center">Supabase ETL</h1>
+  <h1 align="center">ETL</h1>
 
   <p align="center">
-    A Rust crate to quickly build replication solutions for Postgres. Build data pipelines which continually copy data from Postgres to other systems.
+    Build real-time Postgres replication applications in Rust
     <br />
-    <a href="https://github.com/supabase/etl/tree/main/etl-examples">Examples</a>
+    <a href="https://supabase.github.io/etl"><strong>Documentation</strong></a>
+    ·
+    <a href="https://github.com/supabase/etl/tree/main/etl-examples"><strong>Examples</strong></a>
+    ·
+    <a href="https://github.com/supabase/etl/issues"><strong>Issues</strong></a>
   </p>
 </p>
 
-# ETL
+**ETL** is a Rust framework by [Supabase](https://supabase.com) that enables you to build high-performance, real-time data replication applications for PostgreSQL. Whether you're creating ETL pipelines, implementing CDC (Change Data Capture), or building custom data synchronization solutions, ETL provides the building blocks you need.
 
-This crate builds abstractions on top of Postgres's [logical streaming replication protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html) and pushes users towards the pit of success without letting them worry about low level details of the protocol.
+Built on top of PostgreSQL's [logical streaming replication protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html), ETL handles the low-level complexities of database replication while providing a clean, Rust-native API that guides you towards the pit of success.
 
 ## Table of Contents
 
@@ -35,27 +37,88 @@ This crate builds abstractions on top of Postgres's [logical streaming replicati
 
 ## Features
 
-The `etl` crate supports the following destinations:
+**Core Capabilities:**
+- 🚀 **Real-time replication**: Stream changes from PostgreSQL as they happen
+- 🔄 **Multiple destinations**: Support for various data warehouses and databases (coming soon)
+- 🛡️ **Fault tolerance**: Built-in error handling, retries, and recovery mechanisms
+- ⚡ **High performance**: Efficient batching and parallel processing
+- 🔧 **Extensible**: Plugin architecture for custom destinations
 
-- [x] BigQuery
-- [ ] Apache Iceberg (planned)
-- [ ] DuckDB (planned)
+**Supported Destinations:**
+- [x] **BigQuery** - Google Cloud's data warehouse
+- [ ] **Apache Iceberg** (planned) - Open table format for analytics
+- [ ] **DuckDB** (planned) - In-process analytical database
 
 ## Installation
 
-To use `etl` in your Rust project, add the core library and desired destinations via git dependencies in `Cargo.toml`:
+Add ETL to your Rust project via git dependencies in `Cargo.toml`:
 
 ```toml
 [dependencies]
 etl = { git = "https://github.com/supabase/etl" }
-etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }
 ```
 
-The `etl` crate provides the core replication functionality, while `etl-destinations` contains destination-specific implementations. Each destination is behind a feature of the same name in the `etl-destinations` crate. The git dependency is needed for now because the crates are not yet published on crates.io.
+> **Note**: ETL is currently distributed via Git while we prepare for the initial crates.io release.
 
 ## Quickstart
 
-To quickly get started with `etl`, see the [etl-examples](etl-examples/README.md) crate which contains practical examples and detailed setup instructions.
+Get up and running with ETL in minutes using the built-in memory destination:
+
+```rust
+use etl::config::{BatchConfig, PgConnectionConfig, PipelineConfig, TlsConfig};
+use etl::pipeline::Pipeline;
+use etl::destination::memory::MemoryDestination;
+use etl::store::both::memory::MemoryStore;
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    // Configure PostgreSQL connection
+    let pg_connection_config = PgConnectionConfig {
+        host: "localhost".to_string(),
+        port: 5432,
+        name: "mydb".to_string(),
+        username: "postgres".to_string(),
+        password: Some("password".into()),
+        tls: TlsConfig {
+            trusted_root_certs: String::new(),
+            enabled: false,
+        },
+    };
+
+    // Configure the pipeline
+    let pipeline_config = PipelineConfig {
+        id: 1,
+        publication_name: "my_publication".to_string(),
+        pg_connection: pg_connection_config,
+        batch: BatchConfig {
+            max_size: 1000,
+            max_fill_ms: 5000,
+        },
+        table_error_retry_delay_ms: 10000,
+        max_table_sync_workers: 4,
+    };
+
+    // Create in-memory store and destination for testing
+    let store = MemoryStore::new();
+    let destination = MemoryDestination::new();
+
+    // Create and start the pipeline
+    let mut pipeline = Pipeline::new(1, pipeline_config, store, destination);
+    pipeline.start().await?;
+
+    Ok(())
+}
+```
+
+**Need production destinations?** Add the `etl-destinations` crate with specific features:
+
+```toml
+[dependencies]
+etl = { git = "https://github.com/supabase/etl" }
+etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }
+```
+
+For comprehensive examples and tutorials, visit the [etl-examples](etl-examples/README.md) crate and our [documentation](https://supabase.github.io/etl).
 
 ## Database Setup
 
@@ -110,3 +173,9 @@ This limits performance for large tables. We plan to address this once the ETL s
 ## License
 
 Distributed under the Apache-2.0 License. See `LICENSE` for more information.
+
+---
+
+<p align="center">
+  Made with ❤️ by the <a href="https://supabase.com">Supabase</a> team
+</p>
@@ -0,0 +1,4 @@
+# ETL Architecture
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Crate Structure
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Design Philosophy
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Explanation
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Performance Model
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Replication Protocol
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Your First Pipeline
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Installation
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Quick Start
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Configure PostgreSQL
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Implement Custom Destinations
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Debug Replication Issues
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# How-to Guides
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Optimize Performance
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Handle Schema Changes
+
+!!! info "Coming Soon"
+    This page is under development.
@@ -0,0 +1,4 @@
+# Set Up Tests
+
+!!! info "Coming Soon"
+    This page is under development.