Skip to content

Commit 0176c3c

Browse files
authored
feat(branding): Add new logo and update docs (#263)
1 parent cf81f4f commit 0176c3c

33 files changed

+342
-223
lines changed

README.md

Lines changed: 85 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,24 +2,26 @@
22
<p align="center">
33
<a href="https://supabase.io">
44
<picture>
5-
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/supabase-logo-wordmark--dark.svg">
6-
<source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/supabase-logo-wordmark--light.svg">
7-
<img alt="Supabase Logo" width="300" src="https://raw.githubusercontent.com/supabase/supabase/master/packages/common/assets/images/logo-preview.jpg">
5+
<img alt="Supabase Logo" width="100%" src="res/etl-logo-extended.png">
86
</picture>
97
</a>
108

11-
<h1 align="center">Supabase ETL</h1>
9+
<h1 align="center">ETL</h1>
1210

1311
<p align="center">
14-
A Rust crate to quickly build replication solutions for Postgres. Build data pipelines which continually copy data from Postgres to other systems.
12+
Build real-time Postgres replication applications in Rust
1513
<br />
16-
<a href="https://github.com/supabase/etl/tree/main/etl-examples">Examples</a>
14+
<a href="https://supabase.github.io/etl"><strong>Documentation</strong></a>
15+
·
16+
<a href="https://github.com/supabase/etl/tree/main/etl-examples"><strong>Examples</strong></a>
17+
·
18+
<a href="https://github.com/supabase/etl/issues"><strong>Issues</strong></a>
1719
</p>
1820
</p>
1921

20-
# ETL
22+
**ETL** is a Rust framework by [Supabase](https://supabase.com) that enables you to build high-performance, real-time data replication applications for PostgreSQL. Whether you're creating ETL pipelines, implementing CDC (Change Data Capture), or building custom data synchronization solutions, ETL provides the building blocks you need.
2123

22-
This crate builds abstractions on top of Postgres's [logical streaming replication protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html) and pushes users towards the pit of success without letting them worry about low level details of the protocol.
24+
Built on top of PostgreSQL's [logical streaming replication protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html), ETL handles the low-level complexities of database replication while providing a clean, Rust-native API that guides you towards the pit of success.
2325

2426
## Table of Contents
2527

@@ -35,27 +37,88 @@ This crate builds abstractions on top of Postgres's [logical streaming replicati
3537

3638
## Features
3739

38-
The `etl` crate supports the following destinations:
40+
**Core Capabilities:**
41+
- 🚀 **Real-time replication**: Stream changes from PostgreSQL as they happen
42+
- 🔄 **Multiple destinations**: Support for various data warehouses and databases (coming soon)
43+
- 🛡️ **Fault tolerance**: Built-in error handling, retries, and recovery mechanisms
44+
-**High performance**: Efficient batching and parallel processing
45+
- 🔧 **Extensible**: Plugin architecture for custom destinations
3946

40-
- [x] BigQuery
41-
- [ ] Apache Iceberg (planned)
42-
- [ ] DuckDB (planned)
47+
**Supported Destinations:**
48+
- [x] **BigQuery** - Google Cloud's data warehouse
49+
- [ ] **Apache Iceberg** (planned) - Open table format for analytics
50+
- [ ] **DuckDB** (planned) - In-process analytical database
4351

4452
## Installation
4553

46-
To use `etl` in your Rust project, add the core library and desired destinations via git dependencies in `Cargo.toml`:
54+
Add ETL to your Rust project via git dependencies in `Cargo.toml`:
4755

4856
```toml
4957
[dependencies]
5058
etl = { git = "https://github.com/supabase/etl" }
51-
etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }
5259
```
5360

54-
The `etl` crate provides the core replication functionality, while `etl-destinations` contains destination-specific implementations. Each destination is behind a feature of the same name in the `etl-destinations` crate. The git dependency is needed for now because the crates are not yet published on crates.io.
61+
> **Note**: ETL is currently distributed via Git while we prepare for the initial crates.io release.
5562
5663
## Quickstart
5764

58-
To quickly get started with `etl`, see the [etl-examples](etl-examples/README.md) crate which contains practical examples and detailed setup instructions.
65+
Get up and running with ETL in minutes using the built-in memory destination:
66+
67+
```rust
68+
use etl::config::{BatchConfig, PgConnectionConfig, PipelineConfig, TlsConfig};
69+
use etl::pipeline::Pipeline;
70+
use etl::destination::memory::MemoryDestination;
71+
use etl::store::both::memory::MemoryStore;
72+
73+
#[tokio::main]
74+
async fn main() -> Result<(), Box<dyn std::error::Error>> {
75+
// Configure PostgreSQL connection
76+
let pg_connection_config = PgConnectionConfig {
77+
host: "localhost".to_string(),
78+
port: 5432,
79+
name: "mydb".to_string(),
80+
username: "postgres".to_string(),
81+
password: Some("password".into()),
82+
tls: TlsConfig {
83+
trusted_root_certs: String::new(),
84+
enabled: false,
85+
},
86+
};
87+
88+
// Configure the pipeline
89+
let pipeline_config = PipelineConfig {
90+
id: 1,
91+
publication_name: "my_publication".to_string(),
92+
pg_connection: pg_connection_config,
93+
batch: BatchConfig {
94+
max_size: 1000,
95+
max_fill_ms: 5000,
96+
},
97+
table_error_retry_delay_ms: 10000,
98+
max_table_sync_workers: 4,
99+
};
100+
101+
// Create in-memory store and destination for testing
102+
let store = MemoryStore::new();
103+
let destination = MemoryDestination::new();
104+
105+
// Create and start the pipeline
106+
let mut pipeline = Pipeline::new(1, pipeline_config, store, destination);
107+
pipeline.start().await?;
108+
109+
Ok(())
110+
}
111+
```
112+
113+
**Need production destinations?** Add the `etl-destinations` crate with specific features:
114+
115+
```toml
116+
[dependencies]
117+
etl = { git = "https://github.com/supabase/etl" }
118+
etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }
119+
```
120+
121+
For comprehensive examples and tutorials, visit the [etl-examples](etl-examples/README.md) crate and our [documentation](https://supabase.github.io/etl).
59122

60123
## Database Setup
61124

@@ -110,3 +173,9 @@ This limits performance for large tables. We plan to address this once the ETL s
110173
## License
111174

112175
Distributed under the Apache-2.0 License. See `LICENSE` for more information.
176+
177+
---
178+
179+
<p align="center">
180+
Made with ❤️ by the <a href="https://supabase.com">Supabase</a> team
181+
</p>

docs/assets/etl.png

10.1 KB
Loading

docs/assets/etl.svg

Lines changed: 0 additions & 3 deletions
This file was deleted.

docs/design/etl-crate-design.md

Lines changed: 0 additions & 60 deletions
This file was deleted.

docs/design/index.md

Whitespace-only changes.

docs/explanation/architecture.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# ETL Architecture
2+
3+
!!! info "Coming Soon"
4+
This page is under development.

docs/explanation/crate-structure.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Crate Structure
2+
3+
!!! info "Coming Soon"
4+
This page is under development.

docs/explanation/design.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Design Philosophy
2+
3+
!!! info "Coming Soon"
4+
This page is under development.

docs/explanation/index.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Explanation
2+
3+
!!! info "Coming Soon"
4+
This page is under development.

docs/explanation/performance.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Performance Model
2+
3+
!!! info "Coming Soon"
4+
This page is under development.

0 commit comments

Comments
 (0)