stellar-pipelines

This repository provides a collection of SQL pipelines for generating Apache Iceberg tables from the Stellar blockchain’s history using AGT. These pipelines leverages ClickHouse and ch-stellar, a library of ClickHouse UDFs that simplifies extraction and processing of Stellar data.

Pipelines

path	description
ledgers	Extracts and normalizes ledger metadata (sequence, close time, base fee, etc.)
transactions	Parses and structures transaction-level data including source, fees, and memo
operations	Extracts all operation types with relevant parameters and participants
accounts	Builds current and historical account state tables
contracts_data	Processes smart contract data and state changes (Soroban)
changes	Captures ledger changes
events	Normalizes emitted Soroban contract events
liquidity_pools	Tracks pool creation, reserves, and share supply
trades	Extracts trade data from offers and path payments
trustlines	Maps trustline relationships and balance changes

Running with AGT

Here we show how to run the ledgers pipeline, sorting the produced table by the sequence column.

agt run \
    --var="ICEBERG_URL=MY_S3_LOCATION" \
    --var="GALEXIE_URL=MY_GALEXIE_URL" \
    --var="RPC_URL=https://stellar-soroban-testnet-public.nodies.app" \
    ../stellar-pipelines/ledgers/pipeline.yaml

Querying

DuckDB

Inside the DuckDB CLI, you can query the data using SQL.

create or replace secret (
    type s3, 
    region 'auto', endpoint 'agnostic.tech'
);

select
    * 
from iceberg_scan('s3://agnostic-stellar-iceberg-mainnet/operations') 
limit 10;

ClickHouse

Inside the ClickHouse CLI, you can query the data using SQL.

select
    * 
from iceberg('https://agnostic-stellar-iceberg-mainnet.agnostic.tech/operations', nosign, settings iceberg_use_version_hint=1) 
limit 10
settings