Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1,324 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Rocky

Engine CI SDK CI Dagster CI VS Code CI License: Apache 2.0

Rocky checks your whole SQL pipeline before it runs, and tells you what a change will break.

Rocky works with Databricks, Snowflake, BigQuery, and DuckDB. You keep your warehouse and your existing SQL. Apache 2.0.

The failures that cost the most are the quiet ones. A source column changes type. Someone renames a column and three models stop working. A query works in dev and fails in production. Rocky finds all of these at check time.

   you edit SQL          rocky compile              rocky run
        │                      │                        │
        ▼                      ▼                        ▼
   ┌─────────┐        ┌──────────────────┐        ┌───────────┐
   │  model  │───────►│  check the whole │───────►│ warehouse │
   │  files  │        │  pipeline: types,│        │  writes   │
   └─────────┘        │  refs, contracts │        └───────────┘
                      └──────────────────┘
                               │
                               │ a problem is found here,
                               ▼ so nothing runs
                        E010: required column
                        `order_id` is missing

Rocky quickstart: create a project, compile, and run 3 models in under 15s

Try it in 60 seconds

# macOS / Linux
curl -fsSL https://raw.githubusercontent.com/rocky-data/rocky/main/engine/install.sh | bash

# Windows (PowerShell)
irm https://raw.githubusercontent.com/rocky-data/rocky/main/engine/install.ps1 | iex
rocky playground my-first-project
cd my-first-project
rocky compile && rocky test && rocky run

No credentials needed — the playground runs on local DuckDB.

For production deploys, use rocky plan (saves what will change) then rocky apply <plan-id> (runs it). For local work and automation, rocky run does it all in one step.

Who Rocky is for

Rocky is built first for data engineers on Databricks, where a silent failure costs real money and Dagster runs the schedule. The Snowflake and BigQuery adapters are in Beta. See Where Rocky is today.

See it in action

Each demo is in examples/playground/pocs/. Change into a demo directory and run ./run.sh.

See what breaks before you merge, with rocky lineage-diff

Compare two versions of your project. Rocky lists the downstream tables and columns that each change affects. Paste the list into a GitHub pull request comment.

rocky lineage-diff main lists added and removed columns across two models with downstream consumers per change

POC: 06-developer-experience/11-lineage-diff

More demos

  • Schema drift recovery: source column type changes upstream; Rocky detects it and rebuilds safely.
  • Data contracts: missing required columns, dropped protected columns, or unsafe type changes surface as errors (E010, E011, E013) before a row is written.
  • BigQuery cost to the byte: bytes_scanned in the run receipt matches BigQuery's billing number exactly (requires credentials).
  • Named branches + replay: run against an isolated schema copy, inspect, then drop or promote.
  • Agent policy: a [policy] block grades what an agent may do on its own; pinned scenarios catch a loosened rule in CI.
  • Column lineage: trace a column in a downstream model back to its source.
  • Incremental loads: set strategy = "incremental" and Rocky only processes new rows each run.
  • Data masking: tag PII columns, set masking per environment, fail the check if anything goes out unmasked.
  • AI model generation: describe what you want; Rocky writes the SQL, checks it, and retries if something's wrong.

In your editor

The checker runs as a language server in VS Code. You see type mismatches and broken references while you write, not later in CI. Column types show when you hover. Go-to-definition works across all your models.

The Rocky Inspector shows a model's columns, where each column came from, its tests, its cost, and which columns hold sensitive data.

The Rocky Inspector's Overview as a model trust dashboard, its Governance card flagging two classified columns with one left unmasked

Install the VS Code extension →

When an AI agent writes your pipelines

Agents now write real pipeline changes. An agent that is trusted too much, with production access, can destroy real data in seconds. Rocky treats an agent as an operator with a controlled path to production.

Rocky type-checks every change an agent writes. The agent produces a plan. A plan never applies itself. It must first pass the rules you wrote, and every decision goes into a ledger you can query.

   an agent drafts a change
              │
              ▼
   ┌─────────────────────┐
   │ compiler            │   types and contracts are checked
   │                     │   as the agent writes
   └──────────┬──────────┘
              ▼
   ┌─────────────────────┐
   │ plan                │   a plan never applies itself
   └──────────┬──────────┘
              ▼
   ┌─────────────────────┐
   │ rocky apply reads   │   this gate runs before any
   │ your [policy] rules │   SQL reaches the warehouse
   └──────────┬──────────┘
              │
     ┌────────┴────────┬──────────────────┐
     │ require review  │ allow            │ deny
     ▼                 │                  ▼
  ┌──────────────┐     │        ┌───────────────────┐
  │ a human      │     │        │ refused. No SQL   │
  │ approves,    │     │        │ runs, so there is │
  │ then applies │     │        │ nothing to undo.  │
  └──────┬───────┘     │        └─────────┬─────────┘
         │             │                  │
         └──────┬──────┘                  │
                ▼                         │
     ┌────────────────────┐               │
     │ the warehouse runs │               │
     │ the plan           │               │
     └─────────┬──────────┘               │
               ▼                          │
     ┌──────────────────────────┐         │
     │ a rule can require that  │         │
     │ named checks passed here │         │
     └─────────┬────────────────┘         │
               │                          │
               └────────────┬─────────────┘
                            ▼
              ┌──────────────────────────────┐
              │ every decision lands here:   │
              │ rocky audit · rocky brief    │
              └──────────────────────────────┘

The diagram shows the gate at rocky apply. The MCP draft and propose tools read the same rules earlier, before Rocky keeps a file or a plan. Rocky leaves no new file for a denied draft, and writes no plan for a denied proposal.

A rule can also name checks that must pass in that run. If one fails, or never ran, Rocky stops and records the failure. It cannot undo the write: the change stays until a human reverts it.

  • You write the rules in rocky.toml. A [policy] rule says what each principal may do, and where. The answer is allow, require review, or deny. A rule can also set a ceiling on how far a change may reach, with max_downstreams. Rocky downgrades that rule's allow to require review when a change exceeds the ceiling, or when it cannot work out the reach. You can test the rules: [[policy.tests]] scenarios run through the real evaluator, so rocky policy test in CI catches an edit that would have opened a hole.
  • A plan written by AI waits for a human. The agent proposes. rocky apply refuses an unapproved AI-authored plan unless one of your [policy] rules grants that scope. The engine enforces this. It is not a convention you can forget.
  • You can ask the ledger what happened. rocky audit --for <table> says who changed what, under whose authority, and what was verified. rocky review --queue ranks what waits on you. rocky brief is the morning digest, and every line cites the ledger.
  • Rocky tracks what an agent builds back to its recipe. This applies on the content-addressed path. rocky gc --derivable lists artifacts whose recorded recipe matches their bytes. A review gates each eviction, and it leaves a tombstone. rocky restore then rebuilds the exact bytes, or it refuses. Restore works for a recipe that reads no recorded upstreams. It cannot yet rebuild a recipe with several inputs, so eviction is not reversible for every artifact.
  • The agent surface is MCP. rocky mcp exposes 30 tools. They ground an agent in your real schemas and data, draft changes that compile in the same call, and propose the result.

an agent's change to a contracted model is planned, rocky apply run as the agent principal is denied by the policy plane with the rule named, and rocky audit shows the recorded decision

POC: 04-governance/11-agent-policy drives this end to end, and the policy itself is regression-tested: rocky policy test runs pinned scenarios in CI and fails when an edit loosens a rule (POC: 03-ai/07-policy).

An agent earns freedom one step at a time. A retry after a known transient failure is free. You can let a provably additive schema change flow through under policy. Everything else waits for review until you grant it.

Budgets tighten when failures repeat. They recover only as those failures age out of the window you set. rocky policy freeze is the kill switch. To see how an agent writes, proposes, and passes each gate, read Operating Rocky with agents.

Where Rocky is today

These features are ready for production on Databricks: the checker, named branches, replay, column lineage, rule enforcement, and per-model cost. The rest is still in progress.

  • Databricks is the 2026 focus. Snowflake, BigQuery, and Trino run the core loop, but they are less thorough. Talk to us if you need one of them in production now.
  • AI features are early. Generate, check, and fix is shipped. rocky ai-test writes assertions for a model from its stated intent. Large refactors and automatic migration on a type change are still on the roadmap.
  • Replay re-runs your work, and says what it cannot re-run. Every run leaves a content-addressed record. rocky replay reads that record and checks it against the ledger. For a deterministic content-addressed model, rocky replay --execute --verify runs the recorded recipe again and confirms the output is identical, byte for byte. It can do this locally, or on the live warehouse in a separate replay schema. If a model reads a source that can change, Rocky marks it non-replayable instead of quietly re-running it against today's data. Rocky also flags SQL that is not deterministic, so a difference is reported as expected rather than as a failure.
  • Iceberg. Reading from a REST catalog is Beta. Today, content-addressed writes land as Iceberg-readable tables through Delta UniForm. Native Iceberg writes, with no Delta step in between, are on the roadmap.
  • No built-in metrics layer. Use Cube, or whichever metrics layer you already run.
  • Dagster is the one built-in scheduler integration (dagster-rocky). For anything else, use the rocky-sdk Python client or rocky serve. rocky tick can also run cron and freshness schedules with no orchestrator, but it is experimental.

Open a discussion if any of these are a blocker.

You can leave

rocky emit-sql writes your transformation models out as plain SQL, in dependency order. It runs offline and needs no warehouse connection.

Read the exported SQL with these limits in mind.

  • Some models produce no standalone SQL. Rocky inlines an ephemeral model as a CTE. Other models cannot render offline, such as a Snowflake dynamic table, which needs a live warehouse name. Rocky lists what it skipped on stderr, so you never mistake the export for the whole project.
  • An incremental or merge model exports only its steady-state INSERT or MERGE. That statement assumes the target table already exists. rocky run creates the table on the first build, and the exported file cannot. Rocky prefixes each of those statements with a note that says so.
  • Every model renders in one dialect. Rocky resolves one dialect for the whole project from rocky.toml. With no config it uses DuckDB. If your models target more than one adapter, the export matches rocky run only for the models whose target uses that dialect.

It is one command, not a rewrite. Adopting Rocky is not a one-way door. See No lock-in.

Already have a project in another tool? rocky import-dbt converts a dbt Core project in one command. See the import guide.

Subprojects

Path What ships Language What it does
engine/ rocky CLI and rocky-lsp Rust Core engine: SQL checking, drift detection, incremental loads, adapters
sdk/python/ rocky-sdk (PyPI) Python Python client wrapping the CLI, for notebooks and scripts
integrations/dagster/ dagster-rocky (PyPI) Python Dagster resource built on rocky-sdk
editors/vscode/ Rocky VS Code extension TypeScript Live checking, syntax highlighting, AI commands
examples/playground/ (config only) TOML / SQL Sample DuckDB pipeline, no credentials needed

Adapters

Role Adapter Status
Warehouse Databricks Production
Warehouse Snowflake Beta
Warehouse BigQuery Beta
Warehouse DuckDB Local / Testing
Warehouse Trino Beta
Source Fivetran Production
Source Airbyte Beta
Source Iceberg Beta
Source Manual Production

Building a connector for ClickHouse, Redshift, or another warehouse? See the Adapter SDK guide and the skeleton POC.

Building from source

git clone https://github.com/rocky-data/rocky.git
cd rocky
just build   # engine + sdk + dagster + vscode
just test
just lint

See CONTRIBUTING.md for per-subproject build commands.

Releases

Each artifact ships independently via CI-driven tags:

  • engine-v* → Rocky CLI binary on GitHub Releases (macOS, Linux, Windows)
  • sdk-v*rocky-sdk on PyPI
  • dagster-v*dagster-rocky on PyPI
  • vscode-v* → Rocky extension on the VS Code Marketplace

Documentation

Full docs at rocky-data.dev.

New to Rocky? ROCKY_EXPLAINED.md is a plain-English walkthrough of the whole system, with diagrams.

Contributing

See CONTRIBUTING.md. Schema or DSL changes need to update all dependent pieces at once — read the cross-project change guidance before opening a PR.

Sponsoring

Rocky is free and open source. If it saves your team time, consider sponsoring the project.

License

Apache 2.0

About

A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

294 stars

Watchers

0 watching

Forks

Releases

Sponsor this project

Packages

Used by

Contributors

Languages