feat: expected-value assertions for the verified-examples corpus (#78)

`validate_examples` proves a corpus example is *allowed* — it cannot prove the
SQL is *right*. A query with every table permitted, the tenant filter present
and explicit columns can still sum the wrong rows and pass as `status: "valid"`,
and nothing downstream tells it apart from a correct one.

`VerifiedExample` gains four optional fields — `expected` (the certified scalar
answer), `rel_tol`, `abs_tol`, `time_scoped` — and a second pass,
`check_example_answers`, executes the compliant asserted rows and compares,
yielding `match` / `mismatch` / `unassertable` / `error` per row.

    report = validate_examples(examples, contract, explain_adapter=adapter)
    answers = check_example_answers(report, adapter=adapter)
    if not (report.ok and answers.ok):
        sys.exit(1)

The checker takes the *report*, not the examples. A row that violates the
tenant-filter rule is precisely the query that must not be sent to a warehouse
to see what it returns, so consuming `ExampleValidationReport` makes that
ordering a property of the signature rather than a rule in a docstring. The
adapter kinds differ for the same reason: `validate_examples` takes a
plan-only `ExplainAdapter`, and the execute-capable `DatabaseAdapter` enters
only at this second, already-filtered stage.

A relative time window is refused rather than executed: an `expected` pinned
against `WHERE created_at >= CURRENT_DATE - 30` decays on its own. The scan
needs two arms, because sqlglot types a spelling only in the dialects that own
it — `NOW()` is `exp.CurrentTimestamp` under postgres but `exp.Anonymous` under
duckdb, mysql, snowflake, bigquery, tsql and oracle. Arm two matches by name
only when the arguments look like a clock read (none, or a single integer
precision literal), which separates `NOW(3)` from the deterministic
`UNIX_TIMESTAMP(created_at)`.

Default `rel_tol=1e-9`, far tighter than `reconcile_decomposition`'s `1e-4`,
and anchored on `expected` rather than the larger magnitude: an assertion has a
certified reference, while a decomposition compares two measurements with no
privileged side.

Changed: `expected`, `rel_tol`, `abs_tol` and `time_scoped` stop being
free-form corpus keys preserved under `.metadata`, and the `sqlglot` floor
rises to `>=28.6` (the three typed time nodes the scan names do not exist
below it, so the package would not import). Both are detailed in CHANGELOG.md.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Author: flyersworder

.github/workflows/ci.yml

@@ -98,10 +98,22 @@ jobs:
         run: |
           uv run python examples/ops_agent/agent.py > out.txt
           grep -q 'Blocked columns in SELECT' out.txt
-      - name: verified-examples corpus validation
+      - name: verified-examples corpus validation + answer checks
         run: |
           uv run python examples/revenue_agent/verify_examples.py > out.txt
           grep -q 'Example validation:' out.txt
+          # The second pass is the half that executes SQL, so assert it ran AND
+          # that both failure statuses still surface — a silently-empty answer
+          # report would otherwise leave this job green.
+          grep -q 'Answer checks:' out.txt
+          # All three statuses, not just the failures: without the MATCH check a
+          # certified answer that silently went stale would flip its row from
+          # match to mismatch and this job would stay green, since 'MISMATCH' is
+          # still present. The trailing space anchors the status column, because
+          # 'MATCH' is a substring of 'MISMATCH'.
+          grep -q '\[MATCH ' out.txt
+          grep -q 'MISMATCH' out.txt
+          grep -q 'UNASSERTABLE' out.txt
 
   test-pydantic-ai-latest:
     # Early-warning job. pyproject allows pydantic-ai-slim >=2.0 (unbounded), and

CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.44.0] - 2026-08-22
+
+### Added
+
+- **Expected-value assertions for the verified-examples corpus.** `validate_examples` re-checks each example's SQL against the same `Validator` that gates live agent queries — allowed tables, tenant filter, no `SELECT *`, and (with an `ExplainAdapter`) a live schema dry-run — and reports whether the SQL *complies*. It cannot report whether the SQL is *right*. A query that satisfies every one of those rules and still sums the wrong rows passes today with `status: "valid"`; nothing downstream can tell it apart from a correct one. That is the gap this closes: `VerifiedExample` gains four optional fields (`expected`, `rel_tol`, `abs_tol`, `time_scoped`), and a new second pass, `check_example_answers(report, *, adapter, ...)`, executes the compliant, asserted rows and compares the live result against the certified answer within tolerance, yielding `match` / `mismatch` / `unassertable` / `error` per row.
+
+  ```python
+  report = validate_examples(examples, contract, explain_adapter=adapter)
+  answers = check_example_answers(report, adapter=adapter)
+  if not (report.ok and answers.ok):
+      sys.exit(1)
+  ```
+
+- **The checker consumes the validation *report*, not raw examples — the load-bearing choice in the design.** An example that failed contract validation must never be executed: a row that violates the tenant-filter rule is precisely the query that must not be run against a warehouse to see what it returns. Taking `ExampleValidationReport` as the input makes that ordering a property of the function signature rather than a rule stated in a docstring — no ordinary call shape hands it unvalidated SQL. A row is executed only when it is `status == "valid"` **and** declares an `expected`; everything else produces no result. This also keeps `validate_examples`'s own invariant intact: it still only plans (via `ExplainAdapter`) and never executes. The execute-capable `DatabaseAdapter` enters the pipeline only at this second, already-filtered stage.
+
+- **A relative time window refuses the row rather than running it.** `WHERE created_at >= CURRENT_DATE - 30` is correct today and wrong in a month, for no reason the corpus author did anything about — an expected value pinned against that SQL decays on its own. The checker scans the parsed SQL before executing anything and marks a hit `unassertable`, never running the query. The scan needs two arms: sqlglot only normalises a spelling to a typed AST node (`exp.CurrentDate`, `exp.CurrentTimestamp`, ...) in the dialects that own it. `NOW()` becomes `CurrentTimestamp` under postgres but stays an untyped `exp.Anonymous` call under duckdb, mysql, snowflake, bigquery, tsql, and oracle; `GETDATE()` is typed only under tsql and snowflake; `TODAY()` only under duckdb. A typed-node-only scan would therefore miss `NOW()` — the single most common relative spelling — under the dialects most likely to be running it. The second arm matches `exp.Anonymous` calls (`now`, `getdate`, `sysdate`, `today`, `curdate`, and others) by name, when the arguments look like a clock read: none (`NOW()`), or a single integer literal, which is a fractional-seconds precision spec (`NOW(3)`, `SYSDATE(6)`). Both halves of that are load-bearing: matching a *call* means a column named `now_flag` is never flagged, and keying on the argument's *kind* rather than its mere presence means the deterministic `UNIX_TIMESTAMP(created_at)` conversion — whose argument is a column — is not refused alongside the clock reads that share its name, while `NOW(3)` still is refused. Arity alone would get one of those two wrong, and `NOW` / `SYSDATE` are precisely the names with no typed-node arm to fall back on. Setting `time_scoped: true` on the example tells the checker the window is pinned some other way and clears the refusal.
+
+- **The comparison tolerance is anchored on `expected`, not on the larger magnitude the way `math.isclose` anchors on `max(|a|, |b|)`.** An assertion has a privileged side: the certified answer is the fixed point and the query result is what varies against it, unlike `reconcile_decomposition`'s two-measurements case which has no privileged side and anchors on the parent it measured instead. Anchoring on `expected` keeps "within 0.1% of the certified number" meaning the same thing regardless of how far the query result has drifted. The default (`rel_tol=1e-9`, `abs_tol=0.0`) is deliberately far tighter than `reconcile_decomposition`'s `1e-4`: a decomposition identity is approximate by construction (operands carried at limited precision leave a real residual), but a certified answer is meant to be *the* number — the default tolerates floating-point representation noise and nothing else. An answer certified off a dashboard that rounds to whole dollars will not match a full-precision `SUM` at this default; the author sets an explicit per-example `rel_tol` / `abs_tol` matching the answer's actual precision. `rel_diff` is guarded against a zero `expected` the same three-branch way `reconcile_decomposition` guards a zero parent, so `expected: 0.0` never raises — but the guard has a visible consequence: the relative term vanishes at zero, so a zero-valued assertion matches only an exact zero unless the author also sets an `abs_tol`.
+
+### Changed
+
+- **The `sqlglot` floor rises from `>=28.0` to `>=28.6`.** The relative-time scan names `exp.Localtime`, `exp.Localtimestamp` and `exp.Systimestamp` in a module-level tuple, and those three node types first exist in 28.6 — so on 28.0 through 28.5 this release does not import at all (`AttributeError` at `import agentic_data_contracts`, the same failure mode the `sqlglot>=23.0` floor once had for `exp.Revoke`). The tuple is deliberately not built with `getattr(exp, ..., None)` filtering: below 28.6 a bare `LOCALTIMESTAMP` parses as an `exp.Column`, and the name arm matches only function *calls* — so a filtered tuple would not fall back to the other arm, it would silently execute a row whose certified answer is pinned to a decaying window. A floor that fails loudly beats a scan that quietly checks less in one environment than another.
+
+- **Four corpus keys stop being free-form.** `VerifiedExample.from_dict` preserves any key it does not recognise under `.metadata`, untouched — that is a documented feature of the corpus format, and consumers rely on it for `type`, `verified_by`, `last_verified` and whatever else their review flow records. `expected`, `rel_tol`, `abs_tol` and `time_scoped` are now *recognised*, so a corpus already using one of those names for its own purpose changes behaviour on upgrade rather than carrying through inertly. Two shapes to check before bumping: a **non-numeric** value (`expected: "one row per region"` is a plausible note in a question-to-SQL corpus) now raises `ValueError` out of `from_dict`, which will abort the load of the whole corpus rather than that one row; and a **numeric** value that meant something else now reads as a certified answer, so `check_example_answers` will execute that row's SQL against the warehouse and report a `mismatch` against a number that was never an answer. Neither is destructive — the second pass only ever reads — but both are visible, and renaming the offending key (to `expected_shape`, say) before upgrading avoids them. A corpus using none of the four names is unaffected.
+
 ## [0.43.1] - 2026-08-18
 
 ### Fixed

README.md

@@ -14,7 +14,7 @@ You teach agents your business domains, metrics, and governance rules upfront 
 
 - **Governed, not guessed** — the agent uses *your* metric definitions (`SUM(amount) FILTER (WHERE status = 'completed')`), not an ad-hoc query it invented.
 - **Bad SQL blocked before execution** — forbidden operations, disallowed tables, missing tenant filters, `SELECT *`, unbounded scans — caught by static analysis plus an optional EXPLAIN dry-run.
-- **Validate a whole corpus, not just live queries** — re-check a verified-examples database (or a metric's arithmetic identity) against the contract in CI, and catch drift when the contract or the warehouse schema changes.
+- **Validate a whole corpus, not just live queries** — re-check a verified-examples database against the contract *and* its certified answers in CI (or a metric's arithmetic identity), catching drift when the contract or warehouse schema changes and a compliant query that quietly returns the wrong number.
 - **Business context first** — domain descriptions, metric ownership, freshness, and a metric graph (causal *and* arithmetic) guide the agent before it writes a line of SQL.
 - **Resource governance built in** — per-session cost, retry, row, and token budgets, and wall-clock limits.
 - **Per-caller row/column security** — allow/deny tables and filter values by principal, for multi-user bots.
@@ -979,7 +979,43 @@ Each example lands in exactly one `status` — `valid` (statically contract-chec
 - **MR gate** — validate the corpus in CI *before* a human reviews it; fail on `not report.ok`, so the human is no longer the only check.
 - **Drift sweep** — re-run against a *changed* contract; `report.violations` are the examples the change just broke. With an `explain_adapter`, the live EXPLAIN also catches a dropped or renamed column that static checks can't see.
 
-It confirms an example is still *allowed, well-formed, and plannable against the current schema* — never that it still returns the right answer, because it **never executes** the SQL (result correctness stays with your review). For SQL an engine parses but sqlglot cannot (e.g. Denodo/VDP), a parse failure falls back to the engine's own planner; those pass as plannable but policy-unverified, flagged in `report.unverified_compliance`. See [`examples/revenue_agent/verify_examples.py`](examples/revenue_agent/verify_examples.py) for a runnable, DuckDB-backed demo.
+It confirms an example is still *allowed, well-formed, and plannable against the current schema* — never that it still returns the right answer, because it **never executes** the SQL. Result correctness is the second pass's job, [below](#asserting-the-certified-answer-not-just-compliance). For SQL an engine parses but sqlglot cannot (e.g. Denodo/VDP), a parse failure falls back to the engine's own planner; those pass as plannable but policy-unverified, flagged in `report.unverified_compliance`. See [`examples/revenue_agent/verify_examples.py`](examples/revenue_agent/verify_examples.py) for a runnable, DuckDB-backed demo.
+
+### Asserting the certified answer, not just compliance
+
+`validate_examples` proves an example is *allowed* — it never proves the SQL is *right*. A query with every table permitted, the tenant filter present, and explicit columns can still sum the wrong rows and pass with `status: "valid"`. To close that gap, an example can carry the certified answer alongside its SQL, and a second pass, `check_example_answers`, executes just the compliant, asserted rows and compares:
+
+```yaml
+- id: acme-completed-revenue
+  question: "total completed revenue for acme"
+  sql: SELECT SUM(amount) FROM analytics.orders WHERE tenant_id = 'acme' AND status = 'completed'
+  expected: 10700.00     # the certified answer
+  rel_tol: 0.001          # optional, overrides the call-level default for this row
+  abs_tol: 0.0            # optional, likewise
+  time_scoped: false       # optional; see "relative time windows" below
+```
+
+```python
+from agentic_data_contracts.validation import check_example_answers
+
+report = validate_examples(examples, contract, explain_adapter=adapter)
+answers = check_example_answers(report, adapter=adapter)  # a DIFFERENT adapter type — see below
+
+if not (report.ok and answers.ok):
+    print(report.summary())
+    print(answers.summary())
+    # in CI: sys.exit(1)
+```
+
+Note what `answers.ok` means before you paste that gate in: it is True only when at least one assertion was actually *checked* **and** every checked one matched. An **empty** answer report is therefore False, not True — a gate that quietly stopped asserting anything fails rather than passing. The consequence to expect: adopt the recipe before any row carries an `expected` and the build goes red on a corpus with nothing wrong with it. Add the first assertion, or leave `answers.ok` out of the gate until you do.
+
+`check_example_answers` takes `report` — the output of `validate_examples` — not the raw examples. That is deliberate, not incidental: it means the pipeline cannot hand it unvalidated SQL. A row that violates the tenant-filter rule is precisely the query that must not be sent to the warehouse to see what it returns, so a row is executed only when it is `status == "valid"` **and** declares an `expected`; everything else (a violation, an unverified or unchecked row, or a valid row with no `expected`) produces no result at all. Note the two functions also take different adapter *kinds*: `validate_examples` takes an `ExplainAdapter` (plans only, never runs a query), while `check_example_answers` takes a `DatabaseAdapter` (executes) — the execute-capable adapter enters the pipeline only at this second, already-filtered stage.
+
+Each result lands in exactly one `status` — `match`, `mismatch` (both `expected` and `actual` populated, plus `abs_diff` / `rel_diff`), `unassertable`, or `error`. **A SQL statement using a relative time window is refused, not executed**: `WHERE created_at >= CURRENT_DATE - 30` degrades correctly as fixture data ages, so the certified answer would too, for a reason the corpus author never touched. The checker scans for that before running anything — `CURRENT_DATE` / `CURRENT_TIMESTAMP` and friends, plus function-call spellings like `NOW()`, `GETDATE()`, and `TODAY()` — and marks the row `unassertable` when it finds one. Set `time_scoped: true` once you've confirmed the window is pinned some other way (e.g. the SQL binds explicit dates from application code) to run it anyway.
+
+The default tolerance is deliberately tight — `rel_tol=1e-9`, `abs_tol=0.0` — because a certified answer is meant to be *the* number, not an approximation; the default absorbs only floating-point representation noise. Widen `rel_tol` / `abs_tol` per example when the certified answer itself has limited precision — e.g. it was read off a dashboard that rounds to whole dollars — rather than loosening the call-level default for the whole corpus.
+
+One more consequence worth knowing before you write `expected: 0`: the tolerance's relative term is `rel_tol * abs(expected)`, so at `expected == 0` it's always zero and only the absolute term can pass a near-miss. A row asserting "zero failed orders in Q1" matches only an *exact* zero unless you also set an `abs_tol`.
 
 Its sibling `reconcile_decomposition(...)` applies the same CI-first, contract-relative spirit to a metric's declared arithmetic identity, executing the `decompositions` above against live data to assert the identity still holds within tolerance. Its default `rel_tol=1e-4` assumes the operands are exact — see [operand units and precision](#metric-decomposition-and-drill-dimensions) when one of them is a rounded percentage or carries limited decimals.