Conversation
Add pnpm coverage:report: joins the registry, each protocol's testData skip/expectation maps, and the on-disk suites to report runnable, skipped, value-asserted, and suite status per protocol-chain, with an optional vitest JSON join for executed results. Planning logic moves to a vitest-free module shared with the suite runner so the report cannot drift from what actually registers. Record the measured baseline in the methodology spec's new progress table. Baseline it reports: 394 registry actions, 232 runnable across 19 protocol-chains, 117 skipped, 8 value-asserted, 5 protocols with no harness, 1 hard-skipped suite, 1 orphaned testData chain.
scripts/protocol-local.sh reproduces the ephemeral CI environment locally: seeded database on the shared postgres, rig-owned anvil forks (mainnet adopts an archive upstream from env or an existing fork container; Sepolia is recreated per test run for a fresh upstream window, same as CI), production app build and start, CI's chains-row patches, and a suite runner that emits vitest JSON and joins it into pnpm coverage:report. All node invocations are containerized; Turnkey keys pass through process env only, with a placeholder-wallet mode for signing-free suites. Measured: cold up 7m49s including the build, warm up 59s, one full suite (safe, 6/6 with oracle assertions through the real app and fork) validated end to end in 27s - against 30-35 minutes per CI round. Recorded in the methodology spec's progress table.
Every registered action now encode-tests against its declared ABI with deterministic synthetic args (394 actions, catches broken ABIs, renamed functions, selector and arity drift), and every testData chain gets bound-encode goldens: bindings resolved through buildActionWorkflow, encode transforms applied, then the same reshapeArgsForAbi and coerceArgsForAbi pipeline the runtime steps use, with exact calldata and target address compared to checked-in goldens. Skipped actions that are legitimately unencodable record that state deterministically. Fragment resolution accounts for single-tuple params flattened by deriveTupleInputs and for overloads. scripts/protocol-mutation-check.sh proves the layer has teeth: a renamed ABI input in a protocol definition turns the suite red. Full layer runs in 2.4s (476 tests) inside the unit gate. Regenerate goldens with UPDATE_GOLDENS=1. Recorded in the methodology spec's progress table.
Run every registry action for a protocol-chain directly against an anvil fork with no app, webhook, or Turnkey: testData setup provisions state (cheatcode gas, whale/faucet ERC20s with whale gas funding, impersonated approvals, setup protocol steps encoded from the setup workflow's own node configs), reads are eth_call with outputs decoded and asserted through the same oracle, and writes are real impersonated transactions in registry order so later writes see earlier state. Calldata encoding is shared with the Tier 0 goldens via the extracted lib/test-data/encode-action.ts, so all tiers encode identically. Chain entrypoints are env-gated (PROTOCOL_SIM_RPC_<chainId>) and run via scripts/protocol-local.sh sim. First chain-1 sweep: 275 tests, 144 passing in 155s, 86 documented skips, and 45 failures that are newly exposed latent defects (stale yearn fallback vault, codeless chainlink and curve targets, morpho fixture duplicating its setup step, and a stale rocket-pool deposit-pool address that would revert for real users), catalogued in the spec for the repair phase.
…tions Every failure from the first Tier 1 fork sweep traced to a real defect, not the harness; all repaired and re-verified against the fork (chain-1 sweep now 185 pass, 90 documented skips, 0 fail): - yearn and curve actions targeted nothing: userSpecifiedAddress contracts ignore the registry fallback map, so actions without a contractAddress binding resolved an empty target. Bind the live USDC-1 yVault (verified via eth_call) and the canonical 3pool. - chainlink's generic feed reads bind the canonical mainnet ETH/USD aggregator; the four CCIP token checks become documented skips on chain 1 (CCIP-BnM is a testnet-only surface). - morpho set-authorization revoked an authorization that fresh state never grants (reverts "already set" by construction); grant instead. supplyCollateral moves before borrow in the ABI so registry-order write sequences have collateral before borrowing against it. - rocket-pool's deposit-pool address was superseded on-chain; deposits reverted "Invalid or outdated contract" for real users. Updated to the current deployment resolved from RocketStorage. Tier 0 now fails any runnable action that resolves no target address, closing the whole defect class; goldens regenerated.
The protocol-coverage step now emits vitest JSON and fails when the executed-test count drops below a floor (30 with today's secrets), publishing run/skip counts to the step summary - a green exit code can no longer mean the suites silently self-skipped, which is how coverage collapsed to ~35 executed tests unnoticed. Floor verified in both directions locally: real results pass, an all-skipped results file trips it. PROTOCOL_E2E_REPRESENTATIVES=1 shrinks each coverage phase to its first runnable action (documented skips for the rest), implemented in the pure planning layer with unit tests - the mechanism for a future PR-gate/nightly split now that Tier 0/1 carry per-action breadth. Workflow changes validated locally per the new spec recipe: step command sequences on the rig, structure/gate/job-launch via act with a labeled-PR event fixture and the kill-switch var.
Morpho's 18 vault actions bind the live Steakhouse USDC MetaMorpho vault (verified via eth_call) instead of being skipped - the same pattern as yearn; no output piping needed. The core write sequence is margin-hardened (borrow 10, repay 8 to stay under debt plus interest, withdraw-collateral 0.02) after a borderline interest-timing failure at the old amounts. Chain-1 Tier 1 sweep: 203 pass, 72 documented skips, 0 fail, stable across consecutive runs. Pendle stays deferred: its markets expire, so hardcoded bindings rot; it needs the state-snapshot fixture approach. CI: a tier1-simulations job joins the ephemeral e2e workflow - fork plus vitest with no app build, its own 150-executed floor - bringing per-action breadth to CI for the first time in parallel with the e2e jobs. protocol-nightly.yml runs the full e2e suite nightly via workflow_call (gate extended to schedule events) plus the Tier 0 mutation check (script gains RUN_LOCAL for CI hosts with node). Workflows actionlinted and act-validated per the spec recipe.
patch_chains hardcoded 8547/8548 while start_forks, cmd_test, and cmd_sim all honor SEPOLIA_FORK_PORT/MAINNET_FORK_PORT - overriding a port moved the fork but left the chains rows pointing at a dead socket with the fallback nulled.
The executed-test floor was calibrated for full-breadth runs; enabling PROTOCOL_E2E_REPRESENTATIVES=1 on the same step (its stated purpose) would collapse executed counts to ~2 per active suite and trip the floor meant to protect it. Derive the floor from the mode in the step itself so the two cannot drift apart.
Number("garbage") is NaN and executed < NaN is always false, so a
mistyped or swapped CI argument silently disabled the floor - the exact
silent-collapse failure it guards against. Exit 2 on anything that is
not a non-negative number.
…eBinding The simulation setup hand-forked the spender resolution with a laxer failure mode: a missing contract address fell through to an empty spender and an opaque approve(0x0) revert, where resolveBinding throws the actionable error. Reuse the exported resolver; a new binding kind now reaches the simulation path automatically.
… simulation tiers
…nctionAbiEntry type
…ructure in reaper
…hold, not the pending cutoff A durably enqueued run sits 'running' with no step logs until a worker picks up its trigger step, which is indistinguishable in the DB from a pod that never scheduled. Reaping running+no-logs at the 5-min pending cutoff would false-fail healthy runs waiting on a backed-up worker queue - the exact incident conditions the reaper runs in - and inflate the infrastructure counter it exists to keep accurate. Drop the 5-min never-progressed branch; running+no-logs is now reaped at the same 30-min running threshold as a stalled run, and only the classification (infrastructure/P-0001 vs workflow_engine/E-0001) differs by step-log presence.
P-0001 ("the run could not be started", pod never ran) was categorized
workflow_engine in the ERROR_CODES registry, but every writer of the code - the
reaper and the executions route - persists errorCategory='infrastructure' for
it, and its P-000x siblings are all infrastructure. The mismatch meant the
executions route (which derives errorCategory from ERROR_CODES[code].category)
wrote a different category than the reaper for the same code. Align the registry
to infrastructure.
…nutes module-private They have no importer outside reap-stale-executions.ts; reapStaleExecutions is the module's only public surface.
…itch-projects-list fix: refresh analytics projects list on org switch
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…hook-phantom-consume fix: classify never-progressed running executions as infrastructure in the reaper
…rpc-endpoints-in-the-system-error fix: scrub RPC provider URLs from workflow error surfaces
…ge-methodology test: tiered protocol coverage with local-first tooling and CI floors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.