test(e2e): ratchet live assertion surface (#10935)

## Outcome

NemoClaw now inventories direct and transitive live E2E assertions
without executing live test modules. A checked-in baseline and PR growth
guard prevent assertion growth while Epic #10920 reduces the suite.

## Reason

The live E2E suite had no deterministic assertion inventory. Assertions
could move into companion modules or change syntax without making the
increased failure surface visible.

### Related issues

Closes #10934

Part of #10920

## Changes

- Add a TypeScript-parser census for Vitest matchers, Node assertions,
named assertion helpers, failure guards, object-field assertions, and
detectable generated probes.
- Resolve live companion imports statically and report direct,
transitive, per-file, and unique-suite totals without loading live
modules.
- Add a compact current-main baseline that preserves the fixed Epic
#10920 reference and records the lower post-#10722 main state.
- Run the ratchet from `npm run checks:repository` and reject baseline
increases through the existing PR growth guard.
- Document the assertion-disposition and baseline-update workflow in
`test/README.md`.

The new mechanism is required by Epic #10920. Repository checks and
E2E-remediation PRs consume it. A direct assertion count is insufficient
because it can be reduced by moving assertions into helpers or other
assertion forms. `test/repository/e2e-assertion-census.test.ts` protects
the parser, import graph, path controls, census, and ratchet.

## Verification

- `npx vitest run --project integration
test/repository/e2e-assertion-census.test.ts
test/repository/checks-runner.test.ts
test/automation/pull-requests/growth-guardrails.test.ts` — 3 files and
66 tests passed.
- `npm run typecheck:cli` — passed.
- `npm run checks:repository` — passed, including the new
1,977-direct-expect baseline across 86 live test files.
- Commit hooks — pre-commit, commit-msg, and pre-push passed. The
published commit is GitHub Verified.
- No live E2E run applies because this change analyzes source and does
not change live behavior.
- The diff contains no secrets, API keys, or credentials.

## Review notes

`npm run check` completed every pre-commit and repository-wide
governance check, then one existing CLI coverage worker did not exit
after the active workers finished. The run was stopped and is not
claimed as passing.

`npm run test:changed` passed its targeted growth-guard test, then
selected 1,682 unrelated CLI, plugin, and E2E-support files. It reported
62 failures from existing host state, unavailable macOS commands such as
`ip`, and parallel timeout paths. None involved the changed census,
ratchet, runner-registration, or growth-guard tests. Required GitHub CI
remains the authoritative broad gate for this PR.

---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
  - Added tooling to scan end-to-end tests and track assertion counts.
- Added configurable assertion budgets with reporting, JSON export,
validation, and update commands.
- Added safeguards for assertion growth, missing files, renamed tests,
and budget violations.

- **CI & Quality**
- Integrated assertion-budget checks into repository validation and
growth guardrails.
- Added baseline metadata covering current end-to-end test assertion
limits.
- Added comprehensive coverage for budget enforcement and diagnostic
reporting.

- **Documentation**
- Documented the live end-to-end assertion ratchet process and approved
budget updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

Author: ericksoa

.pre-commit-config.yaml

@@ -188,7 +188,7 @@ repos:
         name: Repository checks
         entry: npm run checks:repository
         language: system
-        files: ^(\.pre-commit-config\.yaml$|\.github/workflows/e2e\.yaml$|Dockerfile(?:\.base)?$|agents/openclaw/manifest\.yaml$|agents/hermes/(?:Dockerfile(?:\.base)?|manifest\.yaml|mcp-config-transaction\.py)$|docs/resources/starter-prompt\.md$|nemoclaw-blueprint/blueprint\.yaml$|nemoclaw/package\.json$|package\.json$|scripts/(?:brev-launchable-ci-cpu|check-installer-hash|install-openshell|update-hermes-agent)\.sh$|src/lib/actions/sandbox/openshell-child-visible-credentials\.v[0-9]+\.[0-9]+\.[0-9]+\.json$|bin/.*\.(cjs|js|mjs)$|src/.*\.(cts|mts|ts|tsx)$|scripts/.*\.(cjs|cts|js|mjs|mts|ts|tsx)$|test/.*\.(cjs|cts|js|mjs|mts|ts|tsx)$|nemoclaw/src/.*\.(cts|mts|ts|tsx)$)
+        files: ^(\.pre-commit-config\.yaml$|\.github/workflows/e2e\.yaml$|Dockerfile(?:\.base)?$|agents/openclaw/manifest\.yaml$|agents/hermes/(?:Dockerfile(?:\.base)?|manifest\.yaml|mcp-config-transaction\.py)$|ci/e2e-assertion-budget\.json$|docs/resources/starter-prompt\.md$|nemoclaw-blueprint/blueprint\.yaml$|nemoclaw/package\.json$|package\.json$|scripts/(?:brev-launchable-ci-cpu|check-installer-hash|install-openshell|update-hermes-agent)\.sh$|src/lib/actions/sandbox/openshell-child-visible-credentials\.v[0-9]+\.[0-9]+\.[0-9]+\.json$|bin/.*\.(cjs|js|mjs)$|src/.*\.(cts|mts|ts|tsx)$|scripts/.*\.(cjs|cts|js|mjs|mts|ts|tsx)$|test/.*\.(cjs|cts|js|mjs|mts|ts|tsx)$|nemoclaw/src/.*\.(cts|mts|ts|tsx)$)
         pass_filenames: false
         priority: 10
 

ci/e2e-assertion-budget.json

@@ -0,0 +1,137 @@
+{
+  "$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0\n\nInitial no-growth baseline for issue #10934 and Epic #10920. The epic reference remains fixed at fe18b2f004e2304c52db33cf7bb5895828197f9a. Current main is lower because #10722 removed the live Shields assertion suites before this ratchet landed.",
+  "schemaVersion": 1,
+  "issue": 10934,
+  "epic": 10920,
+  "reference": {
+    "mainSha": "b08eaa0844dc2e972fbfa2806590668802d449b4",
+    "currentMainCollectedTests": 92,
+    "epicMainSha": "fe18b2f004e2304c52db33cf7bb5895828197f9a",
+    "epicCollectedTests": 95,
+    "epicDirectExpectCalls": 2184,
+    "epicLiveExpectCalls": 2677
+  },
+  "limits": {
+    "testFileCount": 86,
+    "liveFileCount": 225,
+    "direct": {
+      "expectCalls": 1977,
+      "matcherAssertions": 1946,
+      "nodeAssertions": 100,
+      "namedAssertionHelpers": 636,
+      "failCalls": 8,
+      "throwGuards": 87,
+      "objectFieldAssertions": 245,
+      "assertionPoints": 3022,
+      "generatedProbeBlocks": 137,
+      "generatedProbeConditions": 357
+    },
+    "unique": {
+      "expectCalls": 2469,
+      "matcherAssertions": 2433,
+      "nodeAssertions": 119,
+      "namedAssertionHelpers": 941,
+      "failCalls": 38,
+      "throwGuards": 654,
+      "objectFieldAssertions": 348,
+      "assertionPoints": 4533,
+      "generatedProbeBlocks": 293,
+      "generatedProbeConditions": 986
+    },
+    "fileMetricOrder": [
+      "directExpectCalls",
+      "directAssertionPoints",
+      "transitiveExpectCalls",
+      "transitiveAssertionPoints",
+      "transitiveGeneratedProbeBlocks"
+    ],
+    "files": {
+      "test/e2e/live/agent-turn-latency.test.ts": [22,26,31,46,1],
+      "test/e2e/live/bedrock-runtime-compatible-anthropic.test.ts": [28,64,28,68,5],
+      "test/e2e/live/bootstrap-install-smoke.test.ts": [0,0,14,30,3],
+      "test/e2e/live/brave-search.test.ts": [19,22,27,32,1],
+      "test/e2e/live/brev-workspace-cleanup.test.ts": [0,0,0,1,0],
+      "test/e2e/live/channels-add-remove.test.ts": [40,93,40,93,0],
+      "test/e2e/live/channels-stop-start.test.ts": [0,0,36,135,9],
+      "test/e2e/live/cloud-inference.test.ts": [12,18,12,18,1],
+      "test/e2e/live/cloud-onboard.test.ts": [23,25,23,25,1],
+      "test/e2e/live/common-egress-agent.test.ts": [26,66,27,74,5],
+      "test/e2e/live/concurrent-gateway-ports.test.ts": [34,43,34,43,6],
+      "test/e2e/live/cron-preflight-inference-local.test.ts": [8,8,8,8,1],
+      "test/e2e/live/dashboard-remote-bind.test.ts": [19,17,19,28,3],
+      "test/e2e/live/device-auth-health.test.ts": [13,18,13,21,0],
+      "test/e2e/live/double-onboard.test.ts": [88,97,88,97,0],
+      "test/e2e/live/external-gateway-health.test.ts": [4,5,4,11,0],
+      "test/e2e/live/full-e2e.test.ts": [30,34,39,72,7],
+      "test/e2e/live/gateway-guard-recovery.test.ts": [48,54,48,57,3],
+      "test/e2e/live/gpu-double-onboard.test.ts": [21,24,21,24,0],
+      "test/e2e/live/gpu-e2e.test.ts": [42,47,75,93,3],
+      "test/e2e/live/hermes-discord.test.ts": [21,41,28,82,16],
+      "test/e2e/live/hermes-e2e.test.ts": [82,93,112,127,3],
+      "test/e2e/live/hermes-gpu-startup.test.ts": [19,29,63,108,13],
+      "test/e2e/live/hermes-inference-switch.test.ts": [61,67,72,92,2],
+      "test/e2e/live/hermes-root-entrypoint-smoke.test.ts": [4,33,4,33,5],
+      "test/e2e/live/hermes-sandbox-secret-boundary.test.ts": [20,27,20,27,4],
+      "test/e2e/live/hermes-slack-e2e.test.ts": [0,0,29,87,14],
+      "test/e2e/live/inference-routing-provider-smoke.test.ts": [8,13,20,33,3],
+      "test/e2e/live/inference-routing.test.ts": [91,145,103,183,16],
+      "test/e2e/live/issue-2478-crash-loop-recovery.test.ts": [9,16,9,16,3],
+      "test/e2e/live/issue-4434-tui-unreachable-inference.test.ts": [28,32,28,32,0],
+      "test/e2e/live/issue-4462-scope-upgrade-approval.test.ts": [12,22,12,22,16],
+      "test/e2e/live/issue-9880-staging-launchable.test.ts": [2,2,2,2,2],
+      "test/e2e/live/jetson-nvmap-gpu.test.ts": [37,50,37,50,4],
+      "test/e2e/live/kimi-inference-compat.test.ts": [18,20,37,43,1],
+      "test/e2e/live/launch-readiness-lease-acceptance.test.ts": [6,6,6,10,6],
+      "test/e2e/live/launchable-smoke.test.ts": [14,30,14,30,3],
+      "test/e2e/live/llama-cpp-dgx-spark-qualification.test.ts": [1,1,1,7,0],
+      "test/e2e/live/llama-cpp-generic-gpu.test.ts": [29,36,44,61,2],
+      "test/e2e/live/managed-image-activation-e2e.test.ts": [0,0,16,35,0],
+      "test/e2e/live/managed-image-multiarch-startup.test.ts": [2,2,2,8,0],
+      "test/e2e/live/managed-image-protected-runtime.test.ts": [0,0,42,79,15],
+      "test/e2e/live/mcp-bridge.test.ts": [50,170,203,511,46],
+      "test/e2e/live/messaging-compatible-endpoint.test.ts": [13,19,13,28,2],
+      "test/e2e/live/messaging-providers.test.ts": [0,11,6,51,12],
+      "test/e2e/live/model-router-provider-routed-inference.test.ts": [4,4,4,5,0],
+      "test/e2e/live/native-runtime-qualification-case.test.ts": [0,0,15,93,0],
+      "test/e2e/live/network-policy.test.ts": [97,101,97,108,4],
+      "test/e2e/live/ollama-auth-proxy.test.ts": [21,29,21,29,0],
+      "test/e2e/live/onboard-policy-preset-sequencing.test.ts": [7,7,7,7,1],
+      "test/e2e/live/onboard-repair.test.ts": [21,25,21,25,0],
+      "test/e2e/live/onboard-resume.test.ts": [59,63,59,63,0],
+      "test/e2e/live/openclaw-discord-pairing.test.ts": [13,20,28,88,10],
+      "test/e2e/live/openclaw-inference-switch.test.ts": [50,56,50,57,2],
+      "test/e2e/live/openclaw-plugin-runtime-exdev.test.ts": [9,17,9,32,3],
+      "test/e2e/live/openclaw-skill-cli.test.ts": [10,17,10,17,1],
+      "test/e2e/live/openclaw-slack-pairing.test.ts": [11,18,26,86,10],
+      "test/e2e/live/openclaw-tui-chat-correlation.test.ts": [40,46,40,52,4],
+      "test/e2e/live/openshell-credential-generation-window.test.ts": [44,94,44,113,18],
+      "test/e2e/live/openshell-gateway-auth-source-contract.test.ts": [0,0,16,28,2],
+      "test/e2e/live/openshell-gateway-upgrade.test.ts": [77,109,77,121,17],
+      "test/e2e/live/overlayfs-autofix.test.ts": [22,25,22,28,2],
+      "test/e2e/live/pi-agent-qualification.test.ts": [30,37,30,52,4],
+      "test/e2e/live/podman-cpu-lifecycle.test.ts": [40,67,53,102,1],
+      "test/e2e/live/podman-portable-uninstall.test.ts": [37,36,50,71,3],
+      "test/e2e/live/portable-cpu-delegation-proof.test.ts": [0,22,0,22,0],
+      "test/e2e/live/portable-profile-rootless-linux.test.ts": [0,75,0,81,2],
+      "test/e2e/live/rebuild-hermes.test.ts": [43,89,69,271,10],
+      "test/e2e/live/rebuild-openclaw.test.ts": [31,64,31,69,3],
+      "test/e2e/live/registry-targets.test.ts": [2,6,5,26,0],
+      "test/e2e/live/runtime-overrides.test.ts": [34,46,34,46,1],
+      "test/e2e/live/sandbox-operations.test.ts": [54,102,54,102,11],
+      "test/e2e/live/sandbox-rlimits-connect.test.ts": [1,1,1,1,0],
+      "test/e2e/live/sandbox-survival.test.ts": [10,27,10,27,1],
+      "test/e2e/live/sessions-agents-cli.test.ts": [20,29,20,31,0],
+      "test/e2e/live/shields-retirement-upgrade.test.ts": [14,29,14,41,5],
+      "test/e2e/live/skill-agent.test.ts": [9,11,9,11,0],
+      "test/e2e/live/snapshot-commands.test.ts": [26,39,26,41,3],
+      "test/e2e/live/spark-express-vllm.test.ts": [35,45,47,65,4],
+      "test/e2e/live/spark-install.test.ts": [11,13,11,15,6],
+      "test/e2e/live/state-backup-restore.test.ts": [8,17,8,17,1],
+      "test/e2e/live/telegram-injection.test.ts": [19,28,26,67,12],
+      "test/e2e/live/token-rotation.test.ts": [28,36,28,36,2],
+      "test/e2e/live/tunnel-lifecycle.test.ts": [0,0,8,17,4],
+      "test/e2e/live/whatsapp-qr-compact.test.ts": [29,35,29,35,1],
+      "test/e2e/live/windows-mxc-openclaw-process-container.test.ts": [7,11,7,73,4]
+    }
+  }
+}

package.json

@@ -49,6 +49,10 @@
     "test:coverage:cli": "npm run clean:cli && npm run build:cli && tsx scripts/check-dist-sourcemaps.mts dist && vitest run --project cli --project integration --coverage --coverage.reporter=text-summary --coverage.reporter=json-summary --coverage.reportsDirectory=coverage/cli --coverage.include=\"bin/**/*.js\" --coverage.include=\"src/**/*.ts\" --coverage.exclude=\"test/**/*.js\" --coverage.exclude=\"test/**/*.ts\" && tsx scripts/check-coverage-ratchet.mts coverage/cli/coverage-summary.json ci/coverage-threshold-cli.json \"CLI coverage\"",
     "test:coverage:plugin": "vitest run --project plugin --coverage --coverage.reporter=text-summary --coverage.reporter=json-summary --coverage.reportsDirectory=coverage/plugin --coverage.include=\"nemoclaw/src/**/*.ts\" --coverage.include=\"nemoclaw/src/**/*.cts\" --coverage.exclude=\"**/*.test.ts\" && tsx scripts/check-coverage-ratchet.mts coverage/plugin/coverage-summary.json ci/coverage-threshold-plugin.json \"Plugin coverage\"",
     "test:live-e2e": "npm run clean:cli && npm run build:cli && NEMOCLAW_RUN_LIVE_E2E=1 vitest run --project e2e-live",
+    "e2e:assertions:scan": "tsx scripts/checks/e2e-assertion-census.mts --report",
+    "e2e:assertions:json": "tsx scripts/checks/e2e-assertion-census.mts --json",
+    "e2e:assertions:check": "tsx scripts/checks/e2e-assertion-census.mts --check",
+    "e2e:assertions:update": "tsx scripts/checks/e2e-assertion-census.mts --write-budget",
     "e2e:unit-gaps": "tsx tools/e2e/unit-test-gaps.mts",
     "test:imports:check": "tsx scripts/checks/no-test-dist-imports.mts",
     "test:projects:check": "tsx scripts/checks/vitest-project-overlap.mts",