feat(analytics): comprehension-check accuracy event — the primary AI metric (#866)

[thomgabriel]

Closes #866 (epic #838, item 32b).

What

The comprehension check that gates applying an AI-proposed patch already exists (components/editor/ai-partner/diff-card.tsx, sealed by lib/ai/check-seal.ts, unlimited free retries). Its first-attempt accuracy — THE primary AI harm metric per the research — was not instrumented; no trackEvent fired anywhere in the AI partner flow. This adds the event only. No behavior change to the check.

New event comprehension_check_answered — payload { lessonId, correct, attempt, courseId? }:

• Not deduped. Every answer is signal. Later attempts fire too, carrying the running counter, so remediation depth is measurable; PostHog filters attempt = 1 for the headline first-attempt-accuracy metric.
• attempt is per CHECK INSTANCE, not per lesson lifetime. The counter is keyed by checkToken — one seal is minted per proposed patch, so the token is the identity of a check instance and a new proposal restarts at 1. Per-lesson-lifetime numbering would make attempt=1 mean "the first check the learner ever saw in this lesson" and silently drop every later patch out of the first-attempt denominator. The keying is documented in-code at the ref.
• correct is the server's sealed verdict (/api/ai/partner/verify), never a client compare — the browser never holds the answer key, so the metric is not client-forgeable.
• A failed verify round trip fires nothing. verifyCheck fails SAFE by returning correct:false, which is indistinguishable from a wrong answer at the card — so this PR adds an explicit VerifyOutcome.failed marker. The UI still fails safe (Accept stays locked); analytics skips it and does not consume the attempt, so a dropped request can't inflate the primary harm metric's error rate with network noise.

eventCtx (the existing ChallengeEventContext already threaded to the AI pane) now reaches the diff card via MessageList; without it the event simply doesn't fire — never a render difference.

Red-proof

The new tests fail at main. Checking out main's diff-card.tsx under this PR's test file: 4 failed | 9 passed — the four positive assertions (attempt 1 wrong→attempt 2 retry, correct first answer, new-seal reset, failed-verify not counted) all fail because no event exists. The fifth new case ("no event when no check is rendered") passes trivially at main, as expected for a negative.

Tests

components/editor/ai-partner/__tests__/diff-card.test.tsx — asserted through the analytics facade (trackEvent), so the payload shape is covered end to end:

• first answer fires attempt: 1 with correct: false; the retry fires attempt: 2 with correct: true
• a right first answer fires attempt: 1, correct: true
• a new seal token restarts at attempt: 1
• no event when the edit doesn't apply (degrade path renders no check)
• a failed verify fires nothing and doesn't consume the attempt — the next real answer is still attempt 1

Plus helper-level coverage in lib/analytics/__tests__/events.test.ts (never deduped; courseId omitted when unknown), and the two use-ai-partner verify-fails-SAFE tests updated for the failed marker.

Verify

• pnpm typecheck clean
• full web suite green (240 files / 2202 tests)
• eslint + prettier clean
• re-verified in an isolated worktree containing only this PR's 10 files on top of origin/main: typecheck clean, 90 tests green

Inventory row added to lib/analytics/README.md per house convention.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
superteam-academy-web	Ready	Preview	Jul 30, 2026 9:07pm

[claude]

Claude finished @thomgabriel's task in 2m 58s —— View job

---

Claude Code review — comprehension-check accuracy event (#866 → #889)

Todo

• Read CLAUDE.md conventions + docs/TASK-CODES.md
• Verify diff meets issue Instrument comprehension-check first-attempt accuracy — the primary AI metric (item 32b) #866's "Done when"
• Review diff-card.tsx, use-ai-partner.ts, partner-types.ts, message-list.tsx, ai-partner-pane.tsx
• Review lib/analytics/events.ts + README + tests for correctness (attempt keying, failed marker, dedup)
• Check for client-forgeable trust boundary issues
• Check secrets/RLS/on-chain surface
• Post findings

Verdict: no blocking findings

Done-when check (#866, epic #838 item 32b)

Cross-referenced against docs/superpowers/specs/2026-07-27-RESEARCH-RECONCILIATION.md:61-62: item 32b's gap was explicitly "no check, no trackEvent in use-ai-partner.ts, metric not flowing." The comprehension check itself (diff-card.tsx, sealed via check-seal.ts) already existed pre-PR (file is MODIFIED, not new) — this PR closes exactly the stated gap: trackComprehensionCheckAnswered now fires from handlePick in diff-card.tsx:160-167, wired end-to-end via eventCtx (message-list.tsx → ai-partner-pane.tsx:278, threaded from the real caller in challenge-interface.tsx:488). Matches "Done when."

Trust boundary / correctness

• correct is the sealed server verdict from /api/ai/partner/verify (use-ai-partner.ts:332-352), never a client compare — not forgeable from the browser. Confirmed SealedCheck/checkToken never ship the answer key (partner-types.ts:86-90).
• Attempt-keying by checkToken (diff-card.tsx:119-122,147-150) is correct: a useRef compared against the current prop on each pick, reset to {token, count:0} on mismatch — verified against the "new seal token restarts at 1" test.
• The failed marker correctly short-circuits both the attempt-counter bump and the event fire (diff-card.tsx:157-167); verifyCheck fails safe to {correct:false, failed:true} on both non-ok response and thrown fetch (use-ai-partner.ts:343-349) — a dropped request can't inflate the primary-metric error rate or consume an attempt.
• The catch block around onVerify (diff-card.tsx:174-176, for callers that throw directly rather than going through verifyCheck's own try/catch) also skips the track call and the counter bump — consistent fail-safe behavior on both paths.
• No PII in the payload (lessonId, correct, attempt, optional courseId only) — matches the README's zero-PII convention.

Scope

No RLS/supabase/schema.sql, on-chain, or secrets surface touched — this is a pure client-analytics addition (events.ts + one new call site). Nothing here needs the human-signoff gates.

Tests

Read all three test files in full. The red-proof claim checks out from the diff structure (the four positive assertions require code that doesn't exist at main). Coverage is genuinely end-to-end: asserted through the trackEvent facade mock, not just the helper, so payload shape is covered through the real component. events.test.ts separately covers the helper's non-dedup and courseId-omission behavior.

Non-blocking

• diff-card.tsx:86-88 / message-list.tsx:22-24: eventCtx is typed as the full ChallengeEventContext (carries challengeKind) even though only lessonId/courseId are read here. Reasonable reuse of an already-threaded type rather than a new narrower one — not worth a separate interface for two fields, just flagging for awareness.

Verification note

I could not execute pnpm vitest in this sandbox (no pnpm on PATH, and shell commands here require interactive approval I don't have in this run) — review is based on static reading of the diff and existing test files, cross-checked against the stated red-proof. CI (CI workflow, run 30581994436) was still in progress at review time — treat it as the hard gate per house convention.