spike: replace Stop-hook curator with Claude Code `monitors`-based long-lived curator

[ovidb]

Motivation

Claude Code v2.1.105 added a top-level monitors key to the plugin manifest: long-lived background processes per plugin whose newline-delimited stdout becomes session notifications. Auto-armed at session start ("when":"always") or first skill invoke ("when":"on-skill-invoke:<skill>"), session-lifetime only. Docs: https://code.claude.com/docs/en/plugins-reference#monitors.

Our current plugin/scripts/stop-curate.sh has to dual-gate (concurrency + frequency) because the Stop hook fires per-turn — a monitor serializes by construction. PR #63 fixed the stampede with a kernel-level concurrency gate, but the underlying architecture is still load-bearing on filesystem gymnastics. A monitor replaces the whole thing with a single tailing process per session.

Goal

Replace the Stop-hook + stop-curate.sh + per-session lockfiles with:

// plugin/monitors/monitors.json
[{
  "name": "curate",
  "command": "${CLAUDE_PLUGIN_ROOT}/scripts/curate-monitor.sh",
  "description": "Incremental rememora curator",
  "when": "always"
}]

…backed by a new streaming producer in the Rust CLI (rememora curate --stream).

Acceptance criteria

• Replaces Stop hook with a single monitors entry; no per-session lockfiles.
• ≤1 curate process per session (structural, not via pgrep gate).
• Monitor handles session rotate + Claude Code restart gracefully.
• Notification volume ≤1 line / 30s per session (per the output-style budget).
• Stop hook retained as a narrow final-flush safety net (or explicitly removed if SessionEnd covers it).
• Full plan + risks in the first comment (per CLAUDE.md).

Out of scope

• Cross-agent liveness monitor (rememora events --follow) — separate ticket.
• Skill-scoped ambient search monitor — separate ticket.

Related

• fix(plugin): debounce Stop-hook curate to prevent stampede #62 (first stampede debounce)
• fix(plugin): enforce curate concurrency with pgrep gate #63 (concurrency gate fix — current state of the world)
• Rememora memory entry: Claude Code monitors manifest key (saved 2026-04-18)

[ovidb]

Detailed plan

Background

Claude Code monitors (v2.1.105) contract, as relevant to this work:

• Declared at plugin root in monitors/monitors.json (or inline in plugin.json as a string path).
• Each entry: name (dedupe key), command (shell, inherits session env + ${CLAUDE_PLUGIN_ROOT} / ${CLAUDE_PLUGIN_DATA}), description (shown in task panel), optional when ("always" default, or "on-skill-invoke:<skill>").
• One process per name for the session lifetime. Claude Code does not restart on crash. Unsandboxed, same trust level as hooks. No stdin. Every newline-delimited stdout line → notification in Claude's session.
• Disabling the plugin mid-session does not stop running monitors. They end at session end only.

Current architecture (post-#63)

Stop hook per turn ─▶ stop-curate.sh
                       │ pgrep gate (concurrency)   ← kernel truth
                       │ mtime gate (frequency)     ← idle-since-finished
                       ▼
                     rememora curate (one-shot, ~5-8 min via `claude -p` detector)

Fragilities:

1. Per-session lockfiles in $TMPDIR — stale files accumulate across reboots (harmless but ugly).
2. Two gates in shell — readable, but the invariant ("≤1 curate per session") lives across the filesystem and kernel; it's not atomic.
3. Curate re-scans the full transcript every run — wastes tokens on content it's already analyzed.

Proposed architecture

monitor (session-start)  ─▶ curate-monitor.sh
                             │
                             ▼
                           tail -F -n 0 "$JSONL_PATH" \
                             | rememora curate --stream --project X --session $SESSION_ID
                             │
                             ▼
                           one long-lived Rust process
                             (batches turns, rate-limits signal-detector,
                              emits ≤1 notification / 30s on stdout)

Structural wins:

• Concurrency invariant is a language guarantee (one process), not a filesystem one.
• Incremental — only new transcript lines are processed, not re-scanned.
• Back-pressure is natural: if curate is slow, tail's pipe fills and blocks; no stacking.
• Rate-limiting moves from shell (pgrep/mtime) to in-process token bucket; trivially tunable.

Key work

Rust CLI — new streaming producer.

• src/main.rs: add rememora curate --stream --project X --session <id> subcommand reading stdin line-by-line.
• New internal module or extension of src/curate.rs (if present) that buffers turns, runs the detector on each delta, writes new memories via src/models/context.rs.
• Token-bucket rate limiter on stdout (e.g., emit "saved: <abstract>" ≤1 line / 30s, collapsing quiet periods).
• Signal-detector prompt reused, but reshaped to accept "delta since last invocation" rather than full transcript — avoids the ~6 min full-scan cost per turn.

Plugin manifest.

• Add plugin/monitors/monitors.json with one entry.
• Add plugin/scripts/curate-monitor.sh — tiny wrapper that resolves $JSONL_PATH (same logic as stop-curate.sh) and execs the tail -F | rememora curate --stream pipeline.
• plugin/hooks/hooks.json: remove Stop entry, keep SessionStart + SessionEnd. (SessionEnd already runs a final curate flush, per current design.)
• Bump plugin version 1.0.2 → 1.1.0 (minor — replaces a runtime subsystem; users should see it in release notes).

Open questions / risks

1. Skill-invoke gating vs always-on. Always-on is simplest but spends tokens even in sessions that never produce curatable material. Evaluate whether "when":"on-skill-invoke:rememora-save" is better — only arms the monitor after the first save, i.e., the session has declared itself "interesting."
2. Interactive-CLI-only. monitors only runs in interactive Claude Code sessions. Headless usages (claude -p, API, CI) get nothing. Stop hook is portable; monitor is not. Decide whether to keep Stop hook as a fallback for headless environments, gated by CLAUDE_CODE_HEADLESS or similar env check.
3. No auto-restart. If the monitor crashes (bad JSONL line, signal-detector network hiccup), it stays dead until next session. Need defensive handling in the wrapper: while true; do tail -F ... | rememora curate --stream; sleep 5; done — but that also means we should surface crashes, not silently loop. Probably emit a single notification line on crash before re-entering the loop.
4. Notification noise. "Every stdout line is a notification" means chatty curate output will annoy users more than Stop-hook output (which is invisible). Must rate-limit aggressively and keep lines terse ("saved 2 memories (decision, case)" not full abstracts).
5. Session JSONL path is Claude Code internal. $HOME/.claude/projects/<encoded-cwd>/<session-id>.jsonl — undocumented path we already rely on in stop-curate.sh. Same risk in monitor. Worth a defensive check + a one-line notification if the path disappears.
6. Downgrade path. Users on Claude Code < v2.1.105 silently get no curation if we drop the Stop hook. Either keep both and let the monitor preempt (risks double-curate in the overlap) or gate by a version check. Prefer: keep Stop hook; let it short-circuit if it detects the monitor running (via pgrep for the monitor's name).

Migration plan

1. Land rememora curate --stream behind a feature flag, no plugin changes yet. Unit tests for delta-scanning.
2. Land monitors/monitors.json + curate-monitor.sh as an opt-in via env var (REMEMORA_USE_MONITOR=1). Stop hook defers when monitor is active (pgrep check).
3. Dogfood for ~1 week; measure: curate count per session, tokens spent, notification count.
4. Flip the default; keep the Stop hook + env flag as a fallback for headless.
5. Remove the Stop hook path in the release after.

Success criteria (repeated from issue body, expanded)

• ≤1 curate process per session verified by the same e2e test used in fix(plugin): enforce curate concurrency with pgrep gate #63 (5 rapid turns → pgrep count = 1).
• No per-session lockfiles in $TMPDIR.
• Notification budget: ≤1 line / 30s per session, averaged over an 8-hour dogfood session.
• Delta curation: signal-detector invocations scale with new transcript lines, not session length. Measurable by logging detector-input-char-count per invocation.
• Headless fallback works: Stop hook still functions in a claude -p invocation with monitors disabled.
• Crash recovery: kill the monitor manually; observe the wrapper restart loop and a single notification line surfaced to Claude.

Effort estimate

• Streaming producer in Rust: ~1–2 days (biggest unknown is the delta-scan reshaping of the detector prompt).
• Plugin wiring + shell wrapper: ~half day.
• Dogfood + rollout: ~1 week elapsed.

Ready to pick up once we're ready to allocate the bandwidth. Not urgent — #63 made the current architecture correct; this is architectural simplification, not a bug fix.

[ovidb]

PR #66 / release v0.3.3 adds a short-term recursion guard that changes the motivation/context for this monitors rewrite.

New finding: the Stop-hook stampede was not just per-session debounce/concurrency. rememora curate launches claude -p for signal detection and AUDN curation. Those child Claude sessions write their own JSONLs and fire Stop hooks with fresh session IDs, so the previous per-session gates do not compose across the recursion chain.

Short-term fix shipped in #66:

• Rust launcher sets REMEMORA_CURATE_CHILD=1 on curate-spawned claude -p children.
• plugin/scripts/stop-curate.sh exits immediately when that sentinel env var is present.
• CLI bumped to 0.3.3; plugin bumped to 1.1.0.

This issue still applies: a monitors-based long-lived curator would structurally serialize curation per real session and avoid the Stop-hook + child-session recursion class entirely.

[ovidb]

Triage (2026-04-19): Moved to Ready-For-Dev.

• Area: plugin / agent
• Priority: P2 Medium
• Size: L (cross-cutting: Rust CLI curate --stream flag + new plugin monitor script + manifest change + remove lockfile logic)
• Implementation status: NOT_STARTED — no monitors/ directory in plugin/, no --stream flag in src/commands/curate.rs, no manifest monitors key in plugin hooks
• Dependencies: none blocked
• Blocker note: None. Current stop-hook architecture works; this is an architectural simplification. Well-scoped acceptance criteria in issue body. Safe to pick up.