fix(capture+dream): pass --verbose to claude -p stream-json#48
Merged
Conversation
Claude Code 2.1.x rejects `--print --output-format stream-json` without `--verbose`: Error: When using --print, --output-format=stream-json requires --verbose Without it, both the dream subagent and the capture-extract hook exit immediately. Visible symptom: session buffers grow into hundreds of JSONL events but `extract.log` is just a wall of the error and zero memories ever get extracted — the user reported "still only 2 memories shown in the UI" after a full day of work that should have produced many. Fix: add `--verbose` to both Popen invocations (`cli/commands/dream.py` and `cli/commands/capture.py`). Tests in `test_dreams.py` and `test_capture_hook.py` now pin the flag list so this can't silently regress when Claude Code adds another required flag in a future release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
amitpaz1
added a commit
that referenced
this pull request
May 8, 2026
After PR #48 unblocked --verbose, the next layer of capture-pipeline silence surfaced: the subagent ran successfully and the LLM correctly identified observations to save, but every mcp__lore__remember_observation call returned "Claude requested permissions to use mcp__lore__remember_observation, but you haven't granted it yet". The subagent finished PROCESSED_THROUGH_SEQ=N with zero memories persisted. Root cause: the --print subagent inherits a fresh permission state, not the parent's allowlist. Default permission mode prompts on every MCP call, and a non-interactive --print run has no way to answer. Fix: pass --permission-mode=bypassPermissions to both Popen sites (cli/commands/capture.py and cli/commands/dream.py). The capture and dream prompts are internally generated by Lore and only invoke mcp__lore__* tools (read + write own memory store), so bypassing prompts is the correct trust posture for these workers — they're not acting on user-supplied prompts and have no access to e.g. Bash. Tests in test_dreams.py and test_capture_hook.py now pin both --verbose AND --permission-mode/bypassPermissions in the args list so neither can silently regress. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
amitpaz1
added a commit
that referenced
this pull request
May 8, 2026
Foundation for the graph-population pipeline (spec:
docs/superpowers/specs/2026-05-08-lore-graph-population-design.md).
This PR adds the extraction service and the persistence-layer ops it
needs; PR B wires it into the create-time route handlers and adds the
backfill endpoint.
New service: src/lore/services/graph_extraction.py
* extract_and_persist(store, *, org_id, memory_id, content, context,
spawn_fn=None, timeout=None) -> ExtractionResult — run a
`claude -p` subagent with a deterministic extraction prompt, parse
the JSON from the final assistant message, and persist entities
(with case-insensitive + alias dedup) / mentions / relationships.
Idempotent: existing edges for the memory are deleted before insert.
* Concurrency capped via LORE_GRAPH_EXTRACTION_CONCURRENCY (default 2)
so dream-finalize bursts don't spawn 50 subprocesses at once.
* Spawn flags pinned: --output-format stream-json --verbose
--permission-mode default. The dream/capture saga (PRs #48, #49)
burned us once on the first two; the regression test at
test_spawn_claude_args.test_passes_required_flags catches future
silent-empty-graph failures.
* Failure modes (timeout / parse / non-zero exit / claude not on PATH)
all swallow and return ExtractionResult.error — no exception bubbles.
* is_enabled() auto-on iff `claude` is on PATH; explicit override via
LORE_GRAPH_EXTRACTION_ENABLED.
New store ops on Postgres + SQLite + protocol:
* find_entity_by_name_or_alias — case-insensitive name + alias lookup
(LOWER(name) match, then alias scan). PG uses jsonb_array_elements_text;
SQLite does a Python-side scan to stay portable across aiosqlite builds.
* replace_memory_mentions — DELETE existing rows for memory_id, INSERT
the supplied set, atomic via transaction.
* replace_memory_relationships — same shape on relationships
WHERE source_memory_id = ?. Active-edge UNIQUE conflicts (the
partial index on (source, target, type) WHERE valid_until IS NULL)
are silently skipped because the edge already exists from another
memory and re-asserting it from a different source isn't an error.
* list_memories_without_mentions — LEFT JOIN entity_mentions IS NULL
so the backfill endpoint (PR B) can find what to process.
Tests: tests/services/test_graph_extraction.py (39 cases — 17 run
unconditionally, 22 gated on the parametrized PG+SQLite store fixture
that CI's python-postgres job exercises). Coverage:
* Prompt builder: content + optional context block, schema lists
every entity type.
* Stream-json parser: picks last assistant text, handles json fences,
skips tool_use mid-stream events, returns None on no-JSON / empty.
* Spawn-args sanity: regression guard for the flag saga.
* Happy path: 2 entities + 1 relationship round-trip via real store.
* Dedup by case-insensitive name; dedup by alias.
* Idempotent re-extraction: replay produces same row count, not
doubled.
* Drops relationships referencing undeclared entities (LLM jitter
guard).
* Failure modes: timeout, parse error, non-zero exit, missing claude.
* Empty extraction persists nothing (clean no-op).
* Feature flag: explicit true/false + auto-on with claude on PATH.
* Concurrency cap holds under 10-task burst.
* Env-knob validators (concurrency min 1, timeout min 1s, invalid
falls back to default).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Capture pipeline was silently dead. Both the dream subagent and the capture-extract hook spawn
claude -p <prompt> --output-format stream-json, but Claude Code 2.1.x now rejects that combo without `--verbose`:```
Error: When using --print, --output-format=stream-json requires --verbose
```
Visible symptom in this user's session: `~/.lore/sessions/aab30235.../buffer.jsonl` had 185 buffered tool-use events from this session, but `extract.log` was 30+ lines of the error message and zero memories were ever extracted. Total memories in the UI stayed at 2 (from the previous day's setup).
Fix: add `--verbose` to both `Popen` invocations (`cli/commands/dream.py:418`, `cli/commands/capture.py:586`). Tests in `test_dreams.py` and `test_capture_hook.py` now pin the flag list so this can't silently regress when Claude Code adds another required flag in a future release.
Test plan
🤖 Generated with Claude Code