fix(backends): fold Claude cache tokens into prompt_tokens total

[anderskev]

Problem

Token counts in review comments showed impossibly-small input values (e.g. "22 in"). The Claude backend emitted Anthropic's raw input_tokens — the uncached remainder only — as ATIF prompt_tokens, and dropped cache_creation_input_tokens entirely. On a cache-heavy turn the displayed input collapsed to the tiny uncached tail.

Fix

Fold all three Anthropic input buckets into the ATIF-defined total at both Claude backend emission sites:

prompt_tokens = input_tokens + (cache_read_input_tokens or 0) + (cache_creation_input_tokens or 0)
cached_tokens = cache_read_input_tokens   # genuine hits only; a write is not a hit

This restores the invariant cached ⊆ prompt_tokens that the recorder and renderer clamp already assume, turning the downstream min() clamp into a correct defensive guard rather than a value-destroyer.

Changes

• daydream/backends/claude.py — new module-level _total_input_tokens(usage) helper (returns None when input_tokens is absent, preserving the "no MetricsEvent without token counts" gate); wired into both the MetricsEvent.prompt_tokens and CostEvent.input_tokens sites; rewrote the emission-site comment block.
• daydream/pr_comment_renderer.py — reframed the clamp comment as a defensive guard (no logic change).
• daydream/trajectory.py, daydream/backends/__init__.py — corrected the D-15 comments/docstrings to state prompt_tokens/input_tokens is the total input and cached_tokens is the read subset.
• Tests — new backend regression test (test_prompt_tokens_include_cache_read_and_creation), new renderer regression lock (test_cached_turn_renders_total_input), and rewrote 4 tests across test_backend_claude_metrics.py and test_pr_comment_integration.py that encoded the old uncached-remainder premise to the corrected total-input contract.

Out of scope (unchanged)

• codex.py / pi.py — Codex's input_tokens is already a total; its comment is correct.
• pricing.compute_cost — no cache-creation write premium. Claude cost comes from total_cost_usd, so no cost regression.

Verification

make check green: ruff clean, mypy daydream tests clean (251 files), 2086 passed / 5 skipped.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f66b456f-8388-408a-9380-f85640df8e0f

📥 Commits

Reviewing files that changed from the base of the PR and between f3cca15 and 180a939.

📒 Files selected for processing (2)

• daydream/backends/claude.py
• tests/test_backend_claude_metrics.py

🚧 Files skipped from review as they are similar to previous changes (2)

• daydream/backends/claude.py
• tests/test_backend_claude_metrics.py

---

Walkthrough

This change updates Claude backend token accounting so cache-read and cache-creation input tokens are folded into reported total input and prompt token values. It adds a helper used for MetricsEvent and CostEvent, revises cached_tokens docs and related comments to describe cache-read subset semantics, reformats the PR comment renderer’s cached-token clamp, and updates backend, integration, and renderer tests to expect the folded totals.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: folding Claude cache tokens into prompt_tokens total.
Description check	✅ Passed	The description matches the changeset and explains the Claude token accounting fix and related updates.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

tests/test_backend_claude_metrics.py (1)

172-241: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Missing coverage for simultaneous cache-read + cache-creation.

All new/updated scenarios exercise cache-read-only or cache-creation-only usage, but never both non-zero in the same turn — a case that's realistically possible per Anthropic's API (e.g. one breakpoint hit, another written) and directly relevant given the folding logic being tested. Worth adding a case with both buckets non-zero to lock in that the sum still folds correctly and cached_tokens still reports only the read subset.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_backend_claude_metrics.py` around lines 172 - 241, Add a test in
test_prompt_tokens_include_cache_read_and_creation that covers a single turn
where both cache_read_input_tokens and cache_creation_input_tokens are non-zero,
using the existing _collect_events helper and
MockAssistantMessageWithUsage/MockResultMessageWithUsage flow. Verify that
MetricsEvent.prompt_tokens and CostEvent.input_tokens include the sum of
input_tokens + cache_read_input_tokens + cache_creation_input_tokens, while
MetricsEvent.cached_tokens and CostEvent.cached_tokens only reflect the
cache_read_input_tokens value.

daydream/backends/claude.py (1)

85-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Docstring's "mutually exclusive buckets" claim is inaccurate.

Anthropic's own docs show cache_read_input_tokens and cache_creation_input_tokens can both be non-zero in the same response (e.g. a request that hits one cache breakpoint but writes another). The summation logic here is correct regardless (it doesn't depend on exclusivity), but the docstring's claim could mislead future maintainers into assuming these are always disjoint.

📝 Suggested docstring fix

-    Anthropic reports `input_tokens` as the *uncached remainder* only, with
-    cache hits and writes split into `cache_read_input_tokens` and
-    `cache_creation_input_tokens` (mutually exclusive buckets). ATIF's
-    `Metrics.prompt_tokens` is the total input, so fold all three. Returns
-    None when `input_tokens` is absent (preserves the no-token-count gate).
+    Anthropic reports `input_tokens` as the *uncached remainder* only, with
+    cache hits and writes split into `cache_read_input_tokens` and
+    `cache_creation_input_tokens` (both can be non-zero on the same
+    request — e.g. one breakpoint hit, another written). ATIF's
+    `Metrics.prompt_tokens` is the total input, so fold all three. Returns
+    None when `input_tokens` is absent (preserves the no-token-count gate).

Based on external documentation confirming cache read and cache-creation buckets can co-occur on a single request.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@daydream/backends/claude.py` around lines 85 - 99, The `_total_input_tokens`
docstring in `daydream/backends/claude.py` incorrectly states that
`cache_read_input_tokens` and `cache_creation_input_tokens` are mutually
exclusive. Update the docstring for `_total_input_tokens` to describe that
Anthropic may return both cache buckets in the same response, and that the
function simply sums `input_tokens`, `cache_read_input_tokens`, and
`cache_creation_input_tokens` when present.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@daydream/backends/claude.py`:
- Around line 85-99: The `_total_input_tokens` docstring in
`daydream/backends/claude.py` incorrectly states that `cache_read_input_tokens`
and `cache_creation_input_tokens` are mutually exclusive. Update the docstring
for `_total_input_tokens` to describe that Anthropic may return both cache
buckets in the same response, and that the function simply sums `input_tokens`,
`cache_read_input_tokens`, and `cache_creation_input_tokens` when present.

In `@tests/test_backend_claude_metrics.py`:
- Around line 172-241: Add a test in
test_prompt_tokens_include_cache_read_and_creation that covers a single turn
where both cache_read_input_tokens and cache_creation_input_tokens are non-zero,
using the existing _collect_events helper and
MockAssistantMessageWithUsage/MockResultMessageWithUsage flow. Verify that
MetricsEvent.prompt_tokens and CostEvent.input_tokens include the sum of
input_tokens + cache_read_input_tokens + cache_creation_input_tokens, while
MetricsEvent.cached_tokens and CostEvent.cached_tokens only reflect the
cache_read_input_tokens value.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e3c37a3e-7c87-45b8-9f4a-62cb75499e2c

📥 Commits

Reviewing files that changed from the base of the PR and between aad5779 and f3cca15.

📒 Files selected for processing (7)

• daydream/backends/__init__.py
• daydream/backends/claude.py
• daydream/pr_comment_renderer.py
• daydream/trajectory.py
• tests/test_backend_claude_metrics.py
• tests/test_pr_comment_integration.py
• tests/test_pr_comment_renderer.py