Merge pull request #75 from qwerfunch/feat/v0.15.7-token-measurement

feat: v0.15.7 — LLM token tracking infrastructure (F-172)

Author: qwerfunch

.claude-plugin/marketplace.json

@@ -11,7 +11,7 @@
       "name": "harness-boot",
       "description": "Multi-agent development harness for Claude Code. Your AI has speed; we give it direction — through living specs and focused specialist agents.",
       "source": "./",
-      "version": "0.15.6",
+      "version": "0.15.7",
       "homepage": "https://github.com/qwerfunch/harness-boot",
       "category": "development"
     }

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "harness-boot",
-  "version": "0.15.6",
+  "version": "0.15.7",
   "description": "Multi-agent development harness for Claude Code. Your AI has speed; we give it direction — through living specs and focused specialist agents.",
   "author": {
     "name": "qwerfunch"

.harness/spec.archive.yaml

@@ -6037,3 +6037,47 @@ features:
       - "AC-6: Project mode default unchanged (product). A new test asserts
         skeleton-only output has no `project.mode` set (falls through to
         DEFAULT_MODE='product')."
+  - id: F-172
+    description: |-
+      Direct response to the v0.15.6 post-release verification gap.
+      The user's real question — "harness vs 그냥 Claude — 작업
+      퀄리티 결과물, 토큰 소비량" — could not be answered because
+      `src/init/tokenLog.ts::recordLlmCall` only fires in init
+      scenarios. General `harness work` cycles never record LLM
+      usage, so we have zero data on per-cycle token consumption
+      across the 162 done features.
+
+      v0.15.7 closes that gap with three additions:
+      1. `harness token --in <num> --out <num> --model <id>` CLI —
+         deterministic write into events.log via the existing
+         `recordLlmCall` helper. Caller-provided values (manual
+         self-report or hook-driven auto-capture).
+      2. `harness metrics` JSON output adds `tokens_input_total`,
+         `tokens_output_total`, `llm_call_count`, and a per-model
+         breakdown — aggregated from `llm_call` events.
+      3. Dashboard surfaces cumulative tokens for the active feature
+         when any llm_call event exists.
+
+      Phase 3 work (A/B case study comparing harness vs vanilla
+      Claude on the same task) becomes feasible once this lands —
+      the measurement plumbing is the gating dependency.
+    acceptance_criteria:
+      - "AC-1: `harness token --in <num> --out <num> --model <id> [--feature
+        F-N] [--kind <subagent|user>]` writes a single `llm_call` event to
+        events.log via `recordLlmCall`. Required flags: --in, --out, --model."
+      - "AC-2: `harness metrics --json` adds `tokens_input_total`,
+        `tokens_output_total`, `llm_call_count`, and `tokens_by_model:
+        {<model_id>: {in, out, calls}}` aggregated from `llm_call` events in the
+        window."
+      - "AC-3: Dashboard active block renders an extra line `tokens: <total_in>
+        in / <total_out> out (<calls> calls)` when the active feature has ≥ 1
+        llm_call event. Hidden otherwise."
+      - "AC-4: Invalid input (non-integer tokens, empty model, etc.) exits with
+        status 3 and a clear error message."
+      - "AC-5: Tests cover the CLI happy path, metrics aggregation (single +
+        multi-model), dashboard render condition (on/off), and the
+        input-validation rejections."
+      - "AC-6: Documentation — `commands/work.md` (sidecar mention) or a new
+        `commands/token.md` short note explaining manual self-report flow.
+        Slash-command UX stays out of scope (TS CLI surface only; hook
+        automation is a follow-up)."

.harness/spec.yaml

@@ -2004,6 +2004,19 @@ features:
     test_strategy: tdd
     digest: 'Direct response to the user motive surfaced after the v0.15.5
       multi-aspect verification: "모든 것은 사용자가 요청하는 것이 아니라, 내부적...'
+  - id: F-172
+    name: work cycle LLM token tracking — measurement infrastructure
+    type: feature
+    status: planned
+    release_target: v0.15.7
+    modules:
+      - cli_token
+      - token_log
+      - metrics
+      - dashboard
+    test_strategy: tdd
+    digest: Direct response to the v0.15.6 post-release verification gap. The user's
+      real question — "harness vs 그냥 Claude — 작업 퀄...
 goals: []
 constraints:
   tech_stack:

.harness/state.yaml

@@ -3941,8 +3941,50 @@ features:
     author: human
   started_at: 2026-05-13T02:20:23Z
   completed_at: 2026-05-13T02:29:52Z
+- id: F-172
+  status: done
+  gates:
+    gate_0:
+      last_result: pass
+      ts: 2026-05-13T03:07:07Z
+      note: "cmd: npm test"
+    gate_1:
+      last_result: pass
+      ts: 2026-05-13T03:07:08Z
+      note: "cmd: npm run typecheck"
+    gate_2:
+      last_result: pass
+      ts: 2026-05-13T03:07:09Z
+      note: "cmd: npm run lint"
+    gate_5:
+      last_result: pass
+      ts: 2026-05-13T03:07:10Z
+      note: "cmd: bash self_check.sh"
+  evidence:
+  - ts: 2026-05-13T03:07:07Z
+    kind: gate_run
+    summary: "Gate gate_0 pass (1.7s) · cmd: npm test"
+    author: llm
+  - ts: 2026-05-13T03:07:08Z
+    kind: gate_run
+    summary: "Gate gate_1 pass (0.9s) · cmd: npm run typecheck"
+    author: llm
+  - ts: 2026-05-13T03:07:09Z
+    kind: gate_run
+    summary: "Gate gate_2 pass (1.6s) · cmd: npm run lint"
+    author: llm
+  - ts: 2026-05-13T03:07:10Z
+    kind: gate_run
+    summary: "Gate gate_5 pass (0.5s) · cmd: bash self_check.sh"
+    author: llm
+  - ts: 2026-05-13T03:07:10Z
+    kind: test
+    summary: F-172 token tracking infrastructure — harness token CLI + LlmCallEvent.feature 확장 + scenario=work · metrics.tokens 누적 + per-model breakdown · dashboard token_line · 862/862 tests + 9 new (5 tokenLog · 2 metrics · 2 dashboard) · 실증 4 시나리오 (CLI happy + metrics 집계 + dashboard 표시 + invalid input 차단)
+    author: human
+  started_at: 2026-05-13T03:07:38Z
+  completed_at: 2026-05-13T03:07:39Z
 session:
   started_at: 2026-04-23T13:02:49Z
-  last_command: /harness:work F-171
+  last_command: /harness:work F-172
   last_gate_passed: gate_5
   active_feature_id: null

CHANGELOG.md

@@ -11,6 +11,52 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versio
 
 ### Queued
 
+## [0.15.7] — 2026-05-13
+
+**LLM token tracking infrastructure.** The v0.15.6 verification cycle
+surfaced a measurement gap: the user's real question — "harness vs
+그냥 Claude — 작업 퀄리티 결과물, 토큰 소비량" — could not be
+answered because `recordLlmCall` only fired in init scenarios.
+General `harness work` cycles never recorded LLM usage, so the
+plugin had zero data on per-cycle token consumption across 162+
+done features. v0.15.7 closes the plumbing so the next-cycle A/B
+case study (vanilla Claude vs harness on the same task) becomes
+actually measurable.
+
+### Added
+
+- **F-172** — `harness token --in <num> --out <num> --model <id>
+  [--feature F-N] [--kind <subagent|user>]` CLI writes a single
+  `llm_call` event via the existing `recordLlmCall` helper.
+  `--in` / `--out` validated as non-negative integers (exit 3 on
+  invalid). `--model` required.
+- **F-172** — `LlmCallEvent` extended: `scenario` union now includes
+  `'work'`, plus an optional `feature` field. Backward-compat:
+  existing init callsites work unchanged.
+- **F-172** — `MetricsReport.tokens` adds `input_total` +
+  `output_total` + `call_count` + per-model `by_model` breakdown.
+  Aggregated from `llm_call` events in the metrics window.
+- **F-172** — Dashboard renders `tokens: X in / Y out (N calls)`
+  line on the active feature when llm_call events exist; hidden on
+  fresh projects.
+
+### Tests
+
+- 854 → 862 (9 new: 5 tokenLog parity · 2 metrics aggregation · 2
+  dashboard render condition).
+- Empirical: 4 scenarios reproduced (CLI happy / metrics aggregation
+  / dashboard visibility / invalid input rejection).
+
+### Follow-up (not in this release)
+
+- Hook automation — auto-capture token usage from Claude Code
+  session boundaries (`SessionEnd` hook or message-level capture).
+  Currently requires manual self-report via `harness token`.
+- Phase 3 A/B case study — same task built two ways
+  (vanilla Claude vs harness) with this measurement plumbing in
+  place. Outcome metrics (code quality + token cost) finally
+  comparable.
+
 ## [0.15.6] — 2026-05-13
 
 **Starter automation.** Direct response to the user motive surfaced

README.ko.md

@@ -6,8 +6,8 @@
 
 Claude Code 위에서 도는 multi-agent 개발 하네스. 다른 AI 도구가 *능력* 을 더할 때, harness-boot 는 *방향* 을 만듭니다.
 
-[![plugin](https://img.shields.io/badge/plugin-v0.15.6-blue)](.claude-plugin/plugin.json)
-[![tests](https://img.shields.io/badge/tests-854%20passing-brightgreen)](tests/parity)
+[![plugin](https://img.shields.io/badge/plugin-v0.15.7-blue)](.claude-plugin/plugin.json)
+[![tests](https://img.shields.io/badge/tests-862%20passing-brightgreen)](tests/parity)
 [![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)
 
 ---

README.md

@@ -6,8 +6,8 @@
 
 harness-boot is a multi-agent development harness for Claude Code. Where most AI tools add *capability*, we add *focus*.
 
-[![plugin](https://img.shields.io/badge/plugin-v0.15.6-blue)](.claude-plugin/plugin.json)
-[![tests](https://img.shields.io/badge/tests-854%20passing-brightgreen)](tests/parity)
+[![plugin](https://img.shields.io/badge/plugin-v0.15.7-blue)](.claude-plugin/plugin.json)
+[![tests](https://img.shields.io/badge/tests-862%20passing-brightgreen)](tests/parity)
 [![license](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)
 
 ---