Skip to content

Commit c913aa3

Browse files
authored
feat(scoring): redesign ordered planning eligibility, detection and target (#39)
* feat(scoring): redesign ordered planning eligibility and planned detection Ordered Planning previously counted trivial multi-file edits (docs, configs, lockfiles) as eligible, excluded large single-file refactors, missed brainstorming/plan-file planning, and ignored cross-session planning. "Planned" was a cheap, gameable binary. Eligibility now requires a code write plus non-trivial size (>=2 code files, code churn, or >=10 substantive calls) and excludes doc/config/lockfile/test-only sessions. Planned detection recognizes plan-file writes and planning skills, gated by a substance floor so bare plan-mode toggles and 2-step throwaway todos no longer count. Adds cross-session consume-once credit matched by cwd within a time window, and a significance floor that drops the term below a minimum eligible-session count. Threshold constants (CHURN_MIN, WINDOW, PLAN_MIN_LINES, PLAN_MIN_STEPS, MIN_ELIGIBLE_SESSIONS, PLANNING_TARGET) land as documented calibration placeholders pending a real-corpus run. AQ 20% / G-Stack 15% weights unchanged; golden vectors unchanged. 983 tests pass. * fix(scoring): detect plans in any /plans/ dir and count planning-skill alone Real-data analysis of a heavy-planner engineer scoring ~1% ordered planning surfaced two detector blindspots in the metric. Plan-file detection required the filename to contain "plan" for /plans/ dirs, so the superpowers writing-plans convention (docs/superpowers/plans/2-foo.md) was never recognized. Now any .md/.mdx/.txt directly inside a plans/ directory at any depth counts, regardless of filename; the .claude/plans, .cursor/plans and .context/*plan* patterns are unchanged and there are no new false positives (deployment-plans/, non-md files, plansomething/ still reject). A planning-skill invocation before the first code write now counts as planned on its own — the skill is itself the planning act — not only when paired with a plan-file. Cross-session credit is unchanged (skill-only does not become a shared plan artifact). 986 tests pass; golden vectors unchanged. * feat(scoring): raise ordered-planning target to 50% coverage Calibrated against real post-fix data: after the plan-detection fixes, the strongest engineers cluster at ~48-67% ordered-planning coverage, so the prior 40% target no longer discriminates (most saturate it). Raising PLANNING_TARGET to 0.50 keeps the metric meaningful at the top while leaving headroom. Propagated to the scoring-philosophy doc and its contract test, the AQ/G-Stack saturation tests (now 5/10 = full credit), and the regenerated golden vectors (target display only; no score/pct drift). 986 tests pass.
1 parent 32aec8e commit c913aa3

11 files changed

Lines changed: 842 additions & 68 deletions

docs/scoring-philosophy.md

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,30 +18,37 @@ The scoring review considered these external recommendations:
1818

1919
Those sources support the direction—conditional planning, relevant pre-write evidence,
2020
evaluated routing, and descriptive volume—but do not establish universal healthy thresholds.
21-
The 40% Planning and 60% Context Intelligence targets are explicit, versioned product
21+
The 50% Planning and 60% Context Intelligence targets are explicit, versioned product
2222
hypotheses: trivial work is excluded and eligible work still has room for direct execution
2323
when a formal plan or retrieval adds no value.
2424

2525
## Decisions
2626

2727
| Area | Decision | Why |
2828
|---|---|---|
29-
| Planning readiness | Grade ordered planning readiness only on eligible non-trivial changes and target 40% coverage | Small tasks should stay direct; larger work benefits from an explicit plan before editing. Planning Skill practice remains a separate educational term. |
29+
| Planning readiness | Grade ordered planning readiness only on eligible non-trivial changes and target 50% coverage | Small tasks should stay direct; larger work benefits from an explicit plan before editing. Planning Skill practice remains a separate educational term. |
3030
| Context Intelligence | Target evidence gathering before the first write in 60% of eligible changes | Grounding should inform implementation, not become after-the-fact ceremony. |
3131
| Model routing | Reward completed, substantive work routed to a lower-tier model when linkage is observable | This teaches efficient model selection without guessing from incomplete telemetry. |
3232
| Existing signals | Keep skills, MCPs, CLIs, ToolSearch, fanout, output, delegation, and model diversity scored | They are educational prompts for capabilities users should learn, not claims of output quality. |
3333
| Recency | Keep the 65% recent / 35% full-window blend | Recent improvement stays visible while established habits retain influence. |
3434

35-
Planning and evidence eligibility requires a write plus either two distinct written files or
36-
ten substantive tool calls. Unsupported or incomplete telemetry is `N/A`, not zero. Score
37-
contracts are versioned; changes between incompatible contracts are not labeled improvement
38-
or regression.
35+
Change-session eligibility requires at least one code write, together with either two distinct
36+
code files, code churn past a net-changed-lines floor, or ten substantive tool calls; doc,
37+
config, lockfile, and test-only sessions are excluded (a mixed code+test session stays eligible
38+
via its code files). Unsupported or incomplete telemetry is `N/A`, not zero. Score contracts are
39+
versioned; changes between incompatible contracts are not labeled improvement or regression.
3940

4041
Eligibility conditions the ordered readiness signal, not every planning-related metric. Actual
41-
planning Skill use remains separately scored to teach the reusable practice. Only Plan Mode or
42-
at least two distinct plan/task steps before the first write prove ordered readiness; a
43-
planning-labelled Agent event does not. These readiness signals do not count as planning Skill
44-
practice.
42+
planning Skill use remains separately scored to teach the reusable practice. A substantive
43+
plan-file write, a planning-skill invocation paired with a plan-file, or at least three distinct
44+
plan/task steps before the first code write prove ordered readiness (raised from two steps). A
45+
bare Plan Mode toggle or a two-step throwaway todo, with no plan-file and no skill, does not
46+
count: planning theater isn't planning. A plan produced in one session can also credit a later
47+
session's eligible execution in the same working directory within a bounded time window
48+
(consume-once — one plan credits exactly one execution), so planning in one session and
49+
executing in another still counts. Below a minimum number of eligible sessions the signal is
50+
dropped (and the remaining terms renormalized) rather than scored on too little data. These
51+
readiness signals do not count as planning Skill practice.
4552

4653
## Volume is descriptive, not AQ
4754

gnomon/cli/accumulator.py

Lines changed: 222 additions & 25 deletions
Large diffs are not rendered by default.

gnomon/scoring/aq.py

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,23 @@
22
from gnomon.config import available_caps
33
from gnomon.scoring.versioning import SCORE_CONTRACT_ID
44

5-
PLANNING_TARGET = 0.40
5+
PLANNING_TARGET = 0.50
66
CONTEXT_INTELLIGENCE_TARGET = 0.60
77

8+
# ---- Ordered-planning redesign (C1-C7) calibration placeholders ------------
9+
# All five constants below are PROVISIONAL calibration placeholders (proposal C5):
10+
# picked from qualitative guidance (Anthropic plan-mode guidance, Fowler's Design
11+
# Stamina Hypothesis), NOT yet fit against a real corpus. Recalibrate all of them
12+
# together once eligible/planned counts are available from production data —
13+
# do not tune one in isolation, they interact (a lower CHURN_MIN admits more
14+
# sessions as eligible, which shifts the denominator PLANNING_TARGET is judged
15+
# against).
16+
CHURN_MIN = 80 # net changed lines (C2): single-file eligibility via churn
17+
WINDOW = 72 * 3600 # seconds (C4): cross-session plan-credit lookback window
18+
PLAN_MIN_LINES = 8 # net lines (C6): minimum substantive plan-file size
19+
PLAN_MIN_STEPS = 3 # distinct todo/task steps (C6): raised from 2 (anti-theater)
20+
MIN_ELIGIBLE_SESSIONS = 5 # sessions (C7): below this, drop+renormalize (noise floor)
21+
822
_MODEL_TIERS = {
923
"anthropic": (("opus", 3), ("sonnet", 2), ("haiku", 1)),
1024
"openai": (("pro", 4), ("mini", 2), ("nano", 1), ("gpt-", 3), ("codex", 3)),
@@ -128,7 +142,11 @@ def has_skill(needles):
128142
task_calls = t.get("task_tool_calls", 0) + _task_skill_uses(skills)
129143
ordered_state = b.get("ordered_facts_state")
130144
eligible = b.get("eligible_change_sessions", 0) or 0
131-
ordered_planning = (None if ordered_state != "measured" or not eligible
145+
# C7 — significance floor: below MIN_ELIGIBLE_SESSIONS the ratio is noise
146+
# (e.g. 40% over 2 sessions), so drop the term (None -> renormalized)
147+
# rather than score it. Placeholder constant, see aq.py's MIN_ELIGIBLE_SESSIONS.
148+
ordered_planning = (None if ordered_state != "measured"
149+
or eligible < MIN_ELIGIBLE_SESSIONS
132150
else sat(b.get("planned_eligible_sessions", 0) / eligible,
133151
PLANNING_TARGET))
134152
planning_skill = 1.0 if has_skill(["writing-plans", "autoplan", "plan"]) else 0.6

gnomon/scoring/gstack.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import json
22

3-
from gnomon.scoring.aq import CONTEXT_INTELLIGENCE_TARGET, PLANNING_TARGET
3+
from gnomon.scoring.aq import (
4+
CONTEXT_INTELLIGENCE_TARGET, PLANNING_TARGET, MIN_ELIGIBLE_SESSIONS,
5+
)
46

57

68
REPO_URL = "https://github.com/Photobombastic/paxel-local"
@@ -270,7 +272,10 @@ def compute_scores(stats):
270272
# ~40% of sessions. See accumulator.plan_sessions.
271273
plan_ceremony = _clamp((b.get("planning_skill_sessions", b.get("plan_sessions", 0)) / sess) / 0.4)
272274
eligible = b.get("eligible_change_sessions", 0) or 0
273-
ordered_plan = (None if b.get("ordered_facts_state") != "measured" or not eligible
275+
# C7 — significance floor: drop below MIN_ELIGIBLE_SESSIONS (noise, not a
276+
# real signal), mirroring the aq.py guard.
277+
ordered_plan = (None if b.get("ordered_facts_state") != "measured"
278+
or eligible < MIN_ELIGIBLE_SESSIONS
274279
else _clamp((b.get("planned_eligible_sessions", 0) / eligible)
275280
/ PLANNING_TARGET))
276281
# reasoning depth needs a source that emits thinking blocks (Antigravity CLI doesn't);
@@ -399,7 +404,9 @@ def _zero_axis(gloss, subs_spec):
399404
plan_ceremony_pct = _clamp(plan_sess_raw / 0.4)
400405
eligible = b.get("eligible_change_sessions", 0) or 0
401406
ordered_raw = b.get("planned_eligible_sessions", 0) / eligible if eligible else 0
402-
ordered_pct = (None if b.get("ordered_facts_state") != "measured" or not eligible
407+
# C7 — significance floor (see compute_scores above for rationale).
408+
ordered_pct = (None if b.get("ordered_facts_state") != "measured"
409+
or eligible < MIN_ELIGIBLE_SESSIONS
403410
else _clamp(ordered_raw / PLANNING_TARGET))
404411
planning_val = _axis_value([(0.30, explore_pct, None), (0.30, thinking_pct, "thinking"),
405412
(0.25, plan_ceremony_pct, "skills"),

gnomon/scoring/inputs.py

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -164,10 +164,13 @@ def build_monthly_scoring_stats(
164164
explore = cats.get("explore", 0) + month_thinking_blocks.get(mk, 0)
165165
doing = cats.get("produce", 0) + cats.get("execute", 0) + cats.get("delegate", 0)
166166
planning_ratio = (explore / doing) if doing else 0
167-
from gnomon.cli.accumulator import derive_ordered_behavior
168-
ordered = [derive_ordered_behavior(events)
169-
for events in (month_session_ordered_tools or {}).get(mk, {}).values()]
170-
eligible = sum(item["eligible"] for item in ordered)
167+
# C4: cross-session consume-once credit, scoped to this month's sessions
168+
# (a plan artifact only credits an execution in the SAME calendar month
169+
# bucket — matching the existing monthly-progression scoping).
170+
from gnomon.cli.accumulator import aggregate_ordered
171+
_month_agg = aggregate_ordered(
172+
(month_session_ordered_tools or {}).get(mk, {}).values())
173+
eligible = _month_agg["eligible"]
171174

172175
stats_full = {
173176
"corpus": {"sources": {s: {} for s in sources_present}},
@@ -197,8 +200,8 @@ def build_monthly_scoring_stats(
197200
(month_planning_skill_sessions or {}).get(mk, set())
198201
& month_sessions.get(mk, set())),
199202
"eligible_change_sessions": eligible,
200-
"planned_eligible_sessions": sum(item["planned"] for item in ordered),
201-
"evidence_eligible_sessions": sum(item["evidence"] for item in ordered),
203+
"planned_eligible_sessions": _month_agg["planned"],
204+
"evidence_eligible_sessions": _month_agg["evidence"],
202205
"ordered_facts_state": "measured" if m_tool_total else "unmeasured",
203206
"delegate_actions": delegate_m,
204207
"background_tasks": background_m,

gnomon/taxonomy.py

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,83 @@
120120
# Knowledge MCPs arm unconditionally (handled separately in accumulator.py).
121121
CI_CONTEXT_SUBCATS = frozenset({"project", "data", "design"})
122122

123+
# ---- Ordered-planning file taxonomy (C1/C2/C3) -----------------------------
124+
# classify_change_target/is_plan_file_target back the ordered-planning eligibility
125+
# and plan-detection redesign: a write's file TYPE decides whether it counts toward
126+
# change-session eligibility (C2) and whether it counts as a plan artifact (C3).
127+
_LOCKFILE_NAMES = frozenset({
128+
"package-lock.json", "yarn.lock", "pnpm-lock.yaml", "npm-shrinkwrap.json",
129+
"cargo.lock", "poetry.lock", "pipfile.lock", "gemfile.lock", "go.sum",
130+
"composer.lock", "mix.lock", "flake.lock", "packages.lock.json",
131+
})
132+
_CODE_EXTS = frozenset({
133+
".py", ".js", ".jsx", ".ts", ".tsx", ".mjs", ".cjs", ".go", ".rb", ".java",
134+
".kt", ".kts", ".swift", ".c", ".h", ".cpp", ".cc", ".hpp", ".rs", ".php",
135+
".cs", ".scala", ".m", ".mm", ".vue", ".svelte", ".dart", ".ex", ".exs",
136+
".sh", ".bash", ".zsh", ".sql", ".lua", ".r", ".pl", ".clj", ".elm",
137+
})
138+
_TEST_NAME_RX = re.compile(
139+
r'(^|[/_.-])tests?([/_.-]|$)|\.test\.[a-z]+$|\.spec\.[a-z]+$|_test\.[a-z]+$|'
140+
r'_spec\.[a-z]+$|^test_|(^|/)__tests__(/|$)|(^|/)spec(/|$)',
141+
re.I,
142+
)
143+
_DOC_EXTS = frozenset({".md", ".mdx", ".rst", ".txt", ".adoc"})
144+
_DOC_NAMES = frozenset({"readme", "changelog", "license", "contributing", "authors"})
145+
_CONFIG_EXTS = frozenset({
146+
".json", ".yaml", ".yml", ".toml", ".ini", ".cfg", ".conf", ".env",
147+
".properties", ".xml",
148+
})
149+
_CONFIG_NAMES = frozenset({
150+
"dockerfile", "makefile", "vagrantfile", "procfile", "gemfile",
151+
".gitignore", ".dockerignore", ".editorconfig", ".eslintrc",
152+
".eslintrc.json", ".eslintrc.js", ".prettierrc",
153+
})
154+
155+
156+
def classify_change_target(path):
157+
"""Classify a write target into code/test/doc/config/lockfile/other for
158+
change-session eligibility (C2) and file-type semantics (P1/P2 fixes).
159+
Order matters: lockfile and test checks run before the generic extension/
160+
name maps, since e.g. package-lock.json is a .json (config-looking) file
161+
and foo.test.ts has a code extension."""
162+
if not path:
163+
return "other"
164+
name = str(path).rsplit("/", 1)[-1]
165+
low = name.lower()
166+
if low in _LOCKFILE_NAMES:
167+
return "lockfile"
168+
ext = ""
169+
if "." in name:
170+
ext = "." + name.rsplit(".", 1)[-1].lower()
171+
if _TEST_NAME_RX.search(str(path)):
172+
return "test"
173+
if ext in _CODE_EXTS:
174+
return "code"
175+
if ext in _DOC_EXTS or low.split(".")[0] in _DOC_NAMES:
176+
return "doc"
177+
if ext in _CONFIG_EXTS or low in _CONFIG_NAMES:
178+
return "config"
179+
return "other"
180+
181+
182+
_PLAN_FILE_RX = re.compile(
183+
r'(^|/)\.claude/plans/[^/]*\.md$|(^|/)\.cursor/plans/|(^|/)\.context/[^/]*plan[^/]*$|'
184+
r'(^|/)plans/[^/]+\.(md|mdx|txt)$',
185+
re.I,
186+
)
187+
188+
189+
def is_plan_file_target(path):
190+
"""True when a write target is a durable plan artifact on disk (C3/C4):
191+
`.claude/plans/*.md`, `.cursor/plans/`, `.context/*plan*`, or any
192+
`.md`/`.mdx`/`.txt` file directly inside a `plans/` directory at any depth
193+
(e.g. the superpowers `docs/**/plans/<n>-<name>.md` convention), regardless
194+
of filename — matched by taxonomy so cross-session credit (C4) can recognize
195+
hand-off plan files regardless of source CLI."""
196+
if not path:
197+
return False
198+
return bool(_PLAN_FILE_RX.search(str(path)))
199+
123200

124201
def classify_mcp_subcategory(server_name, tool_name=""):
125202
low = server_name.lower()

tests/fixtures/scoring_vectors.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3964,14 +3964,14 @@
39643964
},
39653965
{
39663966
"direction": "higher",
3967-
"display_target": "0.4 eligible-session coverage",
3967+
"display_target": "0.5 eligible-session coverage",
39683968
"display_value": "0 eligible-session coverage",
39693969
"is_drag": false,
39703970
"label": "Ordered planning readiness",
39713971
"narrative": "No activity recorded for Ordered planning readiness.",
39723972
"pct": 0.5,
39733973
"score_pct": 50,
3974-
"target": 0.4,
3974+
"target": 0.5,
39753975
"unit": "eligible-session coverage",
39763976
"verdict": "adequate",
39773977
"weight": 0.15,
@@ -4389,14 +4389,14 @@
43894389
},
43904390
{
43914391
"direction": "higher",
4392-
"display_target": "0.4 eligible-session coverage",
4392+
"display_target": "0.5 eligible-session coverage",
43934393
"display_value": "0 eligible-session coverage",
43944394
"is_drag": false,
43954395
"label": "Ordered planning readiness",
43964396
"narrative": "No activity recorded for Ordered planning readiness.",
43974397
"pct": 0.5,
43984398
"score_pct": 50,
4399-
"target": 0.4,
4399+
"target": 0.5,
44004400
"unit": "eligible-session coverage",
44014401
"verdict": "adequate",
44024402
"weight": 0.15,

0 commit comments

Comments
 (0)