Skip to content

Commit f14941d

Browse files
michaeldtimpeclaude
andcommitted
bench+config: ship readiness — pin temp=0 in production, reject IPv4 in citation linter
Two pre-ship fixes that closed the last gap between the temp=0 probe result (8/10) and the production-config confirmation (which initially reported 7/10). configs/single_64gb.yaml — promote temperature: 0.2 → 0.0. The 2026-05-01 PM variance probe established that the variance in the v1 baseline (4-7/10 across three identical runs at temp=0.2) was sampling-driven; pinning temp=0 collapsed pass/fail vectors to deterministic across two back-to-back full sweeps. Greedy decoding also lifted the implement category to 4/4 baseline ceiling out of the box (the lift Phase 1 attributed to structural prompts was plausibly a baseline-variance artifact). The variant override in variants_v1_temp0_probe.yaml stays in place for future variance probes. src/luxe/citations.py — reject IPv4-shaped paths in extract_citations. The citation regex `[\w./_-]+\.[\w]+:\d+` was matching IPv4 host:port references like `127.0.0.1:27001` as `path:line` citations. Without this guard, isomer-quickstart's synthesizer report (which legitimately mentions `http://127.0.0.1:27001/` for the dashboard) reported 2 unresolved citations and lost the build-breaking citation gate. The guard rejects paths matching `(?:^|/)\d+\.\d+\.\d+\.\d+$` — handles both bare `127.0.0.1` and URL-form `//127.0.0.1` (the citation regex greedily eats leading `/` from `http://...`). Two new tests in test_citations_diff_aware.py: one asserts host:port references are dropped while a real `app.py:42` citation in the same text is kept; one asserts dotted filenames with digits (e.g. `v1.2.3.py:10`) survive. Sidecar regrade against the existing isomer-quickstart commit can't test this fix (citations are computed at run time and persisted by the agent loop, not by the grader). Confirmed via fresh re-run: isomer-quickstart now PASSes (cite=0/0). Cumulative confirmation result is 8/10 — v1 ship gate cleared. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8d1fcd3 commit f14941d

3 files changed

Lines changed: 55 additions & 1 deletion

File tree

configs/single_64gb.yaml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,13 @@ roles:
2525
num_ctx: 32768
2626
max_steps: 30
2727
max_tokens_per_turn: 8192
28-
temperature: 0.2
28+
temperature: 0.0 # 2026-05-01 promotion: variance probe showed
29+
# temp=0.2 produced ±2 fixture swings between
30+
# identical bench runs (4-7/10 across 3 baseline
31+
# samples); temp=0 collapses pass/fail vectors to
32+
# deterministic across two back-to-back runs.
33+
# Greedy decoding also lifts implement to 4/4
34+
# ceiling out of the box. See variants_v1_temp0_probe.yaml.
2935
tools:
3036
- read_file
3137
- list_dir

src/luxe/citations.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,15 @@ def is_cleared(self) -> bool:
6161
_CITATION_RE = re.compile(
6262
r"`?(?P<path>[\w./_-]+\.[\w]+):(?P<line>\d+)(?:-(?P<line_end>\d+))?`?"
6363
)
64+
# Reject IPv4-shaped paths — `127.0.0.1:8000` matches the citation regex
65+
# (path=127.0.0.1 because `.1` looks like a `.ext` suffix, line=8000) but is
66+
# almost always a host:port reference in deployment docs, not a file:line
67+
# citation. Without this guard, isomer-quickstart's synthesizer report
68+
# (which mentions `127.0.0.1:27001` for the dashboard URL) reports 2
69+
# unresolved citations and fails the build-breaking gate. Anchor handles
70+
# both bare `127.0.0.1` and URL-form `//127.0.0.1` (the citation regex's
71+
# path group greedily eats leading `/` characters from `http://...`).
72+
_IPV4_PATH_RE = re.compile(r"(?:^|/)\d+\.\d+\.\d+\.\d+$")
6473
_FUZZY_WINDOW = 20
6574

6675

@@ -118,6 +127,9 @@ def extract_citations(text: str) -> list[Citation]:
118127
# 1.2.3:4 would never match because they lack a letter in the extension).
119128
if "." not in path:
120129
continue
130+
# Skip IPv4-shaped paths (`127.0.0.1:8000` is a host:port, not file:line).
131+
if _IPV4_PATH_RE.search(path):
132+
continue
121133
key = (path, line, line_end)
122134
if key in seen:
123135
continue

tests/test_citations_diff_aware.py

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,3 +190,39 @@ def test_dedupe_citations():
190190
text = "`src/foo.py:1` and `src/foo.py:1` again"
191191
cs = extract_citations(text)
192192
assert len(cs) == 1
193+
194+
195+
def test_extract_citations_rejects_ipv4_host_port():
196+
"""`127.0.0.1:8000` is a host:port reference (deployment doc), not a
197+
file:line citation. The extractor must skip IPv4-shaped paths so the
198+
citation linter doesn't flag dashboard URLs as unresolved.
199+
200+
Regression: isomer-quickstart synthesizer reports referenced
201+
`127.0.0.1:27001` for the dashboard and the build-breaking citation
202+
gate then docked the fixture's score (Phase 1 ship-confirmation
203+
2026-05-02). Surgical fix at extractor: reject paths matching
204+
`^\\d+\\.\\d+\\.\\d+\\.\\d+$`.
205+
"""
206+
text = (
207+
"Run with `docker compose up`; the dashboard is at "
208+
"`http://127.0.0.1:27001/`. The mapping `127.0.0.1:27001:27001` "
209+
"is in `docker-compose.yml`. A real citation: `app.py:42`."
210+
)
211+
cs = extract_citations(text)
212+
paths = {c.path for c in cs}
213+
assert "127.0.0.1" not in paths
214+
# The legitimate file:line reference should still be extracted.
215+
assert "app.py" in paths
216+
# Both IP-shaped strings rejected; only the real one survives.
217+
assert len(cs) == 1
218+
219+
220+
def test_extract_citations_keeps_dotted_filenames_with_digits():
221+
"""`v1.2.3.py:10` is a real (if unusual) filename; the IPv4 guard
222+
must NOT reject it. Only paths that fully match `\\d+\\.\\d+\\.\\d+\\.\\d+`
223+
(no extension) are dropped — `v1.2.3.py` has a `.py` extension and
224+
a non-digit prefix in the leading segment."""
225+
text = "See `v1.2.3.py:10` for the override."
226+
cs = extract_citations(text)
227+
assert len(cs) == 1
228+
assert cs[0].path == "v1.2.3.py"

0 commit comments

Comments
 (0)