Repository navigation
Commit 159d72e
GLOOK-54: repair the model's JSON, and bust the proxy cache when repair can't (#73)
* diag(project-insights): log the parse-failure window
/api/project-insights has been returning 500 on roughly one report a week
since 2026-08-25 — five so far — and nobody has been able to say why, because
the log records content_chars and never the content:
[project-insights] report=da2fa78f finish_reason=stop prompt_chars=203278
content_chars=14280 llm_projects=0
FAILURE="json parse: Expected double-quoted property name in JSON at position 2318"
Not truncation (finish_reason=stop, 14k of a 32k budget) and not a cached
failure (insights.ts already refuses to persist one). Data-dependent: three
reports fail while six of the same output size succeed, and the two live
failures break at position 2298 and 2318 — close enough to suggest a structural
artifact rather than randomness.
Logs a 300-char window either side of V8's reported offset, plus the head and
tail, which is where provider artifacts live: a prose preamble, a trailing
comma, an unterminated string. Verified the `position (\d+)` regex against
V8's actual message format — and note that JSON.parse('{"a":1,}') produces
exactly this error family, so a trailing comma is the leading hypothesis for
the window to confirm or kill.
Diagnostics only: no behaviour change, and deliberately console.warn rather
than err.message, because route.ts serialises err.message into the 500 body
and raw model output must not reach the client.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* GLOOK-54: repair the model's JSON, and bust the proxy cache when repair can't
The Top Projects card had been broken for weeks by ONE character. The
diagnostics added in the previous commit caught it:
…"RPS-10766","RPS-10768"],
}, <- trailing comma before }
{"name":"Video/Audio Processing", …
The model closes a jira_keys array, emits a comma, then a brace. JSON forbids
that, so JSON.parse reports "Expected double-quoted property name" at the
brace — it is looking for the next key. Not truncation (finish_reason=stop, 14k
of a 32k budget); the rest of the 14,268-char response is well-formed.
Reloading never helped because the AI Proxy re-serves cached completions. That
was measured, not assumed — a prompt asking for a random hex string:
identical #1 2431ms -> a3f9c2e7b1d4
identical #2 224ms -> a3f9c2e7b1d4 same value, 10x faster => CACHED
message nonce 1866ms -> a3f9c21b7e4d different value => salt works
extraBody nonce 38ms -> a3f9c2e7b1d4 same value => salt IGNORED
So the salt has to go in the user message. A nonce in
smartling_additional_properties is ignored by the proxy, and putting it there
would have shipped a retry that silently re-served the same broken JSON.
Three parts:
- src/lib/llm-json.ts: parseModelJson() strips provider fences, and on a
failed parse strips commas that sit before } or ]. The stripper SCANS
string literals rather than pattern-matching, because this payload is full
of Jira summaries and commit messages that can contain ", }" — a bare
replace(/,\s*([}\]])/g,'$1') corrupts them silently. Same hazard as
GLOOK-41's outsideStringLiterals, but JSON escapes a quote with a backslash
where SQL doubles it, so the scanner tracks escapes instead. Repair is only
attempted after a clean parse fails, so valid output is never touched, and
`repaired` is reported so a drift in model behaviour stays visible.
- insights.ts retries once with a salt when repair cannot help. That covers
artifacts repair cannot fix — report c28781b4 failed with
"Unexpected token 'r'", i.e. prose, not a comma.
- The retry is bounded by a 10-minute per-report marker on globalThis
(matching the progress and stop-signal stores). Successes are cached, so
the model is normally called about once per report — but failures are
deliberately NOT cached (incident 2026-08-03), so every page load re-enters
this path. Today that is cheap because the proxy serves the cached
completion in ~200ms; a salted retry is a deliberate cache miss costing a
full ~200k-char generation, and da2fa78f logged 15 failure calls in one day.
Without the marker, one broken report buys a fresh generation per page load.
ModelJsonError carries the position, window, head and tail for logging but
keeps model output OUT of `message`, because route.ts serialises err.message
into the 5xx body and this payload contains internal commit text. There is a
test asserting exactly that.
Honest scope: repair is the part that can be promised — the artifact is
characterised and the bytes are in hand. Whether a fresh generation avoids the
comma is unknown; the retry is a safety net, not a guarantee. The 2026-08-03
invariant is preserved: a truncated completion is still never repaired and
never cached, with a test pinning it.
129 suites / 1276 tests; tsc --noEmit and npm run build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* GLOOK-54: stop leaking model output into the 500 body, and stop wasting the retry
Review found the confidentiality property this PR claimed — and certified with
a test — was false, and that the test could not fail.
V8 has two SyntaxError shapes and only one is content-free:
Expected double-quoted property name in JSON at position 35
Unexpected token 'I', "INTERNAL-COMMIT-TEXT" is not valid JSON
ModelJsonError interpolated V8's message into super(), so the second shape put
model bytes into .message — which route.ts serialises into the 500 body, and
this payload carries commit messages and Jira summaries. The guard test passed
only because its fixture failed at position 35, the one shape that carries no
snippet. Verified by execution: a position-0 fixture leaks.
`message` is now fixed text. V8's own message moves to a `detail` field
alongside window/head/tail, under the same rule: logs only.
The same fact had broken the diagnostics. `/position (\d+)/` matches only the
positional shape, so for "Unexpected token" the position was -1 and the window
empty — and insights.ts guarded its window log on `position >= 0`. That is
exactly the prose-preamble case (report c28781b4) the window logging was added
for, so the diagnostic was missing precisely where it was needed. The window
now falls back to the head of the document, and the log prints unconditionally.
Three more, all found by review:
- Truncation was never excluded from the retry. finishReason gated nothing,
so a finish_reason=length body reached mayRetry and spent the report's one
regeneration re-sending a prompt the salt had made STRICTLY LONGER at the
same token limit — re-truncating with near-certainty. A permanently
oversized report bought a doomed ~200k-char generation every cooldown
window, indefinitely. Now a truncated completion is never repaired, never
retried, never cached, stated as a rule rather than left as an accident of
the current repair only deleting characters.
- Emptiness was tested on the RAW content, so a fence-only completion — the
artifact stripJsonFences exists for — skipped 'empty completion', failed as
a parse error, and bought a regeneration. Now tested on the stripped text.
- The retry's request shared a try/catch with its parse, so a 429, timeout or
auth expiry became `json parse after salted retry: <provider message>`:
wrong in errors.log, a second route for provider text into the 500 body,
and it consumed the cooldown for something that was never a JSON problem.
It also logged the FIRST attempt's bytes, because the reassignments never
ran. The await is now outside the parse.
- The cooldown marker was never cleared on success. A retry that succeeded
but attributed 0 projects is served uncached, so the next load re-entered,
got the proxy's cached broken completion, found mayRetry() false, and
turned the card into a 500 for the rest of the window — with a good
generation already discarded. Cleared on any usable generation, and the
store is pruned so it cannot grow one entry per failing report forever.
The doc comment now says the bound is PER PROCESS. This ships as a container
image with GHCR publishing and a compose file, so the real ceiling is N
regenerations per window across N replicas, reset on deploy; a true global
bound needs the marker persisted, which is noted on the ticket rather than
done here.
Tests: the guard is now parameterised over all three V8 shapes including the
two that leak, and asserts message equality rather than absence. Added: no
retry on truncation (with call counts pinned on the pre-existing truncation
tests too), fence-only treated as empty, a transport failure labelled as
transport and carrying no provider text, and the cooldown cleared after a
usable generation so a later failure can still retry.
129 suites / 1284 tests; tsc --noEmit and npm run build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: msogin <msogin@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent 165b65a commit 159d72e
4 files changed
Lines changed: 602 additions & 12 deletions
File tree
- src/lib
- __tests__/unit
- projects
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
55 | 58 | | |
56 | 59 | | |
57 | 60 | | |
| |||
62 | 65 | | |
63 | 66 | | |
64 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
65 | 73 | | |
66 | 74 | | |
67 | 75 | | |
| |||
89 | 97 | | |
90 | 98 | | |
91 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
0 commit comments