Skip to content

fix(monitor): tolerate corrupt lines in cycle-history.jsonl (no false "All clear") #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jodunk wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(monitor): tolerate corrupt lines in cycle-history.jsonl (no false "All clear") #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 53 additions & 43 deletions monitor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ PAUSE_FLAG="$PROJECT_DIR/.auto-loop-paused"
LABEL="com.auto-co.loop"
HISTORY_FILE="$LOG_DIR/cycle-history.jsonl"

# jq over the cycle history, tolerant of a stray or corrupt line in the JSONL.
# Without this guard one malformed line aborts `jq -s`, turning every analytics
# flag (--alerts/--costs/--compare/--trend/--dashboard/--health) into parse
# errors that silently report "All clear" -- a false-negative in monitoring.
# `jq -R 'fromjson? // empty'` parses EACH line independently, so a bad line is
# skipped without swallowing its valid neighbours (jq's stream-resync would).
# jqh = slurp mode (whole-array filters); jqhr = streaming mode (per-object).
jqh() { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -s "$@"; }
jqhr() { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -r "$@"; }
Comment on lines +39 to +40

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '"partial record"' '{"cycle":1}' |
  jq -R 'fromjson? // empty' |
  jq -s '[.[] | type]'
# Current result: ["string","object"]; the string must not reach analytics.

Repository: NikitaDmitrieff/auto-co-meta

Length of output: 193

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Files named monitor.sh:\n'
fd -a '^monitor\.sh$' . || true

printf '\nmonitor.sh outline if present:\n'
if [ -f monitor.sh ]; then
  ast-grep outline monitor.sh || true
  echo "\nmonitor.sh relevant content (first 220 lines):\n"
  sed -n '1,220p' monitor.sh | nl -ba
fi

printf '\nSearch jqh/jqhr usages:\n'
rg -n --hidden --glob '!node_modules' 'jqh|jqhr' . || true

printf '\nSearch history file references:\n'
rg -n --hidden --glob '!node_modules' 'HISTORY_FILE|history' . || true

Repository: NikitaDmitrieff/auto-co-meta

Length of output: 369

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'monitor.sh line count:\n'
wc -l monitor.sh

printf '\nmonitor.sh lines 1-260:\n'
awk '{printf "%5d\t%s\n", NR, $0}' monitor.sh | sed -n '1,260p'

printf '\nSearch jqh/jqhr usages:\n'
awk 'NF && (NR>=1 && NR<=260)' monitor.sh | grep -nE 'jqh|jqhr|HISTORY_FILE|model|status|cost|group' -n || true

printf '\nShell behavior probe for common downstream selectors:\n'
tmp="$(mktemp)"
printf '%s\n' '"partial record"' '42' 'true' '{"partial":1}' '{"status":"ok","cost":0,"model":"a"}' > "$tmp"
jq -R 'fromjson? // empty' "$tmp" | \
  jq -R 'fromjson? // empty' "$tmp" | \
  jq -s '
    ["status", "cost", "model"] as $keys |
    {
      valid_cycle: [.[] | select(type=="object" and has("status") and has("cost") and has("model"))],
      only_string: ["partial record"],
      object_no_keys: [{partial:1}],
      all: [.[]]
    }
  '
rm -f "$tmp"

Repository: NikitaDmitrieff/auto-co-meta

Length of output: 14776

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Remaining monitor.sh content from line 260 onward:\n'
awk '{printf "%5d\t%s\n", NR, $0}' monitor.sh | sed -n '260,534p'

printf '\njq error probes mirroring monitor.sh downstream selectors:\n'
tmp="$(mktemp)"
printf '%s\n' \
  '{"status":"ok","cost":0,"model":"a"}' \
  '"partial record"' \
  '42' \
  '{"cycle":1}' \
  '{"status":"ok","cost":0,"model":"a"}' \
  > "$tmp"

for expr in \
  '[.[-10:] | .[] | select(.status=="fail")]' \
  'select(.status=="fail")' \
  '.[-10:] | .[] | select(.status=="fail") | "    Cycle \(.cycle): exit \(.exit_code), \(.duration_s)s"' \
  '[.[].cost] | add' \
  'select(.status=="ok")' \
  'select(.status=="fail")' \
  '[.[].duration_s] | add' \
  'group_by(.model) | map({model: .[0].model, cost: ([.[].cost]|add)})'
do
  echo "--- $expr ---"
  jq -R 'fromjson? // empty' "$tmp" | jq "$expr" 2>&1 || true
done

rm -f "$tmp"

printf '\nProgrammatic extraction of jqh/jqhr call locations in monitor.sh:\n'
python3 - <<'PY'
import re
from pathlib import Path
p=Path("monitor.sh")
text=p.read_text()
for m in re.finditer(r'\b(jqh|jqhr)\b', text):
    lineno=text.count('\n',0,m.start())+1
    start=max(0,m.start()-50); end=min(len(text),m.end()+120)
    snippet=text[start:end].replace('\n','|')
    print(f"{lineno}: {snippet}")
PY

Repository: NikitaDmitrieff/auto-co-meta

Length of output: 24255

Filter JSON-valid non-cycle values in jqh/jqhr.

fromjson? skips malformed lines but keeps JSON scalars like "partial record" or 42; analytics calls then try .status, .cost, .duration_s, .model, etc. and fail. Restrict parsed values to cycle objects, and validate required fields if partial records can be logged.

Proposed fix 
-jqh()  { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -s "$@"; }
-jqhr() { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -r "$@"; }
+jqh()  { jq -R '(fromjson? // empty) | select(type == "object")' "$HISTORY_FILE" | jq -s "$@"; }
+jqhr() { jq -R '(fromjson? // empty) | select(type == "object")' "$HISTORY_FILE" | jq -r "$@"; }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
jqh() { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -s "$@"; }
jqhr() { jq -R 'fromjson? // empty' "$HISTORY_FILE" | jq -r "$@"; }
jqh() { jq -R '(fromjson? // empty) | select(type == "object")' "$HISTORY_FILE" | jq -s "$@"; }
jqhr() { jq -R '(fromjson? // empty) | select(type == "object")' "$HISTORY_FILE" | jq -r "$@"; }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@monitor.sh` around lines 39 - 40, Update the jqh and jqhr helper filters to
retain only parsed cycle record objects, excluding JSON scalars and other
non-cycle values before analytics queries run. Validate the required cycle
fields used by downstream queries, such as status, cost, duration_s, and model,
when partial records may be present.


# Shared alerts logic used by --alerts and --health
check_alerts() {
local history_file="$HISTORY_FILE"
Expand All @@ -40,21 +50,21 @@ check_alerts() {
fi

local total_cycles
total_cycles=$(jq -s 'length' "$history_file")
total_cycles=$(jqh 'length')

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files named monitor.sh:"
fd -a '^monitor\.sh$' . || true

echo
echo "Relevant line ranges:"
if [ -f monitor.sh ]; then
  sed -n '1,110p' monitor.sh
  echo "---"
  sed -n '220,320p' monitor.sh
  echo "---"
  sed -n '350,425p' monitor.sh
  echo "---"
  sed -n '505,525p' monitor.sh
else
  echo "monitor.sh not found at repo root"
fi

echo
echo "Search for total_cycles / jqh / jq:"
rg -n "total_cycles|jqh|jq |jq\\(" monitor.sh || true

echo
echo "Behavioral probe for jq/jqh expression semantics:"
python3 - <<'PY'
import json, subprocess, sys, tempfile, os
cases = [
    ("corrupt only", "{garbage}\n{bad"),
    ("empty", ""),
    ("valid line plus corrupt", '{"cycle":1}\n{bad'),
]
for name, lines in cases:
    with tempfile.NamedTemporaryFile("w+", delete=False) as f:
        f.write(lines)
        path = f.name
    out = subprocess.run(["jqh","if . then length else 0 end", "<", path], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
    print(name, repr(lines).replace('\n', "\\n"), "jqh_rc=", out.returncode, "stdout=", repr(out.stdout), "stderr=", repr(out.stderr))
PY

echo
echo "Check jqh binary availability:"
command -v jqh || true
jqh --help 2>/dev/null | head -n 20 || true

Repository: NikitaDmitrieff/auto-co-meta

Length of output: 20311

🌐 Web query:

jq length null array length return 0 division by zero add empty behavior

💡 Result:

In jq, the length function, division operator, and add filter behave as follows: Length Function The length of a null value is 0 [1][2]. The length of an empty array is also 0 [1][2]. Division by Zero Division by zero (e.g., x / 0) raises a division by zero error [3][4][5]. Historically, implementations of constant folding occasionally caused variations in behavior (such as NaN instead of an error), but current stable versions consistently throw an error when a divisor is zero [3][5]. Add Filter The add filter takes an array as input and sums, concatenates, or merges its elements depending on their types [6][7]. If the input is an empty array, add returns null [6][7]. This is distinct from empty, which produces no results at all [8][6].

Citations:

🌐 Web query:

jq division by zero division by zero error jq documentation

💡 Result:

In jq, division by zero raises a runtime error [1][2]. When a division operation encounters a zero divisor, jq terminates the operation and reports an error, typically displaying: jq: error (at ): number () and number (0) cannot be divided because the divisor is zero [3]. Key aspects of this behavior include: - Error Handling: Because this raises a runtime error, it can be caught using the ? operator [1][2]. For example, the expression .[] | (1 / .)? will gracefully return 1 and -1 when processing an input array like [1, 0, -1], effectively suppressing the error caused by the division by zero [1][2]. - 0/0 Behavior: While division by zero generally raises an error, historical versions of jq had inconsistencies regarding how 0/0 was handled compared to other zero-division operations. Recent updates have standardized behavior to ensure 0/0 also triggers a division error [4]. - Constant Folding: Improvements in the jq engine (specifically around constant folding) have been implemented to ensure that zero-division errors are handled consistently, whether the operation is a literal calculation in the code or a dynamic operation performed on input data [4]. For robust scripting, if you expect a divisor might be zero, you should either validate the input data before performing the division or use the ? operator to catch and ignore the resulting error [1][2].

Citations:

Treat all-corrupt histories as unavailable telemetry.

When every history line is malformed, jqh reduces it to an empty array, so --alerts can abort on integer comparison and --trend / --health hit zero-cycle division or null-cost parsing. Normalize zero valid records to “no history available” in every mode and add a zero-valid-record regression case.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@monitor.sh` at line 53, Update the history handling around total_cycles=$(jqh
'length') so zero valid records are treated as unavailable history in every
mode, including --alerts, --trend, and --health. Guard downstream integer
comparisons, division, and cost parsing from empty or null telemetry, preserving
normal behavior when valid records exist, and add a regression case covering an
entirely malformed history.


# 1. Recent failures (last 10 cycles)
local recent_fails
recent_fails=$(jq -s '[.[-10:] | .[] | select(.status=="fail")] | length' "$history_file")
recent_fails=$(jqh '[.[-10:] | .[] | select(.status=="fail")] | length')
if [ "$recent_fails" -gt 0 ]; then
echo " [WARN] $recent_fails failures in last 10 cycles"
jq -s '.[-10:] | .[] | select(.status=="fail") | " Cycle \(.cycle): exit \(.exit_code), \(.duration_s)s"' -r "$history_file"
jqh -r '.[-10:] | .[] | select(.status=="fail") | " Cycle \(.cycle): exit \(.exit_code), \(.duration_s)s"'
alerts=$((alerts + 1))
fi

# 2. Cost spike detection (last cycle > 2x average)
if [ "$total_cycles" -ge 3 ]; then
local cost_alert
cost_alert=$(jq -s '
cost_alert=$(jqh -r '
if length < 3 then empty
else
(.[:-1] | [.[].cost] | add / length) as $avg |
Expand All @@ -63,7 +73,7 @@ check_alerts() {
" [WARN] Last cycle cost $\($last) is >2x average ($\($avg | . * 100 | floor / 100))"
else empty end
end
' -r "$history_file")
')
if [ -n "$cost_alert" ]; then
echo "$cost_alert"
alerts=$((alerts + 1))
Expand All @@ -72,7 +82,7 @@ check_alerts() {

# 3. Consecutive failure streak
local streak
streak=$(jq -s '[foreach .[] as $x (0; if $x.status == "fail" then . + 1 else 0 end)] | max' "$history_file")
streak=$(jqh '[foreach .[] as $x (0; if $x.status == "fail" then . + 1 else 0 end)] | max')
if [ "$streak" -ge 3 ]; then
echo " [CRIT] Worst consecutive failure streak: $streak cycles"
alerts=$((alerts + 1))
Expand All @@ -96,7 +106,7 @@ check_alerts() {
# 5. Duration anomaly (last cycle > 2x average duration)
if [ "$total_cycles" -ge 3 ]; then
local dur_alert
dur_alert=$(jq -s '
dur_alert=$(jqh -r '
if length < 3 then empty
else
(.[:-1] | [.[].duration_s] | add / length) as $avg |
Expand All @@ -105,7 +115,7 @@ check_alerts() {
" [WARN] Last cycle took \($last)s (>2x avg \($avg | floor)s)"
else empty end
end
' -r "$history_file")
')
if [ -n "$dur_alert" ]; then
echo "$dur_alert"
alerts=$((alerts + 1))
Expand Down Expand Up @@ -216,14 +226,14 @@ case "${1:-}" in
elif command -v jq &>/dev/null; then
echo ""
echo "Per-cycle breakdown:"
jq -r '" Cycle \(.cycle): $\(.cost) (\(.status), \(.duration_s)s, \(.model))"' "$HISTORY_FILE"
jqhr '" Cycle \(.cycle): $\(.cost) (\(.status), \(.duration_s)s, \(.model))"'
echo ""
total=$(jq -s '[.[].cost] | add' "$HISTORY_FILE")
count=$(jq -s 'length' "$HISTORY_FILE")
ok=$(jq -s '[.[] | select(.status=="ok")] | length' "$HISTORY_FILE")
fail=$(jq -s '[.[] | select(.status=="fail")] | length' "$HISTORY_FILE")
avg=$(jq -s 'if length > 0 then ([.[].cost] | add) / length else 0 end' "$HISTORY_FILE")
avg_dur=$(jq -s 'if length > 0 then ([.[].duration_s] | add) / length | floor else 0 end' "$HISTORY_FILE")
total=$(jqh '[.[].cost] | add')
count=$(jqh 'length')
ok=$(jqh '[.[] | select(.status=="ok")] | length')
fail=$(jqh '[.[] | select(.status=="fail")] | length')
avg=$(jqh 'if length > 0 then ([.[].cost] | add) / length else 0 end')
avg_dur=$(jqh 'if length > 0 then ([.[].duration_s] | add) / length | floor else 0 end')
echo "Total: \$$total across $count cycles ($ok ok, $fail failed)"
echo "Average: \$$avg/cycle, ${avg_dur}s/cycle"
else
Expand Down Expand Up @@ -282,27 +292,27 @@ case "${1:-}" in

# Cost summary from history
if [ -f "$HISTORY_FILE" ] && [ -s "$HISTORY_FILE" ] && command -v jq &>/dev/null; then
total=$(jq -s '[.[].cost] | add' "$HISTORY_FILE")
count=$(jq -s 'length' "$HISTORY_FILE")
ok=$(jq -s '[.[] | select(.status=="ok")] | length' "$HISTORY_FILE")
fail=$(jq -s '[.[] | select(.status=="fail")] | length' "$HISTORY_FILE")
avg=$(jq -s 'if length > 0 then ([.[].cost] | add) / length | . * 100 | floor / 100 else 0 end' "$HISTORY_FILE")
avg_dur=$(jq -s 'if length > 0 then ([.[].duration_s] | add) / length | floor else 0 end' "$HISTORY_FILE")
total=$(jqh '[.[].cost] | add')
count=$(jqh 'length')
ok=$(jqh '[.[] | select(.status=="ok")] | length')
fail=$(jqh '[.[] | select(.status=="fail")] | length')
avg=$(jqh 'if length > 0 then ([.[].cost] | add) / length | . * 100 | floor / 100 else 0 end')
avg_dur=$(jqh 'if length > 0 then ([.[].duration_s] | add) / length | floor else 0 end')

echo " ┌─────────────────────────────────────────────────────┐"
printf " │ Cycles: %-5s OK: %-5s Fail: %-5s │\n" "$count" "$ok" "$fail"
printf " │ Total cost: \$%-10s Avg: \$%-8s/cycle │\n" "$total" "$avg"
printf " │ Avg duration: %ss/cycle │\n" "$avg_dur"
# Cost trend sparkline (last 10 cycles)
spark=$(jq -s '[ .[-10:][].cost ] | [ range(length) as $i | {v: .[$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map(["▁","▂","▃","▄","▅","▆","▇","█"][.]) | join("")' "$HISTORY_FILE" 2>/dev/null | tr -d '"')
spark=$(jqh '[ .[-10:][].cost ] | [ range(length) as $i | {v: .[$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map(["▁","▂","▃","▄","▅","▆","▇","█"][.]) | join("")' | tr -d '"')
printf " │ Cost trend (last 10): %-28s │\n" "$spark"
echo " └─────────────────────────────────────────────────────┘"

echo ""
echo " Last 10 cycles:"
printf " %-6s %-6s %-9s %-8s %s\n" "Cycle" "Status" "Cost" "Duration" "Model"
echo " ────── ────── ───────── ──────── ──────"
jq -r '"\(.cycle)\t\(.status)\t$\(.cost)\t\(.duration_s)s\t\(.model)"' "$HISTORY_FILE" \
jqhr '"\(.cycle)\t\(.status)\t$\(.cost)\t\(.duration_s)s\t\(.model)"' \
| tail -10 \
| while IFS=$'\t' read -r cyc st cost dur model; do
if [ "$st" = "ok" ]; then
Expand Down Expand Up @@ -345,7 +355,7 @@ case "${1:-}" in
printf "%-12s %7s %10s %10s %10s %8s\n" "Model" "Cycles" "Total" "Avg Cost" "Avg Dur" "OK Rate"
echo "──────────── ─────── ────────── ────────── ────────── ────────"

jq -s '
jqh -r '
group_by(.model) | .[] |
{
model: .[0].model,
Expand All @@ -357,15 +367,15 @@ case "${1:-}" in
ok_pct: (([.[] | select(.status=="ok")] | length) * 100 / length | floor)
} |
"\(.model)\t\(.count)\t$\(.total)\t$\(.avg_cost)\t\(.avg_dur)s\t\(.ok_pct)%"
' -r "$HISTORY_FILE" \
' \
| sort -t$'\t' -k2 -rn \
| while IFS=$'\t' read -r model count total avg_cost avg_dur ok_pct; do
printf "%-12s %7s %10s %10s %10s %8s\n" "$model" "$count" "$total" "$avg_cost" "$avg_dur" "$ok_pct"
done

echo ""
echo "Cost efficiency (lower = better):"
jq -s '
jqh -r '
group_by(.model) | .[] |
{
model: .[0].model,
Expand All @@ -376,7 +386,7 @@ case "${1:-}" in
)
} |
" \(.model): $\(.cost_per_ok) per successful cycle"
' -r "$HISTORY_FILE"
'
;;

--trend)
Expand All @@ -386,27 +396,27 @@ case "${1:-}" in
exit 1
fi

total_cycles=$(jq -s 'length' "$HISTORY_FILE")
total_cycles=$(jqh 'length')
shown=$((total_cycles < N ? total_cycles : N))

echo "=== Cost & Duration Trend (last $shown of $total_cycles cycles) ==="
echo ""

# Cost sparkline
cost_spark=$(jq -s "[ .[-${N}:][].cost ] | [ range(length) as \$i | {v: .[\$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map([\"▁\",\"▂\",\"▃\",\"▄\",\"▅\",\"▆\",\"▇\",\"█\"][.]) | join(\"\")" "$HISTORY_FILE" 2>/dev/null | tr -d '"')
cost_min=$(jq -s "[ .[-${N}:][].cost ] | min | . * 100 | floor / 100" "$HISTORY_FILE")
cost_max=$(jq -s "[ .[-${N}:][].cost ] | max | . * 100 | floor / 100" "$HISTORY_FILE")
cost_avg=$(jq -s "[ .[-${N}:][].cost ] | add / length | . * 100 | floor / 100" "$HISTORY_FILE")
cost_spark=$(jqh "[ .[-${N}:][].cost ] | [ range(length) as \$i | {v: .[\$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map([\"▁\",\"▂\",\"▃\",\"▄\",\"▅\",\"▆\",\"▇\",\"█\"][.]) | join(\"\")" | tr -d '"')
cost_min=$(jqh "[ .[-${N}:][].cost ] | min | . * 100 | floor / 100")
cost_max=$(jqh "[ .[-${N}:][].cost ] | max | . * 100 | floor / 100")
cost_avg=$(jqh "[ .[-${N}:][].cost ] | add / length | . * 100 | floor / 100")

printf " Cost: %s\n" "$cost_spark"
printf " min: \$%-8s avg: \$%-8s max: \$%-8s\n" "$cost_min" "$cost_avg" "$cost_max"
echo ""

# Duration sparkline
dur_spark=$(jq -s "[ .[-${N}:][].duration_s ] | [ range(length) as \$i | {v: .[\$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map([\"▁\",\"▂\",\"▃\",\"▄\",\"▅\",\"▆\",\"▇\",\"█\"][.]) | join(\"\")" "$HISTORY_FILE" 2>/dev/null | tr -d '"')
dur_min=$(jq -s "[ .[-${N}:][].duration_s ] | min" "$HISTORY_FILE")
dur_max=$(jq -s "[ .[-${N}:][].duration_s ] | max" "$HISTORY_FILE")
dur_avg=$(jq -s "[ .[-${N}:][].duration_s ] | add / length | floor" "$HISTORY_FILE")
dur_spark=$(jqh "[ .[-${N}:][].duration_s ] | [ range(length) as \$i | {v: .[\$i], min: min, max: max} ] | [.[] | if .max == .min then 4 else (((.v - .min) / (.max - .min) * 7) | floor) end] | map([\"▁\",\"▂\",\"▃\",\"▄\",\"▅\",\"▆\",\"▇\",\"█\"][.]) | join(\"\")" | tr -d '"')
dur_min=$(jqh "[ .[-${N}:][].duration_s ] | min")
dur_max=$(jqh "[ .[-${N}:][].duration_s ] | max")
dur_avg=$(jqh "[ .[-${N}:][].duration_s ] | add / length | floor")

printf " Duration: %s\n" "$dur_spark"
printf " min: %-8ss avg: %-8ss max: %-8ss\n" "$dur_min" "$dur_avg" "$dur_max"
Expand All @@ -416,14 +426,14 @@ case "${1:-}" in
echo " Details:"
printf " %-7s %-9s %-9s %-10s %s\n" "Cycle" "Cost" "Duration" "Status" "Model"
echo " ------- --------- --------- ---------- ------"
jq -s ".[-${N}:][] | \"\(.cycle)\t\$\(.cost)\t\(.duration_s)s\t\(.status)\t\(.model)\"" -r "$HISTORY_FILE" \
jqh -r ".[-${N}:][] | \"\(.cycle)\t\$\(.cost)\t\(.duration_s)s\t\(.status)\t\(.model)\"" \
| while IFS=$'\t' read -r cyc cost dur st model; do
printf " %-7s %-9s %-9s %-10s %s\n" "$cyc" "$cost" "$dur" "$st" "$model"
done

# Running total trend
echo ""
total=$(jq -s '[.[].cost] | add | . * 100 | floor / 100' "$HISTORY_FILE")
total=$(jqh '[.[].cost] | add | . * 100 | floor / 100')
echo " Cumulative cost: \$$total across $total_cycles cycles"
;;

Expand Down Expand Up @@ -502,11 +512,11 @@ case "${1:-}" in
# 5. Quick stats
if [ -f "$HISTORY_FILE" ] && [ -s "$HISTORY_FILE" ] && command -v jq &>/dev/null; then
echo "--- Stats ---"
total_cycles=$(jq -s 'length' "$HISTORY_FILE")
ok=$(jq -s '[.[] | select(.status=="ok")] | length' "$HISTORY_FILE")
fail=$(jq -s '[.[] | select(.status=="fail")] | length' "$HISTORY_FILE")
total=$(jq -s '[.[].cost] | add' "$HISTORY_FILE")
rate=$(jq -s '([.[] | select(.status=="ok")] | length) * 100 / length | floor' "$HISTORY_FILE")
total_cycles=$(jqh 'length')
ok=$(jqh '[.[] | select(.status=="ok")] | length')
fail=$(jqh '[.[] | select(.status=="fail")] | length')
total=$(jqh '[.[].cost] | add')
rate=$(jqh '([.[] | select(.status=="ok")] | length) * 100 / length | floor')
echo " Cycles: $total_cycles ($ok ok, $fail failed) | Success rate: ${rate}% | Total cost: \$$total"
fi
;;
Expand Down
53 changes: 53 additions & 0 deletions tests/test-monitor-resilience.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/bin/bash
# test-monitor-resilience.sh -- monitor.sh analytics must tolerate a stray or
# corrupt line in logs/cycle-history.jsonl instead of aborting and falsely
# reporting "All clear" (a false-negative in monitoring).
#
# Regression guard: before the fix, one malformed line aborted `jq -s` for the
# whole file, so --alerts/--costs/--compare/--trend/--dashboard/--health all
# emitted parse errors and reported empty / zero cycle counts.
set -euo pipefail

REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
command -v jq >/dev/null 2>&1 || { echo "[!] jq required"; exit 1; }

TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
H="$TMP/cycle-history.jsonl"

# Two valid cycles + one stray non-JSON line (the real repo had exactly this).
{
printf '{"cycle":1,"status":"ok","cost":0.10,"duration_s":10,"model":"opus","exit_code":0}\n'
printf 'NOT-JSON stray line | cycle=x | corrupt\n'
printf '{"cycle":2,"status":"fail","cost":0.20,"duration_s":20,"model":"opus","exit_code":1}\n'
} > "$H"

pass=0; fail=0
ok() { echo " [+] PASS: $1"; pass=$((pass+1)); }
no() { echo " [-] FAIL: $1"; fail=$((fail+1)); }

echo "Resilience unit (the jqh pattern monitor.sh now uses):"

# 1. The OLD bare-slurp pattern must abort on a corrupt line (documents the bug).
old=$(jq -s 'length' "$H" 2>/dev/null || true)
if [ -z "$old" ]; then ok "bare jq -s aborts on corrupt line (bug reproduced)"; else no "bare jq -s returned '$old' (expected no output)"; fi

# 2. The resilient per-line filter counts ALL valid lines (a neighbour of the
# bad line must NOT be swallowed -- jq stream-resync would drop it).
got=$(jq -R 'fromjson? // empty' "$H" | jq -s 'length')
if [ "$got" = "2" ]; then ok "per-line filter counts 2 valid lines (got $got)"; else no "expected 2 valid lines, got '$got'"; fi

# 3. Failure detection survives the filter (the false-green regression).
fails=$(jq -R 'fromjson? // empty' "$H" | jq -s '[.[-10:] | .[] | select(.status=="fail")] | length')
if [ "$fails" = "1" ]; then ok "fail-count still detects 1 failure through filter"; else no "expected 1 failure detected, got '$fails'"; fi

echo "Structural guard (monitor.sh wired correctly):"

# 4. monitor.sh wires the per-line resilience filter into its history reads.
if grep -qE 'fromjson\?' "$REPO_ROOT/monitor.sh"; then ok "monitor.sh uses per-line fromjson? filter"; else no "monitor.sh missing fromjson? filter"; fi

# 5. No bare slurp/stream jq over HISTORY_FILE remains (all reads route via helper).
if grep -qE 'jq -(s|r) [^|]*"\$HISTORY_FILE"' "$REPO_ROOT/monitor.sh"; then no "bare jq -s/-r over \$HISTORY_FILE remains"; else ok "all history reads route through resilient helper"; fi

echo "Results: $pass passed, $fail failed"
[ "$fail" -eq 0 ]