wip: auto-checkpoint from Cursor agent session 20260419T130425

Author: sjqtentacles

docs/POLYGLOT.md

@@ -1,8 +1,10 @@
 # Forge on Aider polyglot
 
-**Current standing (as of 2026-04-18, single-run, Mode A worker-only):**
-**Python 34/34 (100%)** + **Rust 30/30 (100%)**. 161 of 225 exercises
-remain. See Stage 2 / Stage 2b below for the full numbers and caveats.
+**Current standing (as of 2026-04-19, Mode A worker-only):**
+**Python 34/34 x 3 runs (100%, median $6.53/run)**, **Rust 30/30
+(100%, single-run)**, **Go 5/5 smoke (100%)**. 156 of 225 exercises
+remain (full Go + JS + C++ + Java). See Stage 2 / Stage 2b / Open
+items below.
 
 Forge's second cross-benchmark evaluation. After SWE-Bench
 (`docs/SWE-BENCH.md`) pointed out that real-world agent-scale tasks
@@ -249,16 +251,61 @@ To produce a headline "Forge scores X% on Aider polyglot" number:
       always 0 (key-name mismatch in the harness)
 - [x] Add the 4 remaining language runners (go / javascript / cpp /
       java) with stage-time un-skip preprocessing
-- [ ] Re-run Python x3 and report median + variance (single run
-      isn't publishable). Aggregator is ready in
-      `scripts/_polyglot_aggregate.py`.
-- [ ] Re-run Rust x3 and report median + variance (same caveat)
+- [x] Re-run Python x3 and report median + variance. Three full
+      runs with a live Bedrock key on 2026-04-19:
+      **34/34 / 34/34 / 34/34 PASS**. Median cost **$6.53/run**
+      (range $6.43-$7.46). Median mean-wall-per-exercise **85.8s**
+      (range 81.8-95.9s). Zero regressions across runs. Aggregate
+      at `.forge/eval/polyglot/python-aggregate-20260419T102859.json`.
+      Python baseline is saturated under Mode A.
+- [ ] Re-run Rust x3 and report median + variance (Rust still
+      single-run)
 - [ ] Investigate `scale-generator` 32773s wall-clock anomaly
       (likely measurement artifact; instrument LLM-vs-test wall
       separately)
-- [ ] Run the remaining 161 exercises once a valid Bedrock key is
-      available (S3 Task 3)
-- [ ] Promote to `forge eval polyglot` (S3 Task 4-5)
+- [ ] Run the remaining 161 exercises (Go / JS / C++ / Java).
+      Go 5-exercise smoke: **5/5 PASS**, 276s total, $0.93 total
+      (`.forge/eval/polyglot/go-20260419T125821/`). Full 34-Go
+      sweep is the next target; JS / C++ / Java are the most
+      likely places to see real failures.
+- [x] Promote to `forge eval polyglot` (S3 Task 4-5)
+
+### Reasoning effort: why we did NOT wire `opencode --thinking`/`--variant`
+
+Tested empirically on 2026-04-19 (Sonnet 4.5 via Bedrock, same
+prompt, three configurations):
+
+| Config | input | output | reasoning tokens | cache write | cost |
+| --- | --- | --- | --- | --- | --- |
+| baseline | 2 | 704 | **0** | 11,101 | $0.052 |
+| `--variant high` | 9 | 1,531 | **0** | 22,248 | $0.106 |
+| `--variant max` | 9 | 2,968 | **0** | 22,248 | $0.128 |
+
+Findings:
+
+1. `opencode --thinking` is documented as **"show thinking blocks"**
+   -- a display toggle. It does not change the request. Our probe
+   confirmed: identical tokens/cost to baseline.
+2. `opencode --variant {high,max}` doubles context (~11k extra
+   cache write, likely a different system preset) and grows output
+   2-4x. `tokens.reasoning` is **always 0** in the event stream on
+   Bedrock Sonnet 4.5 and no `<thinking>` block is emitted. Either
+   opencode's Bedrock stream parser folds reasoning into `output`,
+   or Bedrock silently ignores the `reasoningConfig.budgetTokens`
+   it sends. Empirically the cost roughly doubles with no visible
+   quality telemetry.
+3. Baseline Python is **34/34 x 3** (100% three runs in a row).
+   There are no failing Python exercises to measure uplift against,
+   so `--variant` has no demonstrable quality justification at a
+   2-3x cost multiplier.
+
+Conclusion: a minimal `--variant` passthrough was added to
+`LocalRuntime` / `_run_opencode` / `forge eval polyglot` (opt-in,
+default off, zero behaviour change) so it can be A/B-tested in the
+future if a language emerges where baseline actually fails. The
+full runtime-layer TDD wiring (Docker / Routing / ClaudeCode) was
+**not** done: Python / Go baselines give it nothing to prove.
+Revisit once JS / C++ / Java produce real failures.
 
 ## References
 

platform/forge/eval/cli.py

@@ -132,6 +132,7 @@ def main(argv: Sequence[str] | None = None) -> int:
             skip=args.skip,
             max_attempts=args.max_attempts,
             resume=args.resume,
+            variant=(getattr(args, "variant", "") or None),
         )
 
     parser.error(f"Unknown subcommand: {args.subcommand}")

platform/forge/eval/polyglot.py

@@ -325,6 +325,7 @@ def run_exercise(
     exercise_dir: Path,
     lang: str,
     max_attempts: int,
+    variant: str | None = None,
 ) -> ExerciseResult:
     # Lazy imports: only pull in the studio runtime when actually
     # running an exercise. Keeps the module import cheap for tests
@@ -371,6 +372,7 @@ def run_exercise(
                     cwd=str(work),
                     role="worker",
                     timeout=300,
+                    variant=variant,
                 )
                 llm_total += time.time() - t_llm
 
@@ -474,6 +476,7 @@ def run(
     max_attempts: int = 2,
     resume: str = "",
     results_root: Path = RESULTS_ROOT,
+    variant: str | None = None,
 ) -> int:
     """Run the polyglot benchmark for one language. Returns exit code."""
     exercises = _select_exercises(lang, only, skip, limit)
@@ -483,10 +486,11 @@ def run(
         run_dir.mkdir(parents=True, exist_ok=True)
     else:
         ts = time.strftime("%Y%m%dT%H%M%S")
-        run_dir = results_root / f"{lang}-{ts}"
+        suffix = f"-{variant}" if variant else ""
+        run_dir = results_root / f"{lang}-{ts}{suffix}"
         run_dir.mkdir(parents=True, exist_ok=True)
 
-    print(f"[polyglot] lang={lang} n={len(exercises)} run_dir={run_dir}")
+    print(f"[polyglot] lang={lang} n={len(exercises)} variant={variant or '-'} run_dir={run_dir}")
     print(f"[polyglot] first 5: {[e.name for e in exercises[:5]]}")
 
     results: list[ExerciseResult] = []
@@ -512,7 +516,7 @@ def run(
             continue
 
         print(f"[{i:02d}/{len(exercises)}] {ex.name} -- running...", flush=True)
-        r = run_exercise(ex, lang, max_attempts)
+        r = run_exercise(ex, lang, max_attempts, variant=variant)
         results.append(r)
         (run_dir / f"{ex.name}.json").write_text(json.dumps(r.record, indent=2))
         write_summary(run_dir, results, lang=lang)
@@ -569,6 +573,11 @@ def add_subparser(sub) -> None:
         "--resume", default="",
         help="resume prior run directory (skips exercises whose JSON already exists)",
     )
+    p.add_argument(
+        "--variant", default="",
+        help="opencode --variant (provider-specific reasoning effort, e.g. high, max, minimal). "
+             "Probe-only: currently only forwarded by LocalRuntime.",
+    )
 
 
 def main(argv: list[str] | None = None) -> int:
@@ -586,6 +595,7 @@ def main(argv: list[str] | None = None) -> int:
         skip=args.skip,
         max_attempts=args.max_attempts,
         resume=args.resume,
+        variant=(args.variant or None),
     )
 
 

platform/forge/runtime/local.py

@@ -102,6 +102,7 @@ def run(
         model: str,
         timeout: int,
         role: str | None = None,
+        variant: str | None = None,
     ) -> dict[str, Any]:
         env = os.environ.copy()
 
@@ -123,12 +124,14 @@ def run(
             "--format", "json",
             "--dangerously-skip-permissions",
             "-m", model,
-            prompt,
         ]
+        if variant:
+            cmd.extend(["--variant", variant])
+        cmd.append(prompt)
 
         _log.info(
-            "opencode: running (local) model=%s, timeout=%ds, cwd=%s",
-            model, timeout, cwd,
+            "opencode: running (local) model=%s, variant=%s, timeout=%ds, cwd=%s",
+            model, variant or "-", timeout, cwd,
         )
 
         from forge.engine import process_tracker as _pt

studio/workflows/tdd.py

@@ -102,6 +102,7 @@ def _run_opencode(
     cwd: str | None = None,
     model: str | None = None,
     role: str | None = None,
+    variant: str | None = None,
 ) -> dict:
     """Run opencode via the configured WorkerRuntime.
 
@@ -120,13 +121,16 @@ def _run_opencode(
         except Exception:
             timeout = get_timeout("worker")
 
-    return _get_runtime().run(
+    kwargs: dict = dict(
         prompt=prompt,
         cwd=source_dir,
         model=model,
         timeout=timeout,
         role=role,
     )
+    if variant:
+        kwargs["variant"] = variant
+    return _get_runtime().run(**kwargs)
 
 
 def preflight_node(state: ForgeState) -> dict: