docs(skill-benchmark): record cell-1 run 1 (tiko+skill, specified, 13/16) + findings F2-F5 (#412)

* docs(skill-benchmark): record cell-1 run 1 (tiko+skill, specified, 13/16) + findings F2-F5 (#269)

* docs(skill-benchmark): add no-skill baseline (12/16) + with-vs-without comparison (#269)

---------

Co-authored-by: Tomáš Samek <jerry.samek@gmail.com>

Author: tomas-samek

docs/skill-benchmark/2026-06-06-results.md

@@ -5,6 +5,12 @@
 > changes in the skill" only land once the matrix is full enough to
 > support them — until then this file is a recording surface, not a
 > claim.
+>
+> **2026-06-20:** first real data row landed — `tiko + skill
+> (auto-loaded), specified, run 1` (13/16). Run against `0.4.0-SNAPSHOT`
+> installed to local `.m2` (the only build carrying the post-#409
+> corrected bundled skill — see finding F2). Findings F2–F5 recorded
+> below.
 
 ## Fairness disclosure
 
@@ -241,6 +247,88 @@ before any data was recorded against a confused methodology. The
 finding itself validates the harness more than a single dirty data
 point would.
 
+### Finding F2 — Published archetype lags the corrected skill (F1 redux)
+
+Recorded while preparing the `tiko + skill (auto-loaded)` cell on
+2026-06-20. `v0.3.0` was tagged **2026-06-18 09:02 UTC**; every
+bundled-skill correction merged *after* it — #407 (`Scope.REQUEST` /
+`@PostConstruct` fixes) at ~20:28 UTC the same evening, then #409 (full
+canonical sync) and #410 (the #408 anti-drift gate) on 2026-06-19. So
+`tiko-archetype:0.3.0` on Central — the only published version — bundles
+the **pre-fix, drifted** `tiko-build` skill. A clean "auto-loaded from
+Central" cell today would test the *wrong* skill.
+
+This is structurally identical to F1: the headline auto-loaded cell is
+not truthfully runnable against Central until the next release ships the
+corrected archetype. Workaround used for run 1 (cleaner than F1's manual
+injection): build `0.4.0-SNAPSHOT` and `mvn install` it to the local
+`.m2`, then scaffold from `tiko-archetype:0.4.0-SNAPSHOT`. The installed
+archetype's bundled `SKILL.md` was verified as the corrected one (275
+lines, zero `Scope.REQUEST`) before the agent run, and the #408 gate
+guarantees `bundled == canonical` on `main`. Agent column records the
+local-`.m2` SNAPSHOT origin.
+
+**Action:** the next release (0.4.0) must ship the corrected bundled
+skill so auto-loaded-from-Central becomes the clean headline cell with no
+caveat. The #408 gate makes 0.4.0's archetype clean by construction.
+
+### Finding F3 — Standalone archetype apps run zero ITs (silent false green)
+
+The archetype-scaffolded **standalone** app wires no
+`maven-failsafe-plugin`, so `mvn verify` executes **zero** `*IT` classes
+and still reports `BUILD SUCCESS`. CLAUDE.md and the bundled skill state
+failsafe is "wired in the root pom and inherited by every module" — true
+for tiko's *own* reactor, false for a generated standalone app, which has
+no such parent. The agent-under-test's first declared-ready `mvn verify`
+passed exit 0 while silently running none of the mandated integration
+test; it took an explicit correction turn to add failsafe before the IT
+ran at all. This is the single most consequential finding: it is a silent
+trap on exactly the `Done` criterion ("`mvn verify` green" + an IT that
+asserts a notification) that the prompt — and real users — depend on.
+
+**Action:** the archetype pom should include `maven-failsafe-plugin`
+(and bind `integration-test`/`verify`), or the skill must explicitly call
+out that a standalone app needs failsafe wired before `*IT` runs. Doc
+half is ARCH-13 (docs describe shipped reality — for *generated* apps,
+not just the repo).
+
+### Finding F4 — No documented test seam for FakeKafkaBroker; auto-started transport collides
+
+`io.tiko.kafka.test.FakeKafkaBroker` clearly exists for in-process ITs,
+but nothing documents how to make it the *active* transport, and the
+generated `KafkaTransportBootstrap` works against the test:
+
+- it hardcodes the real-client (2-arg) `KafkaBootstrapSupport`
+  constructor and keeps its `sources()` / `sinks()` private, so the only
+  way to drive the fake broker is to hand-rebuild a parallel
+  `KafkaBootstrapSupport` mirroring the generated descriptors;
+- it auto-starts via the `TransportBootstrap` ServiceLoader SPI inside
+  `Tiko.create()`, pointing at an unreachable localhost broker, so the IT
+  must *also actively suppress it* — the agent stripped
+  `META-INF/services/io.tiko.TransportBootstrap` from `target/classes`
+  via antrun and pointed failsafe's `classesDirectory` at that stripped
+  dir (because the packaged jar still carried the SPI registration).
+
+That is a lot of framework-internal reverse-engineering for what should
+be a one-liner. Note this is the agent solving a *real* gap correctly —
+but the cost (3 of its correction turns trace here, transitively) is the
+finding.
+
+**Action:** a documented "use the fake broker in a test" recipe, a
+`TikoOptions` / property to disable transport autostart under test, or a
+test-support helper that wires `FakeKafkaBroker` as the transport. Likely
+a new issue; relates to the transport/SPI work (#117/#118).
+
+### Finding F5 — Kafka config key-case doc contradiction
+
+The bundled skill states Kafka config keys are camelCase-exact
+(`bootstrapServers`); the shipped `tiko-kafka` `defaults.yaml` uses
+kebab-case (`bootstrap-servers`). It did not bite run 1 (the agent used
+defaults), but it is a live contradiction in agent-facing docs.
+
+**Action:** reconcile the skill against `defaults.yaml` (ARCH-13
+doc-coherence). Small skill/doc fix.
+
 ## Per-run data (raw)
 
 > **One row per run.** Aggregates are computed at the bottom, but the
@@ -251,6 +339,13 @@ point would.
 | cell | prompt | agent | run | tokens in | tokens out | wall (s) | first-build pass? | correction turns | rubric vector | rubric total | transcript | source |
 |---|---|---|---|---|---|---|---|---|---|---|---|---|
 | harness-smoke-test (single-Main archetype output only) | specified | no-agent (smoke test of the harness script itself) | 0 | — | — | 7.3 | yes | 0 | — — — — — — — — — — — — — — — — | — | scaffolded via `mvn archetype:generate` with prompt-verbatim coordinates 2026-06-06; ran `scripts/run-benchmark.ps1` with `MVN=W:\tools\apache-maven\bin\mvn.cmd`; bare archetype output `mvn verify` exit 0. See finding F1 above for why a real with-skill cell-1 was not attempted. | _(generated by `tiko-archetype` 0.1.0 from Maven Central; see finding F1)_ |
+| tiko + skill (auto-loaded, `tiko-archetype:0.4.0-SNAPSHOT` from local `.m2` — see F2) | specified | `claude-opus-4-8[1m]` @ 2026-06-20 (`.ai-skills/tiko-build/SKILL.md` auto-loaded; Claude Code clean-room subagent, no framework-repo access) | 1 | ≈128k combined¹ | ¹ | 1075 | no² | 3 | 0 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1 | 13 | local subagent JSONL³ | `W:\workspace\tiko-bench\notify-app` (local)³ |
+| _comment for run 1_ | | | | | | | | | | | B1=0²: first declared-ready `mvn verify` exited 0 but ran **zero** ITs (standalone archetype has no failsafe — F3); a vacuous green, so `Done` was not met first try. 3 correction turns followed: (1) add failsafe; (2) strip the auto-started Kafka transport SPI; (3) point failsafe at the SPI-stripped classes dir (F4). | |
+| _comment for run 1_ | | | | | | | | | | | T1=0: single Maven module — the four services are co-hosted consumers, not one module/artifact each. O1=1 (judgment): `DataSource` produced via `@Produces` ✓ (the seam O1 guards), but H2 `JdbcDataSource`, not HikariCP as the item literally names → flag for rubric **v2**: "a pooled/**managed** `DataSource` via `@Produces`" (embedded H2 makes a Hikari pool pointless). | |
+| _comment for run 1_ | | | | | | | | | | | H2=0: no test-isolation mechanism — `DB_CLOSE_DELAY=-1` + a fixed `mem:notify` name, no reset / isolated schema; safe only because there is exactly one test. ¹ subagent reported ≈128,409 tokens **combined** (in+out); the Task-subagent harness does not split them. ² `first-build pass? = no` per B1 above. ³ transcript + source are local paths — permalink (gist/branch) still owed per the permanence rule. | |
+| tiko + no skill (scrubbed: `.ai-skills` deleted, skill pointers stripped from `CLAUDE.md`/`AGENTS.md`; CLAUDE.md reference retained) | specified | `claude-opus-4-8[1m]` @ 2026-06-20 (clean-room subagent, no framework-repo access) | 1 | ≈139k combined⁴ | ⁴ | 1146 | no⁵ | 1⁶ | 0 1 1 0 1 1 1 1 0 1 1 1 1 1 0 1 | 12 | local subagent JSONL | `W:\workspace\tiko-bench-noskill\notify-app` (local) |
+| _comment for no-skill run 1_ | | | | | | | | | | | O1=0: no `@Produces` — a hand-rolled `Database` `@Component` wraps `JdbcConnectionPool` and hands repositories raw `Connection`s. Avoids the DataSource-in-repository anti-pattern but skips the `@Produces` DataSource cookbook the with-skill run followed (O1=1 there). **This is the single rubric item separating the two runs**; judgment call (the wrapper is clean, just not the prescribed factory). | |
+| _comment for no-skill run 1_ | | | | | | | | | | | B1=0⁵: first declared-ready `mvn verify` failed **honestly** — this cell's scaffold already had failsafe (scaffolded after the #413 fix), so the IT ran and failed on the F4 dual-transport collision; 1 correction turn (a thread-context-classloader hiding the transport SPI) to green. ⁶ correction-turns are **not** comparable to the with-skill run's 3: that cell was scaffolded *before* #413 and spent a turn adding failsafe; this one got it free. Normalized, both are ~1–2 turns on the same F4 friction. A pre-ready compile iteration to learn the (undocumented-without-skill) `@KafkaSource` contract is recorded as design discovery, not a correction turn. ⁴ ≈138,640 tokens combined; in/out not split by the harness. | |
 
 ### How to score the rubric vector
 
@@ -261,6 +356,60 @@ For non-tiko cells, write `—` for the O-items (the framework doesn't
 have those concepts) so a reader can tell the difference between
 "missing concept" and "agent missed it."
 
+## With-skill vs without-skill — specified, run 1
+
+The load-bearing comparison. Same archetype, same locked prompt, same
+clean-room harness; the **only** intended variable is whether
+`.ai-skills/tiko-build/SKILL.md` was present (the no-skill cell is the
+"scrubbed" variant — CLAUDE.md's framework reference is retained).
+
+| run | B1 | B2 | B3 | T1 | T2 | T3 | T4 | T5 | O1 | O2 | O3 | O4 | O5 | H1 | H2 | H3 | total |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| with-skill | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | **1** | 1 | 1 | 1 | 1 | 1 | 0 | 1 | **13** |
+| no-skill   | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | **0** | 1 | 1 | 1 | 1 | 1 | 0 | 1 | **12** |
+
+**Single rubric delta: O1** — the `@Produces` DataSource cookbook.
+With-skill produced the H2 `DataSource` via `@Produces`; no-skill
+hand-rolled a `Database` component wrapping `JdbcConnectionPool`. Every
+other item is identical: both nailed the event chain (O2/O3), the Kafka
+bridges, upserts (T2), DB-backed lookups (T3), skip-on-missing (T4),
+startup schema (T5), and invented-annotation hygiene (O4); both collapsed
+the four services into one module (T1) and shared the same H2
+test-isolation gap (H2).
+
+**Reading it honestly** (n=1 per cell — noise is large; ≥3 runs needed
+before any aggregate):
+
+- **The skill's marginal effect here was one rubric item.** With
+  CLAUDE.md's reference present in *both* arms, the orchestrator-model
+  essentials were already carried by CLAUDE.md; the SKILL.md's
+  incremental teaching showed up only in the `@Produces` cookbook. A
+  sharper signal would come from the **fresh-without floor** (no CLAUDE.md
+  at all) — a distinct future cell.
+- **Where the skill helped but the rubric didn't capture it:** with-skill
+  got the `@KafkaSource` / `@KafkaSink` contract straight from the skill;
+  no-skill reverse-engineered it from compiler errors + generated sources
+  (a pre-ready compile iteration). Both reached correct usage, so the
+  binary rubric shows no difference — but the skill saved a discovery
+  loop. An effort/time benefit the 0/1 vector doesn't measure.
+- **Where no-skill was arguably better:** it used repository **interfaces**
+  (`ProductRepository` + an `H2…` impl, injected by interface), matching
+  tiko's interfaces-over-impls house style, where with-skill used concrete
+  repos; its first build was an **honest red** (IT ran and failed) where
+  with-skill's was a **false green** (F3: IT silently didn't run); and it
+  solved the F4 transport collision in **one** elegant turn (TCCL
+  SPI-hiding) vs with-skill's two.
+- **F4 dominated both runs regardless of the skill.** Neither doc covers
+  driving `FakeKafkaBroker`, so both agents burned their hardest correction
+  turns reverse-engineering the transport — reinforcing #414 as the
+  highest-value DX fix this benchmark has surfaced.
+
+**Bottom line:** on tiko's home turf with the baseline CLAUDE.md present,
+the `tiko-build` skill moved the needle by one rubric item (the `@Produces`
+cookbook) and saved one discovery loop (the `@KafkaSource` contract) — a
+real but modest margin. The dominant friction (F4) and the structural
+misses (T1 multi-module, H2 isolation) were skill-independent.
+
 ## Aggregates
 
 > Computed once the matrix has ≥ 3 runs per cell. Until then this