Fifth instance of the capability/finding family, found in the logs of run 34141226492 (PR #4048) while planning the structural fix.
packages/export (vitest) -> pass (pass 0, fail 0, total 2)
Two tests collected, zero executed, scored kind: pass and used as a green baseline. verdict() in scripts/lib/revert-oracle.mjs guards baseline.total === 0 but never baseline.passed === 0, so a file behind a describe.skipIf is indistinguishable from a file that ran and passed.
Why that matters rather than being cosmetic. The oracle's whole contract is "the baseline passed, then I reverted production and the tests went red". If the baseline never executed anything, the reverted run cannot go red either, and the branch renders UNOBSERVED — a finding about the author's tests — when the truth is that nothing was measured. That is the same shape as #4050 (unclassified), #4085 (not collected), #4107 (runner absent) and #4104 (wrong owner): a missing link in the measurement chain falling through to a PR-shaped verdict.
It is also the second cause named in #4050's own text, so it has been known in outline and never filed on its own.
Fix, minimal: treat collected > 0 && executed === 0 as a capability gap (all-skipped), not a pass. It must not be silently tolerated and it must not be reported as UNOBSERVED.
Do not fix by failing on any skip. Skipping is legitimate here: fixtures under tests/models are not committed and AGENTS.md requires tests to skip rather than throw when one is absent. The defect is scoring an all-skipped file as measured, not the skipping itself. A partially-skipped file that still executed something is fine.
Belongs to the umbrella issue for this family.
Fifth instance of the capability/finding family, found in the logs of run 34141226492 (PR #4048) while planning the structural fix.
Two tests collected, zero executed, scored
kind: passand used as a green baseline.verdict()inscripts/lib/revert-oracle.mjsguardsbaseline.total === 0but neverbaseline.passed === 0, so a file behind adescribe.skipIfis indistinguishable from a file that ran and passed.Why that matters rather than being cosmetic. The oracle's whole contract is "the baseline passed, then I reverted production and the tests went red". If the baseline never executed anything, the reverted run cannot go red either, and the branch renders
UNOBSERVED— a finding about the author's tests — when the truth is that nothing was measured. That is the same shape as #4050 (unclassified), #4085 (not collected), #4107 (runner absent) and #4104 (wrong owner): a missing link in the measurement chain falling through to a PR-shaped verdict.It is also the second cause named in #4050's own text, so it has been known in outline and never filed on its own.
Fix, minimal: treat
collected > 0 && executed === 0as a capability gap (all-skipped), not a pass. It must not be silently tolerated and it must not be reported asUNOBSERVED.Do not fix by failing on any skip. Skipping is legitimate here: fixtures under
tests/modelsare not committed and AGENTS.md requires tests to skip rather than throw when one is absent. The defect is scoring an all-skipped file as measured, not the skipping itself. A partially-skipped file that still executed something is fine.Belongs to the umbrella issue for this family.