Skip to content

[quality] CI gate scripts generate_skill_index.py and update_coverage_snapshot.py had zero unit coverage #767

Description

@kubestellar-hive

Finding

scripts/ contains 10 Python entry points. Eight have a matching tests/unit/test_*.py. Two did not:

script lines CI gate
scripts/generate_skill_index.py 165 .github/workflows/docs-validate.yml runs it with --check
scripts/update_coverage_snapshot.py 186 .github/workflows/pr-validate.yml runs it with --check

Both are merge gates. Their failure modes are quiet:

  • generate_skill_index.collect() enforces nine front-matter validations (required fields, category/status enums, entry_point self-match, id == name). A regression that drops a check lets a malformed skill into index.json with a green build.
  • generate_skill_index.main(--check) compares index.md against render_md(skills, current['generated_at']) — deliberately reusing the committed date rather than date.today(). Nothing pinned that behaviour; switching it to today's date would make the docs gate fail every day after the catalog is committed (the same class of failure tracked in dakota-iso#137).
  • update_coverage_snapshot.parse_scenarios() implements non-obvious Gherkin tag semantics: feature tags inherit, scenario tags reset after each scenario, and Rule:/Background:/Examples: discard pending tags. classify() layers a precedence order on top (quarantine > hardware_blocked/future/pending > active). A silent miscount here corrupts the suite-map coverage table that this repo uses as its coverage source of truth — and the whole point of that script is to stop humans hand-editing those numbers.
  • update_file() returns three distinct exit codes (0 ok, 1 stale, 2 markers missing) that CI branches on; none were pinned.

Recommendation

Add unit tests for both scripts. Done in PR #766 (hold-gated):

  • tests/unit/test_generate_skill_index.py — 36 tests
  • tests/unit/test_update_coverage_snapshot.py — 49 tests

85 tests, all passing. No production code changed.

Priority

  • Impact: high (both scripts are required merge gates with silent failure modes)
  • Effort: low (both are pure functions plus a --repo-root seam; no mocking of external services needed)

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: f368ee5

— hive: agent=quality backend=copilot model=claude-opus-5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions