Skip to content

fix(ci): let check-issue-queue accept honest Refs #N partial work - #4161

Merged
louistrue merged 4 commits into
mainfrom
fix-4147-issue-queue-refs-partial-work
Sep 8, 2026
Merged

fix(ci): let check-issue-queue accept honest Refs #N partial work#4161
louistrue merged 4 commits into
mainfrom
fix-4147-issue-queue-refs-partial-work

Conversation

@BIMvoice

@BIMvoice BIMvoice commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • closingIssuesReferences is populated only by a closing keyword, so a PR that is deliberately a slice of a larger issue could not honestly link it without also claiming (and auto-closing) work it did not finish. Five PRs hit exactly this in one evening, all correctly writing Refs #N instead of lying with Closes #N, and all needing a manual unqueued waiver whose meaning ("did not need to wait for the queue") is the opposite of the truth.
  • Adds a second, narrower accepted shape in scripts/lib/issue-refs.mjs, consulted only when closingIssuesReferences closes nothing: the PR body may name an issue with a non-closing keyword (Refs, References, Part of, Towards, case-insensitive). That issue must independently be OPEN and carry ready from an authorised applier — the exact same check a closing link already goes through — so:
    • a PR referencing an issue that is not ready still fails (NO_LINKED_ISSUE)
    • a PR referencing a ready issue that is CLOSED still fails (a closed issue is not a queue entry)
    • a self-applied ready label on the referenced issue still fails (SELF_APPLIED_LABEL, same authority check)
    • a PR referencing no issue at all still fails, and unqueued still works unchanged
    • Closes #N on a ready issue is unaffected — it passes before this new code path is ever reached
  • closingIssuesReferences stays the primary, preferred signal; nothing here weakens it or adds a self-service bypass.
  • A match now requires intent, not just the word. The body regex originally matched Refs #N anywhere in the text, including inside a fenced code block, an inline code span, or a blockquote, and as an ordinary verb mid-sentence ("this function refs Compare codebase with IFC libraries #12 in a loop") — none of which name a queue entry on purpose. Since this module can only widen a fail into a pass, an intentless match was a live bypass: any PR could pass the gate by quoting a ready issue number in a code sample or a quoted reply. extractRefIssueNumbers now strips fenced code blocks, inline code spans, and blockquote lines before scanning, and requires the keyword to start its line (optionally after a list marker: -, *, +, 1., 1)) — which is how a real Refs #N is actually written, and which also rejects the mid-sentence-verb case without a separate rule for it. Refs #N, References #N, Part of #N, Towards #N, and refs: #N all keep working in their ordinary positions.

Implementation notes

  • New verdict: PARTIAL_WORK (pass), printed with the same detail as READY_ISSUE.
  • Live mode does a second, cheap gh api graphql round trip for the referenced issue(s), but only when the first read already closed nothing and the body names a candidate — an ordinary PR that closes its issue pays no extra cost and cannot be failed by this path.
  • All new logic lives in scripts/lib/issue-refs.mjs (kept check-issue-queue.mjs at its existing 957-line module-size budget without raising it — the file's own header carried a genuine pre-existing formatting defect, a NO_LABELS/NO_TIMELINE/NO_CLOSING_ISSUES paragraph spliced mid-sentence into an unrelated story, which is also fixed here as a byproduct of reflowing the prose to make room).
  • Related issue Add a gate for a PR whose body admits partial coverage under a bare Closes/Fixes/Resolves keyword #4154 proposes the opposite-direction check (a PR claiming Closes while admitting partial coverage); not implemented here.
  • Not a markdown parser: the pre-pass recognises fenced/inline/blockquote by their line-level markers, not full CommonMark (a markdown link's display text, for example, is left alone) — scoped enough to close the confirmed hole without a new dependency, matching this repo's dependency-light gates.

Test plan

  • node --test scripts/check-issue-queue.test.mjs scripts/lib/issue-refs.test.mjs — 95/95 pass, including end-to-end cases and unit tests covering the regex, the query builder, the fail-closed live-fetch wrapper, the verdict logic, and the intent-scoping pre-pass.
  • Mutation: reverted the change — 3 top-level test() cases in check-issue-queue.test.mjs go RED (one of them loops over 4 keyword variants), everything else stays green.
  • Mutation: dropped the state === 'OPEN' filter — the CLOSED-issue test (both the integration test and its lib/issue-refs.test.mjs unit twin) goes RED; reverted.
  • Mutation: disabled the fenced/inline-span/blockquote strip — the fenced-code-block test goes RED (the inline-span and blockquote cases are additionally covered by the line-start anchor itself, so they stay green under this specific mutation; both mechanisms independently reject those two shapes); reverted.
  • Mutation: removed list-marker support from the keyword regex — the "after a list marker" honest-form test goes RED; reverted.
  • node scripts/check-module-size.mjs — OK, 0 new over budget.
  • node scripts/check-test-wiring.mjs — OK.

Closes #4147

)

closingIssuesReferences is populated only by a closing keyword, so a PR that
is deliberately a slice of a larger issue could not link it without also
claiming (and auto-closing) work it did not finish. Five PRs hit this in one
evening, all correctly writing `Refs #N` instead of lying with `Closes #N`,
and all needing a manual `unqueued` waiver whose meaning is the opposite of
the truth.

Add a second, narrower accepted shape in scripts/lib/issue-refs.mjs: when
closingIssuesReferences closes nothing, the PR body may still name an issue
with a non-closing keyword (Refs/References/Part of/Towards). That issue must
independently be OPEN and carry `ready` from an authorised applier -- the
same check a closing link already goes through -- so a PR referencing a
not-ready or closed issue still fails exactly as before. The self-applied-
label rule and the escape-label mechanism are unchanged.

Closes #4147
@BIMvoice
BIMvoice requested a review from louistrue as a code owner September 8, 2026 09:56
@coderabbitai

coderabbitai Bot commented Sep 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 5 billable files and costs up to $1.25.

Or wait 59 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 8bbd6770-359f-46d3-bee5-8633d672739b

📥 Commits

Reviewing files that changed from the base of the PR and between 1791e64 and c763fba.

📒 Files selected for processing (5)
  • .github/workflows/issue-queue.yml
  • scripts/check-issue-queue.mjs
  • scripts/check-issue-queue.test.mjs
  • scripts/lib/issue-refs.mjs
  • scripts/lib/issue-refs.test.mjs

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Claude review - no findings for 5a7694d9e

Reviewed this diff and found nothing to flag.

@github-actions github-actions Bot added the llm-reviewed A review was verified as posted for this PR's head. label Sep 8, 2026
extractRefIssueNumbers matched Refs #N anywhere in a PR body, including
inside a fenced code block, an inline code span, a blockquote, or as an
ordinary verb mid-sentence ("this function refs #12 in a loop"). Since
this module can only widen a PARTIAL_WORK fail into a pass, a match with
no authorial intent behind it is a live bypass: an unrelated PR could
pass the issue-queue gate by quoting a ready issue number in a code
sample or a quoted reply.

extractRefIssueNumbers now strips fenced code blocks, inline code spans,
and blockquote lines before scanning, and requires the keyword to start
its line (optionally after a list marker: -, *, +, 1., 1)). Refs #N,
References #N, Part of #N, Towards #N, and refs: #N all keep working in
their ordinary positions; two pre-existing unit test fixtures that ran
multiple keywords together on one line were split one-per-line to match
how these are actually written.
@vercel

vercel Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
ifc-lite-dev Ignored Ignored Sep 8, 2026 11:41am UTC
ifc-lite-viewer-embed Ignored Ignored Sep 8, 2026 11:41am UTC

@github-actions github-actions Bot removed the llm-reviewed A review was verified as posted for this PR's head. label Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Claude review - no findings for 910d38fb5

Reviewed this diff and found nothing to flag.

@github-actions github-actions Bot added the llm-reviewed A review was verified as posted for this PR's head. label Sep 8, 2026
…SSUE fail

910d38f's tightening of extractRefIssueNumbers is correct and stays as-is:
a Refs #N wrapped in backticks mid-sentence, or quoted inside a fence, no
longer grants a PARTIAL_WORK pass. But that tightening left the failure
message silent about why a plausible-looking reference didn't count -- the
exact phrasing "...`Refs #N`, and requesting the `unqueued` label..." showed
up in two PR bodies today, and would now fail with no clue what to fix.

Adds findNearMissRefIssueNumbers (lib/issue-refs.mjs): the permissive
keyword+#N scan the strict matcher replaced, run only outside fenced code
(the confirmed exploit shape stays silent, on purpose) and excluding numbers
the strict matcher already accepted. Its only consumer is nearMissRefsNote,
which appends a REMEDY line to the NO_LINKED_ISSUE failure output naming the
issue and the accepted line-start form. Neither function is read by
evaluate()'s ok/verdict decision -- verified by mutation: disabling the hint
call, and removing the fence exclusion, each turn exactly one targeted test
red and nothing else.

Kept scripts/check-issue-queue.mjs at its allowlisted 957 lines and
scripts/lib/issue-refs.mjs at the 400-line module-size limit by combining
the two new call sites onto existing lines rather than adding new ones.
@github-actions github-actions Bot removed the llm-reviewed A review was verified as posted for this PR's head. label Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Claude review - no findings for 4ce585df2

Reviewed this diff and found nothing to flag.

@github-actions github-actions Bot added the llm-reviewed A review was verified as posted for this PR's head. label Sep 8, 2026
@BIMvoice

BIMvoice commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Ran check-issue-queue.mjs from this PR's head (4ce585df2, plus one later commit) live against every open PR whose Issue queue check is currently failing (4/22 red: #4159, #4160, #4151, #4152). Verdicts below are actual gate output, not prediction.

2 of 4 clear once #4161 merges:

PR Refs Verdict here Why
#4159 #4153 ✅ PARTIAL_WORK line-start Refs #4153; #4153 is OPEN, ready applied by louistrue 2026-09-08T09:44:58Z
#4160 #4153 ✅ PARTIAL_WORK same — first line of the body is Refs #4153.

2 stay red — need a manual unqueued, not this fix:

PR Refs Verdict here Why not
#4151 #3612 ❌ NO_LINKED_ISSUE body has `Refs #3612` mid-sentence, backtick-wrapped ("...the actual trigger. Refs #3612, and requesting..."), not at line start — the tightened matcher doesn't count it even though #3612 is OPEN with ready (louistrue, 2026-09-02T09:16:43Z)
#4152 #3357, #4018 ❌ NO_LINKED_ISSUE same shape defect (mid-sentence, "...classification gaps). Refs #3357 #4018...") and both #3357 and #4018 are chore: version packages PRs, not issues (#3357 merged, #4018 open) — wouldn't qualify even with a fixed reference form

Also checked #4167, which the same Refs-#4153 shape would clear (verified: PARTIAL_WORK) — but its Issue queue check is currently queued, not fail, on GitHub, so it isn't part of the 4-PR red count above.

#4151 and #4152 still need unqueued from you regardless of this PR merging.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Claude review - no findings for c763fba40

Reviewed this diff and found nothing to flag.

@github-actions github-actions Bot added llm-reviewed A review was verified as posted for this PR's head. and removed llm-reviewed A review was verified as posted for this PR's head. labels Sep 8, 2026
@louistrue
louistrue merged commit fd63ca7 into main Sep 8, 2026
32 checks passed
BIMvoice added a commit that referenced this pull request Sep 8, 2026
BIMvoice added a commit that referenced this pull request Sep 8, 2026
BIMvoice added a commit that referenced this pull request Sep 8, 2026
BIMvoice added a commit that referenced this pull request Sep 8, 2026
BIMvoice added a commit that referenced this pull request Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

llm-reviewed A review was verified as posted for this PR's head.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check-issue-queue forces partial work to over-claim: a stacked or sliced PR cannot honestly say Closes

2 participants