Repository navigation
Conversation
wjhrdy
added a commit
that referenced
this pull request
Aug 20, 2026
Adds this repo's first GitHub Actions workflow. On every PR touching a server.yml, diffs only the changed model(s) against upstream vLLM community serving guidance (vllm-project/recipes, via the hosted recipes.vllm.ai API) and posts findings as a job summary + sticky PR comment. Advisory only -- always exits 0. Deliberately conservative, based on lessons from a manual full-repo review done this session (see the reasoning-parser fix in #106): - tool-call-parser/enable-auto-tool-choice are never flagged as missing/extra -- nm-cicd's OCP harness independently injects these at deploy time regardless of server.yml, and this repo can't see that harness to know if it applies. Only an explicit value conflict is surfaced, and only as a secondary note. - kv-cache-dtype is always a performance note, never correctness -- vLLM's own docs frame the default as the higher-fidelity choice for standard attention backends. - A reasoning-parser conflict (both sides set, disagreeing) is the most prominent finding -- nothing downstream overrides it, and it's exactly the bug class this check exists to catch. - Fuzzy/ambiguous recipe matches are shown separately and never drive the main findings. - Expected-divergence flags (max-model-len, tensor-parallel-size, chat-template, trust-remote-code, ...) are filtered out entirely. - Network failures and "no upstream recipe" are treated as normal, non-error outcomes.
…g-parser A hardcoded wrong tool-call-parser is the same class of bug as a wrong reasoning-parser -- both mis-select a model-specific output-parsing implementation, and neither is compensated for once the value is explicitly (wrongly) set in server.yml. Missing/extra is still suppressed (the OCP harness may legitimately inject it), but an explicit value conflict now renders top-tier instead of buried in the advisory details.
wjhrdy
force-pushed
the
ci/recipes-check
branch
from
August 20, 2026 20:39
3f7c589 to
5e2fc83
Compare
A reasoning-parser/tool-call-parser/tokenizer-mode/config-format/ load-format conflict is now a blocking failure, not just an advisory comment -- this is exactly the class of bug the check exists to catch, and the whole point of adding it was to prevent a repeat of the Qwen3.6-35B-A3B reasoning-parser bug from merging silently again. Everything else (missing/extra, performance notes, fuzzy matches) still never fails the build.
wjhrdy
added a commit
that referenced
this pull request
Aug 20, 2026
wjhrdy
added a commit
that referenced
this pull request
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY:
Adds this repo's first GitHub Actions workflow (
.github/workflows/recipes-check.yml+.github/scripts/check_recipes.py). On every PR touching aserver.yml, it diffs only the changed model(s) against upstream vLLM community serving guidance (vllm-project/recipes, via the hosted recipes.vllm.ai API) and posts findings as a job summary + sticky PR comment, linking each matched model to its recipe page as evidence. A high-confidence correctness conflict (reasoning-parser/tool-call-parser/tokenizer-mode/config-format/load-format) fails the check and blocks merge (--fail-on correctness); everything else (missing/extra, performance notes, fuzzy matches) is advisory-only and never fails the build.This follows a manual, repo-wide review done this session that found a real bug (wrong
reasoning-parseronQwen/Qwen3.6-35B-A3B, fixed in #106/#104) but also produced two false alarms once checked against actual vLLM/harness behavior. The check is deliberately tuned to avoid reproducing that noise on every PR:tool-call-parser: a hardcoded wrong value is the same class of bug as a wrongreasoning-parser(both mis-select an output-parsing implementation), so an explicit conflict renders top-tier and blocks merge, same asreasoning-parser/tokenizer-mode/config-format/load-format. "Missing"/"extra" is still never flagged for it — nm-cicd's OCP harness independently injects it at deploy time, and this repo can't see that harness to know if it applies.enable-auto-tool-choice(a boolean toggle, not a parser selection): same missing/extra suppression, but conflicts stay advisory-only, non-blocking.kv-cache-dtype: always a performance note, never correctness — vLLM's own docs frame the default as the higher-fidelity choice for standard attention backends.max-model-len,tensor-parallel-size,chat-template,trust-remote-code, ...) are filtered out entirely.TEST PLAN:
reasoning-parserdiff prominently), bug fixed (no top finding),common/-only change (graceful skip message), unrelated file change (graceful no-op), unmatched model (graceful "no recipe found"), vendoredargv_to_configunit-sanity check against upstream's documented example — all pass.server.yml, so thepull_requestpath filter doesn't fire automatically here. Confirmed end-to-end on a realpull_requestevent (not justworkflow_dispatch) via throwaway PR [DEMO, do not merge] Evidence: tool-call-parser top-tier + recipe links #109: deliberately reintroducedreasoning-parser: deepseek_r1+tool-call-parser: hermesonQwen/Qwen3.6-35B-A3B, confirmed both rendered top-tier with a working recipe-page link in the posted PR comment (evidence).--fail-on correctnessactually blocks merge: reintroducing thereasoning-parserbug on [DEMO, do not merge] Evidence: tool-call-parser top-tier + recipe links #109 made thecheck-recipesstatus fail (run); reverting to the correct value made it pass (run). Note: this only actually blocks a merge button ifcheck-recipesis added as a required status check under branch protection — that's a separate, repo-wide setting not touched by this PR.