Add agent guidance update skills - #32366
Open
Edward Chen (edgchen1) wants to merge 16 commits into
Open
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/agent_guidance_update_skills
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define deterministic handling for collection output, marker failures, empty windows, candidate precedence, new-skill proposals, and overlapping guidance PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Audit guidance only against committed default-branch state and remove open-PR coordination from both guidance maintenance workflows. Simplify redundant-prose retirement wording and remove unnecessary PR template fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Centralize collection marker parsing and validation, make collection window discovery deterministic and portable, and validate PR descriptions against frozen collection output. Remove the unused audit marker and simplify related skill guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a generic workflow that discovers Python unit tests in every agent skill scripts directory. Clarify collection PR scope and report suspicious future markers when using an automatic cutoff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prevent the agent skill script test workflow from passing when no test directories are found. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Streamline the collection workflow wording and remove the unused audit commit argument so both skills reflect their actual input models. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Candidate filtering can omit valid PRs, and the collection guidance contains conflicting provenance and enforcement rules.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds workflows and tooling for maintaining ONNX Runtime agent guidance.
Changes:
- Adds guidance auditing and review-collection skills.
- Adds collection-marker and PR-discovery utilities with tests.
- Adds CI for skill scripts.
File summaries
| File | Description |
|---|---|
AGENTS.md |
Updates the guidance documentation link. |
docs/Agent_Coding_Guidance.md |
Documents guidance units and lifecycle. |
.github/workflows/agent-skill-script-tests.yml |
Runs skill-script unit tests. |
.github/skills/audit-agent-guidance/SKILL.md |
Defines guidance auditing workflow. |
.github/skills/collect-agent-guidance-from-reviews/SKILL.md |
Defines review-driven collection workflow. |
.../scripts/collection_marker.py |
Implements collection markers. |
.../scripts/list_merged_prs.py |
Discovers collection windows and PRs. |
.../scripts/validate_pr_description.py |
Validates frozen-window markers. |
.../scripts/test_list_merged_prs.py |
Tests PR discovery. |
.../scripts/test_validate_pr_description.py |
Tests marker validation. |
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Exclude only PRs with valid collection markers, clarify provenance and enforcement handling, and normalize unittest imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Mutable PR bodies and delayed search indexing can corrupt the cutoff or permanently omit merged PRs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 2
- Review effort level: Balanced
| continue | ||
|
|
||
| try: | ||
| _, harvested_through = parse_marker(pull_request.get("body") or "") |
| def list_candidates(repository: str, since: datetime, through: datetime) -> list[dict[str, Any]]: | ||
| """Lists merged main-targeting PRs in the half-open collection window.""" | ||
| date_range = f"{format_utc_timestamp(since)}..{format_utc_timestamp(through)}" | ||
| pull_requests = search_pull_requests(repository, f"is:pr is:merged base:main merged:{date_range}") |
| @@ -0,0 +1,198 @@ | |||
| --- | |||
| name: audit-agent-guidance | |||
| description: "Audit ONNX Runtime agent guidance for conflicts, stale claims, ineffective scope, duplication, misplaced detail, and opportunities for mechanical enforcement; open a PR that refines, consolidates, relocates, or retires guidance with one logical change per commit." | |||
Contributor
There was a problem hiding this comment.
Why only one change per commit?
Contributor
There was a problem hiding this comment.
Would be great to see what is produced by running this skill for changes in say the last month as a draft PR.
|
|
||
| ## Safety and Authority | ||
|
|
||
| - Validate guidance claims against the current source code. |
Contributor
There was a problem hiding this comment.
nit: if this applies to instructions and skills what are the 'claims'?
| | [`.github/skills/audit-agent-guidance/SKILL.md`](../.github/skills/audit-agent-guidance/SKILL.md) | Audit existing guidance for revisions or retirement. | | ||
|
|
||
| Keep each piece of guidance in one canonical location. Other layers should point to it rather than restating it. | ||
| Keep each piece of guidance in one canonical location and use the narrowest layer and scope that reliably load for the |
Contributor
There was a problem hiding this comment.
It's a little fuzzy to me what 'guidance' is and what 'one canonical location' equates to if it potentially includes instructions and skills.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces two new skills for maintaining and updating ONNX Runtime agent guidance. The changes formalize and document the workflows for auditing committed guidance and collecting reusable lessons from PR reviews.
New skills and documentation:
.github/skills/audit-agent-guidance/SKILL.mdto define the process for auditing and maintaining agent guidance, including audit dimensions, workflow, and reporting structure..github/skills/collect-agent-guidance-from-reviews/SKILL.mdto describe how to systematically collect, generalize, and propose agent guidance updates from merged PR reviews.Supporting utilities:
.github/skills/collect-agent-guidance-from-reviews/scripts/for use in the skill workflow.Motivation and Context
Goal is to enable more agent guidance update automation. These new skills can be run by agents periodically.