-
Notifications
You must be signed in to change notification settings - Fork 12
feat: implement repository analysis agent with recommendation API #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8ff6641
519ab06
b2e3205
8c6a580
fc607b3
4a0ad3f
a65019a
3ae89e1
719cb76
212bcfd
1db7a60
23b3710
3dd71e1
544b3eb
bdb4227
1c6cf40
23d60e1
4fefa76
63adcf9
9d40d7a
ecbcbd4
b8e8154
37b9e60
1b58651
cf410a0
4bb73b3
b773cee
2f1c0b9
9b4c323
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| rules: | ||
| - description: "Block merges when PRs change filter validation logic without failing on invalid inputs" | ||
| enabled: true | ||
| severity: "high" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/**/vector-query.ts" | ||
| - "packages/core/src/**/graph-rag.ts" | ||
| - "packages/core/src/**/filters/*.ts" | ||
| require_patterns: | ||
| - "throw\\s+new\\s+Error" | ||
| - "raise\\s+ValueError" | ||
| forbidden_patterns: | ||
| - "return\\s+.*filter\\s*$" | ||
| how_to_fix: "Ensure invalid filters raise descriptive errors instead of silently returning unfiltered results." | ||
|
|
||
| - description: "Require regression tests when modifying tool schema validation or client tool execution" | ||
| enabled: true | ||
| severity: "medium" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| source_patterns: | ||
| - "packages/core/src/**/tool*.ts" | ||
| - "packages/core/src/agent/**" | ||
| - "packages/client/**" | ||
| test_patterns: | ||
| - "packages/core/tests/**" | ||
| - "tests/**" | ||
| min_test_files: 1 | ||
| rationale: "Tool invocation changes have previously caused regressions in clientTools streaming." | ||
|
|
||
| - description: "Ensure every agent exposes a user-facing description for UI profiles" | ||
| enabled: true | ||
| severity: "low" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/agent/**" | ||
| required_text: | ||
| - "description" | ||
| message: "Add or update the agent description so downstream UIs can render capabilities." | ||
|
|
||
| - description: "Block merges when URL or asset handling changes bypass provider capability checks" | ||
| enabled: true | ||
| severity: "high" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/agent/message-list/**" | ||
| - "packages/core/src/llm/**" | ||
| require_patterns: | ||
| - "isUrlSupportedByModel" | ||
| forbidden_patterns: | ||
| - "downloadAssetsFromMessages\\(messages\\)" | ||
| how_to_fix: "Preserve remote URLs for providers that support them natively; only download assets for unsupported providers." |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -111,6 +111,52 @@ parameters: | |
| excluded_branches: ["feature/*", "hotfix/*"] | ||
| ``` | ||
|
|
||
| ### Diff-Aware Validators | ||
|
|
||
| Watchflow can now reason about pull-request diffs directly. The following parameter groups plug into diff-aware validators: | ||
|
|
||
| #### `diff_pattern` | ||
|
|
||
| Use this to require or forbid specific regex patterns inside matched files. | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/**/vector-query.ts" | ||
| require_patterns: | ||
| - "throw\\s+new\\s+Error" | ||
| forbidden_patterns: | ||
| - "console\\.log" | ||
| ``` | ||
|
|
||
| #### `related_tests` | ||
|
|
||
| Ensure core source changes include matching test updates. | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| source_patterns: | ||
| - "packages/core/src/**" | ||
| test_patterns: | ||
| - "tests/**" | ||
| - "packages/core/tests/**" | ||
| min_test_files: 1 | ||
| ``` | ||
|
|
||
| #### `required_field_in_diff` | ||
|
|
||
| Verify that additions to certain files include a text fragment (for example, enforcing `description` on new agents). | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/agent/**" | ||
| required_text: | ||
| - "description" | ||
| ``` | ||
|
|
||
| These validators activate automatically when the parameters above are present, so you do not need to declare an `actions` block or manual mapping. | ||
|
|
||
| ## Severity Levels | ||
|
|
||
| ### Severity Configuration | ||
|
|
@@ -219,6 +265,39 @@ rules: | |
| required_teams: ["senior-engineers"] | ||
| ``` | ||
|
|
||
| ## Diff-Aware Validators | ||
|
|
||
| Watchflow supports advanced validators that inspect actual PR diffs to enforce code-level patterns: | ||
|
|
||
| ### diff_pattern | ||
| Enforce regex requirements or prohibitions within file patches. | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| file_patterns: ["packages/core/src/**/vector-query.ts"] | ||
| require_patterns: ["throw\\s+new\\s+Error"] | ||
| forbid_patterns: ["silent.*skip"] | ||
| ``` | ||
|
|
||
| ### related_tests | ||
| Require test file updates when core code changes. | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| file_patterns: ["packages/core/src/**"] | ||
| require_test_updates: true | ||
| min_test_files: 1 | ||
| ``` | ||
|
|
||
| ### required_field_in_diff | ||
| Ensure new additions include required fields (e.g., agent descriptions). | ||
|
|
||
| ```yaml | ||
| parameters: | ||
| file_patterns: ["packages/core/src/agent/**"] | ||
| required_text: "description" | ||
| ``` | ||
|
|
||
|
Comment on lines
+268
to
+300
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This section on "Diff-Aware Validators" appears to be a duplicate of the one added earlier in this file (lines 114-159). The content is also slightly inconsistent between the two sections (e.g., |
||
| ## Best Practices | ||
|
|
||
| ### Rule Design | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| # Mastra Repository Analysis | ||
|
|
||
| Mastra (`mastra-ai/mastra`) is a TypeScript-first agent framework for building production-grade AI assistants. The project has roughly **280 contributors**, **134 open pull requests**, and active CI coverage via GitHub Actions. This document captures the agreed-upon analysis from November 2025 so we can align on rule proposals before shipping automation. | ||
|
|
||
| ## Repository Snapshot | ||
|
|
||
| - **Focus**: AI agents with tooling, memory, workflows, and multi-step orchestration | ||
| - **Primary language**: TypeScript with pnpm-based monorepo | ||
| - **Governance signals**: Detailed `CONTRIBUTING.md`, CODEOWNERS, changeset automation, active doc set | ||
| - **Pain points**: Complex LLM/provider integrations, repeated validation gaps, and regression risk in shared tooling layers | ||
|
|
||
| ## Pull Request Sample (Nov 2025) | ||
|
|
||
| | PR | Title | Outcome | Notes | | ||
| | --- | --- | --- | --- | | ||
| | [#10180](https://github.com/mastra-ai/mastra/pull/10180) | feat: add custom model gateway support with automatic type generation | ✅ merged | Large feature: gateway registry, TS type generation, doc updates | | ||
| | [#10269](https://github.com/mastra-ai/mastra/pull/10269) | AI SDK tripwire data chunks | ✅ merged | Fixes & changeset for SDK data chunking bug | | ||
| | [#10141](https://github.com/mastra-ai/mastra/pull/10141) | fix: throw on invalid filter instead of silently skipping filtering | ✅ merged | Addressed regression where invalid filters returned unfiltered data | | ||
| | [#10300](https://github.com/mastra-ai/mastra/pull/10300) | Add description to type | ✅ merged | Unblocked Agent profile UI by exposing description metadata | | ||
| | [#9880](https://github.com/mastra-ai/mastra/pull/9880) | Fix clientjs clientTools execution | ✅ merged | Fixed client-side tool streaming regressions | | ||
| | [#9941](https://github.com/mastra-ai/mastra/pull/9941) | fix(core): input tool validation with no schema | ✅ merged | Restored validation for schema-less tool inputs | | ||
|
|
||
| ## Pattern Summary | ||
|
|
||
| - **Validation & safety gaps (≈40%)** – invalid filters or schema-less tools silently bypassed safeguards. | ||
| - **Tooling & integration regressions (≈33%)** – clientTools streaming, AI SDK data chunking, URL handling. | ||
| - **Experience polish gaps (≈17%)** – missing agent descriptions prevented UI consistency. | ||
| - **High merge velocity** – most fixes merged quickly; reinforces need for automated guardrails so regressions are caught before release. | ||
|
|
||
| ## Recommended Watchflow Rules | ||
|
|
||
| Rules intentionally avoid the optional `actions:` block so they remain compatible with the current loader. Enforcement intent is described in each `description` and reflected in `severity`. | ||
|
|
||
| ```yaml | ||
| rules: | ||
| - description: "Block merges when PRs change filter validation logic without failing on invalid inputs" | ||
| enabled: true | ||
| severity: "high" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/**/vector-query.ts" | ||
| - "packages/core/src/**/graph-rag.ts" | ||
| - "packages/core/src/**/filters/*.ts" | ||
| require_patterns: | ||
| - "throw\\s+new\\s+Error" | ||
| - "raise\\s+ValueError" | ||
| forbidden_patterns: | ||
| - "return\\s+.*filter\\s*$" | ||
| how_to_fix: "Ensure invalid filters raise descriptive errors instead of silently returning unfiltered results." | ||
|
|
||
| - description: "Require regression tests when modifying tool schema validation or client tool execution" | ||
| enabled: true | ||
| severity: "medium" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| source_patterns: | ||
| - "packages/core/src/**/tool*.ts" | ||
| - "packages/core/src/agent/**" | ||
| - "packages/client/**" | ||
| test_patterns: | ||
| - "packages/core/tests/**" | ||
| - "tests/**" | ||
| min_test_files: 1 | ||
| rationale: "Tool invocation changes have previously caused regressions in clientTools streaming." | ||
|
|
||
| - description: "Ensure every agent exposes a user-facing description for UI profiles" | ||
| enabled: true | ||
| severity: "low" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/agent/**" | ||
| required_text: | ||
| - "description" | ||
| message: "Add or update the agent description so downstream UIs can render capabilities." | ||
|
|
||
| - description: "Block merges when URL or asset handling changes bypass provider capability checks" | ||
| enabled: true | ||
| severity: "high" | ||
| event_types: ["pull_request"] | ||
| parameters: | ||
| file_patterns: | ||
| - "packages/core/src/agent/message-list/**" | ||
| - "packages/core/src/llm/**" | ||
| require_patterns: | ||
| - "isUrlSupportedByModel" | ||
| forbidden_patterns: | ||
| - "downloadAssetsFromMessages\\(messages\\)" | ||
| how_to_fix: "Preserve remote URLs for providers that support them natively; only download assets for unsupported providers." | ||
| ``` | ||
|
|
||
| These concrete rules rely on the diff-aware validators recently added to Watchflow: | ||
|
|
||
| - `diff_pattern` ensures critical patches keep throwing exceptions or performing capability checks. | ||
| - `related_tests` requires PRs touching core modules to include matching test updates. | ||
| - `required_field_in_diff` verifies additions to agent definitions include a `description` so downstream UIs stay in sync. | ||
|
|
||
| Because the PR processor now passes normalized diffs into the engine, these validators operate deterministically without LLM fallbacks. | ||
|
|
||
| ## PR Template Snippet | ||
|
|
||
| ```markdown | ||
| ## Repository Analysis Complete | ||
|
|
||
| We've analyzed your repository and identified key quality patterns based on recent PR history. | ||
|
|
||
| ### Key Findings | ||
| - 40% of recent fixes patched validation or data-safety gaps (filters, schema-less tools). | ||
| - 33% addressed tool/LLM integration regressions (clientTools, AI SDK, URL handling). | ||
| - Tests/documentation often lag behind critical fixes, creating follow-up churn. | ||
|
|
||
| ### Recommended Rules | ||
| - Block filter-validation changes that stop throwing on invalid inputs. | ||
| - Require regression tests when modifying tool schemas or clientTools execution. | ||
| - Enforce agent descriptions so UI consumers can present profiles. | ||
| - Block URL/asset handling changes that skip provider capability checks. | ||
|
|
||
| ### Installation | ||
| 1. Install the Watchflow GitHub App and grant access to `mastra-ai/mastra`. | ||
| 2. Add `.watchflow/rules.yaml` with the rules above (see snippet). | ||
| 3. Watchflow will start reporting violations through status checks immediately. | ||
|
|
||
| Questions? Reach out to the Watchflow team. | ||
| ``` | ||
|
|
||
| ## Validation Plan | ||
|
|
||
| 1. Keep the rule definitions in `docs/samples/mastra-watchflow-rules.yaml`. | ||
| 2. Run `pytest tests/unit/test_mastra_rules_sample.py` to ensure every rule loads via `Rule.model_validate`. | ||
| 3. (Optional) Use the repository analysis agent once PR-diff ingestion ships to simulate Mastra commits before opening an automated PR with these rules. | ||
|
|
||
| This keeps the deliverable lightweight, fully tested, and ready for the PR template automation flow discussed with Dimitris. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There appears to be duplicated and inconsistent documentation for
Diff-Aware Validators. This section is added here, but a similar, slightly different version is also added earlier in the file (starting around line 114). The two versions use conflicting parameter names and formats, which will be confusing for users. For example:forbidden_patterns, while this one usesforbid_patterns.related_testsexample here usesrequire_test_updates: true, while the other usessource_patterns,test_patterns, andmin_test_files.required_textparameter is a string here, but a list in the other example.Please consolidate these into a single, consistent section to avoid confusion.