Replace migrating-dbt-core-to-fusion with comprehensive triage skill#86
Merged
Replace migrating-dbt-core-to-fusion with comprehensive triage skill#86
Conversation
Replace the existing migration workflow skill with a triage-focused skill that classifies errors into 4 actionable categories: auto-fixable, guided fixes, needs input, and blocked on Fusion updates. This helps users understand what they can fix vs what requires engine updates. Changes: - Replace SKILL.md with triage classification framework - Add error-patterns-reference.md and classification-categories.md - Remove old reference files (custom_configuration, dynamic_sql, misspelled_config_keys) - Add two eval scenarios (basic classification, blocked classification) - Bump dbt-migration plugin version to 1.1.0 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port remaining eval-harness tasks to the skill-eval framework:
- Cat A: static analysis in analyses (auto-fixable)
- Cat B: dict.meta_get error, empty SELECT, unexpected config keys,
unused schema entries, YAML syntax error
- Cat C: hardcoded FQN permission error (needs user input)
All 9 triage scenarios (18 total runs) pass successfully.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The truncate() filter issue has a valid workaround ([:64] slicing) that isn't technical debt. Update grading criteria to accept either a blocked classification or a clean workaround suggestion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add eval scenario using the seed materialization dispatch bug (dbt-fusion#1345) — a real Fusion engine bug with NO user-side workaround. This tests that the agent correctly identifies blocked issues and does NOT attempt fixes. Also expand error-patterns-reference.md with missing patterns from the original skill spec: - Package lockfile issues (dbt1001) - Source without tables (dbt0102) - Misspelled config keys after autofix - PIVOT IN ANY static analysis - Package incompatibility guidance - Connection errors (dbt1308) - Engine crash signals (panic!, RUST_BACKTRACE) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
venkaa28
commented
Mar 3, 2026
evals/scenarios/fusion-migration-triage-basic/context/dbt_compile_output.txt
Outdated
Show resolved
Hide resolved
…earch behavior
Major fixes:
- Fix skill path in all skill-sets.yaml (was skills/migrating-dbt-core-to-fusion,
should be skills/dbt-migration/skills/migrating-dbt-core-to-fusion)
- Replace all fake compile output with real `fs compile` output
- Update config_api scenario to test config.require('key_in_meta') pattern
which actually errors in current Fusion (config.require('meta').key works now)
- Use real dbt.log for static analysis scenario
- Use real Fusion compile output for hardcoded FQN (dbt0214)
- Remove unreproducible seed dispatch bug scenario
- Reframe blocked scenario to test GitHub issue search behavior
- Add WebFetch and gh CLI to blocked scenario allowed_tools
- Strengthen Category D language in SKILL.md and classification-categories.md
Verified: blocked eval now successfully searches GitHub, finds issue #1318,
references the fix PR, and provides actionable context to the user.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix yaml-syntax compile output to include actual error line. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…contexts These were accidentally committed from local fs compile runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eliasdefaria
reviewed
Mar 4, 2026
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
skills/dbt-migration/skills/migrating-dbt-core-to-fusion/SKILL.md
Outdated
Show resolved
Hide resolved
Per skill authoring best practices, reference files over 100 lines should include a TOC so Claude can see the full scope when previewing.
Align with dbt-migration plugin version bump for the triage skill rewrite.
b-per
approved these changes
Mar 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
migrating-dbt-core-to-fusionskill with a triage-focused skill that classifies migration errors into 4 actionable categories: auto-fixable, guided fixes, needs input, and blocked on Fusion updateserror-patterns-reference.md,classification-categories.md) and remove the old ones (custom_configuration.md,dynamic_sql.md,misspelled_config_keys.md)fusion-migration-triage-basic,fusion-migration-triage-blocked) using the repo's skill-eval framework with pre-captured dbt compile output so no livedbt-fusionbinary is requireddbt-migrationplugin version from1.0.1to1.1.0Why this change
The existing skill is a step-by-step migration workflow that attempts to fix everything. The new triage skill is more honest about what's fixable vs blocked:
Validation
python scripts/validate_repo.pypasses all checks related to our changes (the only failure is a pre-existingreview_run_results.mdlink issue inusing-dbt-for-analytics-engineering)name,description,allowed-tools,compatibility,metadata)dbt_compile_output.txtso they don't require a live Fusion binaryTest plan
python scripts/validate_repo.pypasses (only pre-existing failure in unrelated skill)fusion-migration-triage-basictests Category B classification,fusion-migration-triage-blockedtests Category D classification🤖 Generated with Claude Code