[DNI] Add issue analysis toolset for quality gap identification #9033
+2,639
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds comprehensive GitHub issue and PR analysis tools to identify quality gaps in the Slang codebase: Tools: - fetch_github_issues.py: Fetch issues/PRs from GitHub API with enrichment - analyze_issues.py: General quality analysis (3,535 issues, 5,392 PRs) - analyze_critical_issues.py: Deep dive on crashes and ICEs - analyze_bugfix_files.py: File-level hotspot analysis Documentation: - README.md: Quick start guide and usage - SUMMARY.md: Executive summary with top 3 quality gaps - GENERAL_FINDINGS.md: Complete analysis with backend comparison - CRITICAL_FINDINGS.md: Critical issues analysis (1,049 issues) - ANALYSIS_PLAN.md: Analysis methodology - DATA_FORMAT.md: Data structure reference Key Findings: - slang-lower-to-ir.cpp: shader-slang#1 crash source (83 critical fixes) - SPIRV: 827 issues, highest volume - HLSL: 292 days avg fix time - Test coverage: 43.7% of PRs lack tests Data included for immediate analysis (61 MB): - Pre-fetched issues and PRs (Nov 2025) - Processed CSV files for Excel/custom analysis
Collaborator
|
Do we have a github issue for this? |
Enhances bug detection, component categorization, and user experience: - Bug detection: Include infrastructure bugs, improve pattern matching * Now detects 1,913 bug fix PRs (35.5%) with better accuracy * Recognize crashes, ICEs, validation errors, correctness issues - Component categorization: Expand to 30+ categories, reduce "other" by 92.7% * Add build-system, gfx-rhi, reflection, language-server, etc. * Show source file breakdown by component - Fetch workflow: Simplify CLI, remove timeline API, add instant issue refs * Single --incremental flag for fast updates * Always fetch PR files, extract issue refs without API calls - Documentation: Rewrite README - Repository: Exclude 61MB data/ directory, add .gitignore
Contributor
Author
Related issue is #9039. Goal is to extract data from issues and PRs (and test coverage data) to identify common bug types and components where to concentrate on improving testing quality. |
Track bug fix frequency (fixes per 1000 LOC) to identify files that require frequent bug fixes, helping distinguish truly problematic files from simply large files. - Add get_file_loc() to calculate lines of code - Filter reports to source/ directory only - Use correct terminology: "bug fix frequency" (counts PRs, not bugs) - Remove trailing whitespace
Extract common code into analyze_common.py for better maintainability: - Shared functions: get_file_loc, get_component_from_file, data loading - Unified component categorization across all analysis scripts - Reduces code duplication and ensures consistency Update analysis methodology to be resource-agnostic: - Add ANALYZE_RESULTS_PLAN.md - Replace timeline-based recommendations (0-3mo, 3-6mo, 6-12mo) with short-term and long-term priorities ordered by impact - No assumptions about available resources or timelines Enhance executive summary with comprehensive metrics: - Expand from top 3 to top 5 risk areas - Add new section: top 5 areas in open issues and PRs - Provides current status alongside historical analysis Update README documentation to reflect latest tool features and simplified workflow.
Contributor
Author
|
Archiving for later use. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds comprehensive GitHub issue and PR analysis tools to identify quality gaps in the Slang codebase:
Tools:
Documentation:
Related issue #9039