[UR][CI] Add unified test result reporting for UR tests by kekaczma · Pull Request #22909 · intel/llvm

kekaczma · 2026-08-10T14:37:25Z

Introduce common tooling for collecting and reporting results from UR conformance and adapter-specific test executions.

The implementation:

provides a common execution and reporting path for UR test suites,
collects detailed test statuses and execution statistics,
parses LIT output and JUnit XML to obtain test-level information,
generates human-readable console output and GitHub Step Summary,
preserves full execution logs and XML results as CI artifacts,
separates test execution, result parsing, and summary generation to support future consumers of test result data.

This is the first implementation step towards a broader test management and test health monitoring system. Database integration, historical analysis, and automated PR triage are outside the scope of this change.

Introduce common tooling for collecting and reporting results from UR conformance and adapter-specific test executions. The implementation: - provides a common execution and reporting path for UR test suites, - collects detailed test statuses and execution statistics, - parses LIT output and JUnit XML to obtain test-level information, - generates human-readable console output and GitHub Step Summary, - preserves full execution logs and XML results as CI artifacts, - separates test execution, result parsing, and summary generation to support future consumers of test result data. This is the first implementation step towards a broader test management and test health monitoring system. Database integration, historical analysis, and automated PR triage are outside the scope of this change.

kekaczma · 2026-08-19T14:21:15Z

failing test is related with known issue #22858

bratpiorka · 2026-08-20T08:37:46Z

Note: this should be enough on GitHub, but we could consider copying these artifacts and storing them on a server for longer.

rbanka1

there are also a lot of unnecessary comments, I think most of them could be deleted

rbanka1 · 2026-08-24T16:52:19Z

+        build_dir: build
+        artifact_name: 'ur-${{matrix.adapter.name}}-${{steps.artifact_suffix.outputs.suffix}}-adapter-specific'
      # Don't run adapter specific tests when building multiple adapters
      if: ${{ matrix.adapter.other_name == '' }}


In workflow files, please place if: before uses: / with: to match the usual convention in this repo.

rbanka1 · 2026-08-24T16:55:32Z

adapter_tests_results.xml is effectively a dead path

rbanka1 · 2026-08-24T16:55:48Z

adapter_tests_results.xml is effectively a dead path

Good catch. You're right — this fallback is effectively a dead path, since the runner is responsible for providing the actual XML output path. I'll remove it and rely on the runner output instead. If no XML path is provided, it will be treated as unavailable rather than trying to guess a fallback filename.

rbanka1 · 2026-08-24T16:58:08Z

remove unused Iterator from the typing import

rbanka1 · 2026-08-24T17:07:30Z

LIT options are defined in two places: devops/scripts/ur_test_tools/test_runner.py via lit_opts and hardcoded flags in unified-runtime/test/**/CMakeLists.txt when UR_STANDALONE_BUILD=ON. Integrated ur-build-hw uses the Python path; standalone builds use CMake. effectively we have two places to maintain the same intention

Good point. I agree that the reporting-related LIT options currently express the same intention in two places and could drift over time.

I'll move the shared reporting options on the Python side out of test_runner.py into a documented LIT_COMMON_REPORTING_OPTIONS constant, leaving only CI-specific and dynamically constructed options in the runner. I'll also document the corresponding standalone options in the relevant CMakeLists.txt files and add an explicit reference to the shared reporting configuration so that the relationship is visible when either side is modified.

I don't think the two invocations should be made fully identical, though. The integrated CI path needs verbose output and JUnit XML for parsing/reporting, while standalone execution is developer-facing and uses --succinct, so options such as -v vs --succinct are intentionally different.

I considered introducing a single configuration file consumed by both CMake and Python, but for this small set of options that would add extra coupling and complexity. I'd prefer to keep the two execution paths independent while making the shared reporting intent explicit and documented.

rbanka1 · 2026-08-24T17:10:50Z

 junitparser==3.2.0
 google-cloud-storage==3.3.0
 PyGithub==2.8.1
+defusedxml==0.7.1


For ur-build-hw, defusedxml in unified-runtime/third_party/requirements_testing.txt is sufficient. here is unnecessary

rbanka1 · 2026-08-24T17:21:22Z

it returns the count from the first matching stats line only, not the sum of both but LIT emits separate stat lines for both. So probably we could miss some data

Fixed. The parser now accumulates counts from repeated statistics lines instead of keeping only the first match, so no data is lost.

rbanka1 · 2026-08-24T17:28:39Z

right now, the Skipped and Unsupported categories are treated interchangeably in several places, while LIT output keeps them separate. that makes the summary hard to read, please split or keep one group with a clear title

Fixed. Skipped and Unsupported are now kept as separate statuses throughout parsing, reconciliation, and reporting, consistent with LIT output. The summary reports them as separate categories as well.

rbanka1 · 2026-08-24T17:36:12Z

you could consider writing the test summary to $GITHUB_STEP_SUMMARY, similar to other devops/actions/run-tests/* flows, so devs could see results on the workflow run page without opening each job and expanding steps. for example https://github.com/intel/llvm/actions/runs/32737557539
I think it will be minimal change with big benefit devops/scripts/ur-test summary "$LOG_FILE" "${XML_FILE:-}" | tee -a "$GITHUB_STEP_SUMMARY"

Implemented. I added a compact GitHub Step Summary with aggregated statistics and testing time. Test names are included only for statuses requiring attention (failed, unexpected pass, timeout and unresolved), with the list capped to keep the workflow summary readable.

…date test type and filter constants

Clarify the distinction between common reporting options (what test categories to show) and execution-specific options (how to format output for CI vs standalone builds).

New architecture: - Parser layer: produces ParsedLogData and ParsedXMLData observations - Reconciliation layer: combines observations into TestRunResult - Summary generator: consumes TestRunResult with group_by_status()

This change introduces common test result collection and reporting for Unified Runtime conformance and adapter-specific CI tests. Compared to the initial version, the implementation has been updated based on review feedback to: - keep LIT test statuses such as SKIPPED and UNSUPPORTED distinct, - reconcile LIT and JUnit XML results without losing LIT-specific status information, - separate test execution, parsing, reconciliation, and presentation responsibilities, - simplify the internal API and remove redundant wrappers and helpers, - keep CI reporting options separate from execution-specific LIT options. Focused regression tests were added for the non-trivial result mapping and reconciliation behavior identified during review.

kekaczma force-pushed the ur-ci-test-tools branch from 059d872 to 074c6b1 Compare August 10, 2026 15:21

kekaczma force-pushed the ur-ci-test-tools branch from 074c6b1 to 982c30e Compare August 17, 2026 21:59

kekaczma marked this pull request as ready for review August 19, 2026 14:17

kekaczma requested review from a team as code owners August 19, 2026 14:17

bratpiorka reviewed Aug 20, 2026

View reviewed changes

bratpiorka reviewed Aug 21, 2026

View reviewed changes

rbanka1 reviewed Aug 24, 2026

View reviewed changes

kekaczma added 3 commits August 25, 2026 12:01

Address code review feedback

1f301e2

Move ZE_ENABLE_LOADER_DEBUG_TRACE to workflow for L0-only and consoli…

1c120a3

…date test type and filter constants

Use GitHubActionsOutput static methods directly

f9f3d97

kekaczma marked this pull request as draft August 28, 2026 06:09

kekaczma added 7 commits August 28, 2026 08:31

Refactor test_runner: cleanup and explicit output contract

03d33f3

Separate LIT reporting semantics from execution policy

a71a717

Clarify the distinction between common reporting options (what test categories to show) and execution-specific options (how to format output for CI vs standalone builds).

Fix formatting: split arguments and remove trailing blank line

47366c0

Remove fallback file names - runner is single source of truth

1fde48d

Refactor test result model to address review comments

9fd79d8

New architecture: - Parser layer: produces ParsedLogData and ParsedXMLData observations - Reconciliation layer: combines observations into TestRunResult - Summary generator: consumes TestRunResult with group_by_status()

Add compact GitHub step summary for test results

ebbd0e7

bratpiorka requested review from bratpiorka and rbanka1 September 4, 2026 08:39

kekaczma marked this pull request as ready for review September 4, 2026 08:57

Uh oh!

Conversation

kekaczma commented Aug 10, 2026

Uh oh!

kekaczma commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbanka1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants