[CI][Bench] Combine all benchmark-related jobs in one place #20439
+133
−148
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,59 +1,19 @@ | ||
| # A combined workflow for all benchmarks-related jobs for SYCL and UR. | ||
| # Supports both manual triggering (dispatch) and nightly runs. | ||
| # It also tests changes to benchmark scripts/framework in PR, if modified. | ||
| name: SYCL Run Benchmarks | ||
|
|
||
| on: | ||
| schedule: | ||
| # 3 hours ahead of SYCL nightly | ||
| - cron: '0 0 * * *' | ||
| # Run on pull requests only when a benchmark-related files were changed. | ||
| pull_request: | ||
| # These paths are exactly the same as in sycl-linux/windows-precommit.yml (to ignore over there) | ||
| paths: | ||
| - 'devops/scripts/benchmarks/**' | ||
| - 'devops/actions/run-tests/benchmark/**' | ||
| - '.github/workflows/sycl-ur-perf-benchmarking.yml' | ||
| workflow_dispatch: | ||
| inputs: | ||
| preset: | ||
|
|
@@ -66,6 +26,8 @@ on: | |
| - Minimal | ||
| - Normal | ||
| - Test | ||
| - Gromacs | ||
| - OneDNN | ||
| default: 'Minimal' # Only compute-benchmarks | ||
| pr_no: | ||
| type: string | ||
|
|
@@ -115,13 +77,14 @@ on: | |
| options: | ||
| - 'level_zero:gpu' | ||
| - 'level_zero_v2:gpu' | ||
|
|
||
| permissions: read-all | ||
|
|
||
| jobs: | ||
| # Manual trigger (dispatch) path: | ||
| sanitize_inputs_dispatch: | ||
| name: '[Dispatch] Sanitize inputs' | ||
| if: github.event_name == 'workflow_dispatch' | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| COMMIT_HASH: ${{ inputs.commit_hash }} | ||
|
|
@@ -169,25 +132,25 @@ jobs: | |
| echo "Final sanitized values:" | ||
| cat $GITHUB_OUTPUT | ||
|
|
||
| build_sycl_dispatch: | ||
| name: '[Dispatch] Build SYCL' | ||
| needs: [ sanitize_inputs_dispatch ] | ||
| uses: ./.github/workflows/sycl-linux-build.yml | ||
| with: | ||
| build_ref: ${{ needs.sanitize_inputs_dispatch.outputs.build_ref }} | ||
| build_cache_root: "/__w/" | ||
| build_cache_suffix: "prod_noassert" | ||
| build_configure_extra_args: "--no-assertions" | ||
| build_image: "ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest" | ||
| cc: clang | ||
| cxx: clang++ | ||
| changes: '[]' | ||
| toolchain_artifact: sycl_linux_prod_noassert | ||
|
|
||
| benchmark_dispatch: | ||
| name: '[Dispatch] Benchmarks' | ||
| needs: [ build_sycl_dispatch, sanitize_inputs_dispatch ] | ||
| if: always() && !cancelled() && needs.build_sycl_dispatch.outputs.build_conclusion == 'success' | ||
| strategy: | ||
| matrix: | ||
| include: | ||
|
|
@@ -197,17 +160,102 @@ jobs: | |
| uses: ./.github/workflows/sycl-linux-run-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| name: "Benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}, preset: ${{ matrix.preset }})" | ||
| runner: ${{ matrix.runner }} | ||
| image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest | ||
| image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN | ||
| target_devices: ${{ matrix.backend }} | ||
| tests_selector: benchmarks | ||
| benchmark_upload_results: ${{ inputs.upload_results }} | ||
| benchmark_save_name: ${{ needs.sanitize_inputs_dispatch.outputs.benchmark_save_name }} | ||
| benchmark_preset: ${{ inputs.preset }} | ||
| benchmark_exit_on_failure: ${{ inputs.exit_on_failure }} | ||
| repo_ref: ${{ needs.sanitize_inputs_dispatch.outputs.build_ref }} | ||
| toolchain_artifact: ${{ needs.build_sycl_dispatch.outputs.toolchain_artifact }} | ||
| toolchain_artifact_filename: ${{ needs.build_sycl_dispatch.outputs.toolchain_artifact_filename }} | ||
| toolchain_decompress_command: ${{ needs.build_sycl_dispatch.outputs.toolchain_decompress_command }} | ||
| # END manual trigger (dispatch) path | ||
|
|
||
| # Nightly benchmarking path: | ||
| build_nightly: | ||
| name: '[Nightly] Build SYCL' | ||
|
There was a problem hiding this comment. I didn't look at the code for the jobs in depth but why do we need separate jobs depending on the workflow call type? There was a problem hiding this comment. as I mentioned in the comment above - this is the first approach just to merge all benchmarking jobs in one workflow, then I'll try to make it simpler - we need to figure out, why there are e.g. different params for each build... and which one is the best to use for all jobs. There was a problem hiding this comment. @sarnex , please let me know if it is ok to merge it as-is and work on this later on. If you're good with this change, you can please merge it. There was a problem hiding this comment. Seems fine for now, hope to see continued improvements |
||
| if: github.repository == 'intel/llvm' && github.event_name == 'schedule' | ||
| uses: ./.github/workflows/sycl-linux-build.yml | ||
| secrets: inherit | ||
| with: | ||
| build_cache_root: "/__w/" | ||
| build_configure_extra_args: '--no-assertions' | ||
| build_image: ghcr.io/intel/llvm/ubuntu2404_build:latest | ||
|
|
||
| toolchain_artifact: sycl_linux_default | ||
| toolchain_artifact_filename: sycl_linux.tar.gz | ||
|
|
||
| benchmark_nightly: | ||
| name: '[Nightly] Benchmarks' | ||
| needs: [build_nightly] | ||
| if: always() && !cancelled() && needs.build_nightly.outputs.build_conclusion == 'success' | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| runner: ['["PVC_PERF"]', '["BMG_PERF"]'] | ||
| backend: ['level_zero:gpu', 'level_zero_v2:gpu'] | ||
| include: | ||
| - ref: ${{ github.sha }} | ||
| save_name: 'Baseline' | ||
| preset: 'Full' | ||
| uses: ./.github/workflows/sycl-linux-run-tests.yml | ||
| secrets: inherit | ||
| with: | ||
| name: "Benchmarks (${{ matrix.runner }}, ${{ matrix.backend }}, preset: ${{ matrix.preset }})" | ||
| runner: ${{ matrix.runner }} | ||
| image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest | ||
| image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN | ||
| target_devices: ${{ matrix.backend }} | ||
| tests_selector: benchmarks | ||
| benchmark_upload_results: true | ||
| benchmark_save_name: ${{ matrix.save_name }} | ||
| benchmark_preset: ${{ matrix.preset }} | ||
| repo_ref: ${{ matrix.ref }} | ||
| toolchain_artifact: ${{ needs.build_nightly.outputs.toolchain_artifact }} | ||
| toolchain_artifact_filename: ${{ needs.build_nightly.outputs.toolchain_artifact_filename }} | ||
| toolchain_decompress_command: ${{ needs.build_nightly.outputs.toolchain_decompress_command }} | ||
| # END nightly benchmarking path | ||
|
|
||
| # Benchmark framework builds and runs on PRs path: | ||
| build_pr: | ||
| name: '[PR] Build SYCL' | ||
| if: github.event_name == 'pull_request' | ||
| uses: ./.github/workflows/sycl-linux-build.yml | ||
| with: | ||
| build_ref: ${{ github.sha }} | ||
| build_cache_root: "/__w/" | ||
| build_cache_suffix: "default" | ||
| # Docker image has last nightly pre-installed and added to the PATH | ||
| build_image: "ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest" | ||
| cc: clang | ||
| cxx: clang++ | ||
| changes: ${{ needs.detect_changes.outputs.filters }} | ||
| toolchain_artifact: sycl_linux_default | ||
|
|
||
| # TODO: When we have stable BMG runner(s), consider moving this job to that runner. | ||
| test_benchmark_framework: | ||
| name: '[PR] Benchmark suite testing' | ||
| needs: [build_pr] | ||
| if: always() && !cancelled() && needs.build_pr.outputs.build_conclusion == 'success' | ||
| uses: ./.github/workflows/sycl-linux-run-tests.yml | ||
| with: | ||
| name: 'Framework test: PVC_PERF, L0, Minimal preset' | ||
| runner: '["PVC_PERF"]' | ||
|
There was a problem hiding this comment. @intel/llvm-reviewers-benchmarking Perhaps we should change this to BMG runner? I guess PVC runner is used more often for manual runs, so we wouldn't have to wait for benchmarks to finish to only test the framework...? What do you think? There was a problem hiding this comment. Once there are two stable BMG machines, this would be a good move. For now, the PVC machine is much, much, much more stable than the BMG one, so I would stay with the current config. Add TODO to consider the change in the future, though. There was a problem hiding this comment. You're so right. TODO added. |
||
| image: ghcr.io/intel/llvm/sycl_ubuntu2404_nightly:latest | ||
| image_options: -u 1001 --device=/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --privileged --cap-add SYS_ADMIN | ||
| target_devices: 'level_zero:gpu' | ||
| tests_selector: benchmarks | ||
| benchmark_upload_results: false | ||
| benchmark_preset: 'Minimal' | ||
| benchmark_dry_run: true | ||
| benchmark_exit_on_failure: true | ||
| repo_ref: ${{ github.sha }} | ||
| toolchain_artifact: ${{ needs.build.outputs.toolchain_artifact }} | ||
| toolchain_artifact_filename: ${{ needs.build.outputs.toolchain_artifact_filename }} | ||
| toolchain_decompress_command: ${{ needs.build.outputs.toolchain_decompress_command }} | ||
| # END benchmark framework builds and runs on PRs path | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a next step I'd like to verify if we could work with a single build job - right now I just copy-pasted build jobs from other workflows... but they differ in paramaters - e.g. sometimes we use clang compiler and sometimes (default) gcc - this would probably be discussed and we should just use the same build for all benchmark jobs, if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I don't know the reason for different compilers usage, too. It seems a bit odd to be able to compare on-demand results with nightly ones with sycl built with a totally different compiler.