[rocprofiler-systems] Enhance PyTest and port more tests by kcossett-amd · Pull Request #3158 · ROCm/rocm-systems

kcossett-amd · 2026-02-09T20:11:11Z

Motivation

Add more logic to pytest to handle CI, extra markers, detection of runtime capabilities and to port more CTests

To build:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cmake -B rocprof-sys-build -S .                                                  \
       -D CMAKE_INSTALL_PREFIX=/opt/rocprofiler-systems                          \
       -D ROCPROFSYS_USE_PYTHON=ON      -D ROCPROFSYS_BUILD_DYNINST=ON           \
       -D ROCPROFSYS_BUILD_TBB=ON       -D ROCPROFSYS_BUILD_BOOST=ON             \
       -D ROCPROFSYS_BUILD_ELFUTILS=ON  -D ROCPROFSYS_BUILD_LIBIBERTY=ON         \
       -D ROCPROFSYS_BUILD_FOR_THEROCK=ON
cmake --build rocprof-sys-build --target all --parallel 8

To run:

# Without package (see README.md to use package)

# --dev flag bundles up some QOL flags
pytest rocprof-sys-build/share/rocprofiler-systems/tests/pytest/ --dev 

# Simulate what TheRock would see
# (Tests marked with "ci_disable", "slow", and "mpi" markers are disabled)
pytest rocprof-sys-build/share/rocprofiler-systems/tests/pytest/ --ci-mode

If running with pytest-xdist (i.e. using -n <nprocs>), you MUST specify --dist=loadgroup.

Technical Details

Note: At some point down the line, CTests will eventually call PyTests. This will come in a future PR.

JIRA ID

Test Plan

Tested Locally. RCCL Tests fail but they also fail for CTests.
NOTE!!!!: No new tests will run in --ci-mode.

Test Result

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

kcossett-amd

Some remarks

Copilot

Pull request overview

This PR migrates additional rocprofiler-systems CTest coverage into the PyTest-based harness and expands the PyTest infrastructure to support more modes (e.g., causal, python), richer environment/capability checks, and standalone execution.

Changes:

Ported multiple existing tests to use the RocprofsysTest base class + unified run_test/assertion helpers.
Added new PyTest modules for MPI/UCX/NIC/causal/python/code-coverage/etc. and expanded marker/CI/CTest integration in conftest.py.
Improved validation + runner plumbing (regex handling, file regex validation, frozen/standalone Python selection, new runner types).

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
projects/rocprofiler-systems/tests/pytest/test_videodecode.py	Refactors test to `RocprofsysTest` and parametrizes modes/rules.
projects/rocprofiler-systems/tests/pytest/test_user_api.py	Adds new user API tests across multiple modes.
projects/rocprofiler-systems/tests/pytest/test_units.py	Adds build-mode-only unit test runner.
projects/rocprofiler-systems/tests/pytest/test_ucx.py	Adds UCX/MPI transport tests with perfetto validation.
projects/rocprofiler-systems/tests/pytest/test_transpose.py	Large refactor to parametrized tests + MPI optional handling + ROCProfiler validation.
projects/rocprofiler-systems/tests/pytest/test_time_window.py	Refactors time window tests to shared mode-parametrized implementation.
projects/rocprofiler-systems/tests/pytest/test_thread_limit.py	Adds thread-limit stress tests with dynamic thread count selection.
projects/rocprofiler-systems/tests/pytest/test_roctx.py	Refactors ROCTx tests to `RocprofsysTest` and CI-disable markers.
projects/rocprofiler-systems/tests/pytest/test_rewrite_caller.py	Adds rewrite caller-include coverage.
projects/rocprofiler-systems/tests/pytest/test_rccl.py	Refactors RCCL tests with CI disable + mode parametrization.
projects/rocprofiler-systems/tests/pytest/test_python.py	Adds Python profiling tests (external/builtin/source/etc.).
projects/rocprofiler-systems/tests/pytest/test_pthread.py	Adds pthread lock/overhead tests across instrumentation modes.
projects/rocprofiler-systems/tests/pytest/test_presets.py	Adds preset smoke tests for sample/run binaries.
projects/rocprofiler-systems/tests/pytest/test_overflow.py	Adds sampling-overflow tests.
projects/rocprofiler-systems/tests/pytest/test_openmp.py	Refactors OpenMP tests and adds duration/no-tmp coverage.
projects/rocprofiler-systems/tests/pytest/test_nic_perf.py	Adds NIC counter/perfetto validation test using wget download workload.
projects/rocprofiler-systems/tests/pytest/test_mpi.py	Adds MPI + MPIP tests including perfetto merge validation.
projects/rocprofiler-systems/tests/pytest/test_lulesh.py	Adds lulesh Kokkos/perfetto/timemory coverage.
projects/rocprofiler-systems/tests/pytest/test_jpegdecode.py	Refactors test to `RocprofsysTest` and parametrizes modes/rules.
projects/rocprofiler-systems/tests/pytest/test_hip_stream.py	Adds group-by-queue/stream tests for HIP stream behavior.
projects/rocprofiler-systems/tests/pytest/test_gpu_connect.py	Refactors connectivity test and adds conditional skip logic.
projects/rocprofiler-systems/tests/pytest/test_fork.py	Adds fork-related tests across modes and GPU conditional target.
projects/rocprofiler-systems/tests/pytest/test_config.py	Refactors config failure tests to use config-file helper.
projects/rocprofiler-systems/tests/pytest/test_code_coverage.py	Adds code coverage tests + cross-test output reuse.
projects/rocprofiler-systems/tests/pytest/test_causal.py	Adds causal tests + end-to-end validation of causal JSON output.
projects/rocprofiler-systems/tests/pytest/test_binaries.py	Refactors binary CLI tests to `RocprofsysTest` and new markers.
projects/rocprofiler-systems/tests/pytest/test_attach.py	Adds attach test (CI disabled) using helper script.
projects/rocprofiler-systems/tests/pytest/test_annotate.py	Adds annotate-mode tests with PAPI conditional perfetto checks.
projects/rocprofiler-systems/tests/pytest/rocprofsys/validators.py	Adds file-regex validation + frozen-safe Python selection + JSON arg fix.
projects/rocprofiler-systems/tests/pytest/rocprofsys/runners.py	Improves runner robustness, adds CausalRunner/PythonRunner, stdin handling.
projects/rocprofiler-systems/tests/pytest/rocprofsys/gpu.py	Disables GPU detection when `ROCPROFSYS_USE_ROCM=OFF`.
projects/rocprofiler-systems/tests/pytest/rocprofsys/config.py	Adds capabilities + python discovery + better executable discovery.
projects/rocprofiler-systems/tests/pytest/rocprofsys/capabilities.py	New system capability probing for conditional skips/markers.
projects/rocprofiler-systems/tests/pytest/rocprofsys/init.py	Exposes new runners/validators.
projects/rocprofiler-systems/tests/pytest/conftest.py	Major harness expansion: modes, markers, CTest integration, helpers, base test class.
projects/rocprofiler-systems/tests/pytest/build_standalone.sh	Updates standalone build to PyInstaller + bundled plugins/deps.
projects/rocprofiler-systems/tests/pytest/README.md	Documents xdist loadgroup usage + new flags and standalone constraints.
projects/rocprofiler-systems/tests/pytest/CMakeLists.txt	Installs pytest package files including new capabilities module and builds standalone exe.
projects/rocprofiler-systems/tests/CMakeLists.txt	Builds/installs capability checker binary for PyTest-based system probing.
projects/rocprofiler-systems/examples/fork/CMakeLists.txt	Adjusts HIP fork example output directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add option to print config then exit (for CTests) - Add support for ROCPROFSYS_USE_ROCM=OFF - Rename disable marker to ci_disable to make it clear it is for --ci-mode only - Add gpu mark to rccl tests - Fix bug with gpu marker - Remove pytest-order - Fix discovery of python scripts used for tests

- Tell user to use --dist=loadgroup when xdist is used - Group labels

- config.py - Add ROCPROFSYS_CI=ON to get_base_python_environment() - Add ROCPROFSYS_CI_TIMEOUT to each base environment - Add get_base_causal_environment - Adjust doc strings - get_base_environment() now uses ROCPROFSYS_LOG_LEVEL="trace" - capabilities.py - Add missing timeouts and adjust them all to 10 - Search stderr for fail regex in ucx_availability() - Add missing OMPI env to ucx_availabilty() - Refactor mpi_implementation() to reduce code duplication - runners.py - All runners now pass in a respective base environment to the BaseRunner class - conftest.py - Remove duplicate MPI label from generic_functional_markers - Add functional markers (python) - Use snake case - Rework collect_output_path() (Added OutputPathStore class) - Remove Optional from cond in add_marker_if() - test_presents.py - Make PRESETS a module level variable to reduce code duplication - test_causal.py - Adjust causal_env() to account for get_base_causal_environment() - test_binaries.py - Fix typo - test_code_coverage.py - Adjust code to account for changes in collect_output_path() - test_lulesh.py, test_rccl.py, test_python.py - Fix module doc string - test_time_window.py, test_thread_limit.py, test_openmp.py - Remove deprecated env vars - test_ucx.py - Adjust PASS and FAIL regexes - Remove deprecated env vars

- Rework ci_disable marker. Now use ci_enable to selet tests in --ci-mode. This can be overrided using ci_disable("all") or ci_disable(<assert name>) - Only the tests that were already running in CI before this PR will have ci_enable. Future tests will be enabled incrementally so as to not block this PR.

kcossett-amd · 2026-02-19T12:16:21Z

Verifying PR does not break existing tests on TheRock

…/pytest-transfer

kcossett-amd · 2026-02-20T17:24:03Z

Enabled tests pass on TheRock: https://github.com/ROCm/TheRock/actions/runs/22230038737/job/64319819866?pr=3537

sputhala-amd

LGTM

github-actions Bot added the project: rocprofiler-systems label Feb 9, 2026

systems-assistant Bot added the organization: ROCm label Feb 9, 2026

kcossett-amd commented Feb 10, 2026

View reviewed changes

Comment thread projects/rocprofiler-systems/tests/pytest/test_fork.py Outdated

Comment thread projects/rocprofiler-systems/tests/pytest/test_code_coverage.py

Comment thread projects/rocprofiler-systems/tests/pytest/CMakeLists.txt Outdated

kcossett-amd added testing Adding an addition test or example WIP labels Feb 10, 2026

kcossett-amd commented Feb 10, 2026

View reviewed changes

Comment thread projects/rocprofiler-systems/tests/pytest/test_annotate.py

kcossett-amd changed the title ~~[rocprofiler-systems] Port more CTests to PyTests~~ [rocprofiler-systems] Enhance PyTest and port more tests Feb 10, 2026

kcossett-amd force-pushed the users/kcossett-amd/pytest-transfer branch from 5f6943a to af00044 Compare February 10, 2026 20:17

kcossett-amd marked this pull request as ready for review February 11, 2026 13:28

kcossett-amd requested a review from jrmadsen as a code owner February 11, 2026 13:28

Copilot AI review requested due to automatic review settings February 11, 2026 13:28

kcossett-amd requested review from a team as code owners February 11, 2026 13:28

Copilot AI reviewed Feb 11, 2026

View reviewed changes

mradosav-amd reviewed Feb 12, 2026

View reviewed changes