diff --git a/README.md b/README.md index 74bc50a..6bbfd9a 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Basic: ```yaml steps: - name: Run tests - uses: Particular/run-tests-action@v1.6.0 + uses: Particular/run-tests-action@v1.8.0 ``` With a reset script between each test run: @@ -21,7 +21,7 @@ With a reset script between each test run: ```yaml steps: - name: Run tests - uses: Particular/run-tests-action@v1.6.0 + uses: Particular/run-tests-action@v1.8.0 with: reset-script: | echo "Do whatever is necessary to reset the test infrastructure between runs of each framework" @@ -33,7 +33,7 @@ In cases where the test matrix subdivides by target framework, you can also shor ```yaml steps: - name: Run tests - uses: Particular/run-tests-action@v1.6.0 + uses: Particular/run-tests-action@v1.8.0 with: framework: net6.0 ``` @@ -43,7 +43,7 @@ By default, only failed tests are reported. To report warnings for tests that ha ```yaml steps: - name: Run tests - uses: Particular/run-tests-action@v1.6.0 + uses: Particular/run-tests-action@v1.8.0 with: report-warnings: true ``` @@ -53,11 +53,62 @@ By default, `dotnet test` uses `x64` as the target platform. This can be overrid ```yaml steps: - name: Run tests - uses: Particular/run-tests-action@v1.6.0 + uses: Particular/run-tests-action@v1.8.0 with: target-platform: x86 ``` +## Running a subset of projects + +By default the action discovers every `*.csproj` under `src/` that references `Microsoft.NET.Test.Sdk` and runs all of them. Pass `projects` to run an explicit, newline-delimited list of project paths instead, skipping discovery entirely. (Added in v1.8.0) + +This is the intended integration point for repositories that subdivide their test suite by category and select a subset of assemblies per matrix job. For example, ServiceControl's [`tools/select-test-projects.ps1`](https://github.com/Particular/ServiceControl/blob/master/tools/select-test-projects.ps1) writes each category's project list to `$GITHUB_OUTPUT` as a multiline `test-projects` value, which can be passed straight through: + +```yaml + steps: + - id: select + shell: pwsh + run: ./tools/select-test-projects.ps1 + - name: Run tests + uses: Particular/run-tests-action@v1.8.0 + with: + projects: ${{ steps.select.outputs.test-projects }} +``` + +When `projects` is combined with `framework`, each listed project is run only against that framework (projects that do not target it are skipped), mirroring the behavior of the discovery path. + +## Parallel execution + +By default the action runs `dotnet test` sequentially. Pass `max-parallel` (1–16) to run several test assemblies concurrently. (Added in v1.8.0) + +```yaml + steps: + - name: Run tests + uses: Particular/run-tests-action@v1.8.0 + with: + projects: ${{ steps.select.outputs.test-projects }} + max-parallel: 4 +``` + +When `max-parallel > 1`, each run's stdout and stderr are buffered to temp files and replayed inside a `::group::` block once that run completes, because interleaved live `dotnet test` output is unreadable. The step fails if any run exits non-zero. + +### Per-run parallel index + +Every spawned `dotnet test` process has the environment variable `PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX` set to its 0-based position in the flattened run list, immediately before it is spawned (so the child inherits it). The value is unique across all runs in the invocation, so concurrent runs always see distinct indices. In sequential mode (`max-parallel == 1`) the index is always `0`. + +Consumers that need per-run distinct resources — ports, temp directories, or anything else — can read this env var and derive what they need from the index. The action itself does no port arithmetic, keeping it repository-agnostic. For example, a suite using RavenDB.Embedded (which binds a fixed port and would otherwise collide across concurrent runs) can compute its port from the index: + +```csharp +var index = int.Parse(Environment.GetEnvironmentVariable("PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX") ?? "0"); +var port = 33334 + (index * 10); +``` + +Consumers that do not need per-run distinction simply ignore the variable. + +### Interaction with `reset-script` + +`reset-script` runs between consecutive target frameworks on the sequential path (`max-parallel == 1`), as it always has. When `max-parallel > 1`, runs are flattened across frameworks, so "between frameworks" no longer has a meaningful boundary and running the script concurrently with in-flight test processes is unsafe. In that case the reset script is ignored and the action emits a `::warning::` to make the skip visible. If you need a reset between batches, run sequential (`max-parallel: 1`) or invoke the reset script from a separate workflow step. + ## What about filters? This action does not support the [dotnet test filter syntax](https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests). This is because it's impossible to distinguish between the following cases: diff --git a/action.yml b/action.yml index 096f97c..f95d2f7 100644 --- a/action.yml +++ b/action.yml @@ -2,7 +2,7 @@ name: 'Run tests' description: 'Runs dotnet test using target frameworks appropriate for the current platform' inputs: reset-script: - description: 'pwsh expression to be run between test runs to reset infrastructure, if required' + description: 'pwsh expression to be run between test runs to reset infrastructure, if required. Only used when max-parallel is 1; ignored (with a warning) when max-parallel > 1.' required: false framework: description: Specifies the target framework to run tests for @@ -14,6 +14,22 @@ inputs: description: Specifies the RunConfiguration.TargetPlatform for dotnet test. Defaults to 'x64'. required: false default: x64 + projects: + description: >- + Optional newline-delimited list of project paths to run instead of + auto-discovering test projects. When provided, project discovery is + skipped and only the listed projects are tested. When absent, today's + discovery behavior (all *.csproj under src/ referencing + Microsoft.NET.Test.Sdk) is preserved. + required: false + max-parallel: + description: >- + Maximum number of `dotnet test` processes to run concurrently. Defaults + to 1 (sequential), which preserves the historic behavior. Values greater + than 1 enable parallel execution with buffered, per-run output replay, + and disable reset-script. + required: false + default: '1' runs: using: "composite" steps: @@ -37,4 +53,6 @@ runs: REPORT_WARNINGS: ${{ inputs.report-warnings }} TARGET_PLATFORM: ${{ inputs.target-platform }} TEST_FILTER: ${{ inputs.filter }} + TEST_PROJECTS: ${{ inputs.projects }} + MAX_PARALLEL: ${{ inputs.max-parallel }} run: ${{ github.action_path }}/run-tests.ps1 diff --git a/run-tests.ps1 b/run-tests.ps1 index c78f336..d152df3 100644 --- a/run-tests.ps1 +++ b/run-tests.ps1 @@ -1,100 +1,313 @@ -$testFrameworks = New-Object Collections.Generic.HashSet[String] -$testProjects = @{} +# Runs dotnet test for the test projects in the repository. +# +# Two execution modes: +# +# * Sequential (MAX_PARALLEL == 1, the default and historic behavior) -- runs `dotnet test` once +# per (project, framework), grouped by framework, and invokes RESET_SCRIPT between consecutive +# frameworks when one was supplied. This path is preserved verbatim from the pre-v1.8.0 action so +# existing consumers see no change. +# +# * Parallel (MAX_PARALLEL > 1) -- runs several `dotnet test` processes at once, buffering each +# run's stdout/stderr to temp files and replaying it inside a ::group:: on completion, because +# interleaved live `dotnet test` output is unreadable. Ported from ServiceControl's +# tools/run-tests.ps1. reset-script is ignored in this mode (with a warning), because running it +# concurrently with in-flight test processes is unsafe and "between frameworks" has no meaning +# once runs are flattened. +# +# Per-run parallel index: +# +# Every spawned `dotnet test` process (in both modes) has PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX +# set to its 0-based position in the flattened run list immediately before it is spawned, so the +# child inherits the value. The index is unique across all runs in the invocation, so concurrent +# runs always see distinct indices. In sequential mode the index is always 0 (only one run is +# active at a time). Consumers that need per-run distinct resources -- e.g. ServiceControl derives +# its RavenDB port as 33334 + (index * 10) -- read this env var; the action itself does no port +# arithmetic. +# +# Project selection: +# +# * If TEST_PROJECTS is set, it is a newline-delimited list of project paths; discovery is skipped +# and only those projects are tested. +# * Otherwise, today's discovery is used: every *.csproj under src/ that references +# Microsoft.NET.Test.Sdk. +# +# EXPLICIT_TEST_FRAMEWORK short-circuits framework discovery to a single value in both modes. + +$ErrorActionPreference = 'Stop' $explicitFramework = $Env:EXPLICIT_TEST_FRAMEWORK $isExplicitFramework = -not ([string]::IsNullOrEmpty($explicitFramework)) -if ($isExplicitFramework) { - $testFrameworks.Add($explicitFramework) - Write-Output "Target framework '$explicitFramework' defined by parameter. This is the only framework that will be tested." +$testFrameworks = New-Object Collections.Generic.HashSet[String] +if ($isExplicitFramework) { + $testFrameworks.Add($explicitFramework) > $null + Write-Output "Target framework '$explicitFramework' defined by parameter. This is the only framework that will be tested." } Write-Output "Target Platform = $($Env:TARGET_PLATFORM)" -$projects = Get-ChildItem -Path src -Include "*.csproj" -Recurse +# --- Project selection -------------------------------------------------------- +$explicitProjectsRaw = $Env:TEST_PROJECTS +$hasExplicitProjects = -not ([string]::IsNullOrEmpty($explicitProjectsRaw)) -$projects | ForEach-Object { - $project = $_.FullName +$testProjects = @{} - $testSdkNodes = Select-Xml -Path $project -XPath "/Project/ItemGroup/PackageReference[@Include='Microsoft.NET.Test.Sdk']" +if ($hasExplicitProjects) { + $projectPaths = $explicitProjectsRaw -Split "`n" | ForEach-Object { $_.Trim() } | Where-Object { $_ } + Write-Output "Using explicit project list ($($projectPaths.Count) project(s)) supplied via 'projects' input; project discovery is skipped." - if ( $testSdkNodes -ne $null ) { + foreach ($project in $projectPaths) { $projectFrameworks = New-Object Collections.Generic.List[String] - $testProjects.Add($project, $projectFrameworks) # In case of multiple target frameworks Select-Xml -Path $project -XPath "/Project/PropertyGroup/TargetFrameworks" | ForEach-Object { $frameworks = $_.node.InnerText -Split ';' - foreach( $framework in $frameworks) { - $testProjects.$project.Add($framework) - - if (-not $isExplicitFramework) { - $testFrameworks.Add($framework) > $null - } + foreach ($framework in $frameworks) { + $projectFrameworks.Add($framework) } } # In case of a single target framework Select-Xml -Path $project -XPath "/Project/PropertyGroup/TargetFramework" | ForEach-Object { - $testProjects.$project.Add($_.node.InnerText) + $projectFrameworks.Add($_.node.InnerText) + } - if (-not $isExplicitFramework) { - $testFrameworks.Add($_.node.InnerText) > $null + if ($isExplicitFramework) { + # Honor the framework override: keep only the explicit framework, and only if the + # project actually targets it. Mirrors the Contains($framework) check the discovery + # path uses when iterating. + $filtered = New-Object Collections.Generic.List[String] + foreach ($f in $projectFrameworks) { + if ($f -eq $explicitFramework) { + $filtered.Add($f) + } } + $projectFrameworks = $filtered } + else { + foreach ($f in $projectFrameworks) { + $testFrameworks.Add($f) > $null + } + } + + $testProjects.Add($project, $projectFrameworks) } } +else { + $projects = Get-ChildItem -Path src -Include "*.csproj" -Recurse -$testProjects = $testProjects.GetEnumerator() | Sort-Object Name -$testFrameworks = $testFrameworks.GetEnumerator() | Sort-Object -$reportWarnings = 'false' + $projects | ForEach-Object { + $project = $_.FullName + $testSdkNodes = Select-Xml -Path $project -XPath "/Project/ItemGroup/PackageReference[@Include='Microsoft.NET.Test.Sdk']" + + if ( $testSdkNodes -ne $null ) { + $projectFrameworks = New-Object Collections.Generic.List[String] + $testProjects.Add($project, $projectFrameworks) + + # In case of multiple target frameworks + Select-Xml -Path $project -XPath "/Project/PropertyGroup/TargetFrameworks" | ForEach-Object { + $frameworks = $_.node.InnerText -Split ';' + foreach ($framework in $frameworks) { + $testProjects.$project.Add($framework) + + if (-not $isExplicitFramework) { + $testFrameworks.Add($framework) > $null + } + } + } + + # In case of a single target framework + Select-Xml -Path $project -XPath "/Project/PropertyGroup/TargetFramework" | ForEach-Object { + $testProjects.$project.Add($_.node.InnerText) + + if (-not $isExplicitFramework) { + $testFrameworks.Add($_.node.InnerText) > $null + } + } + } + } +} + +$reportWarnings = 'false' if ($Env:REPORT_WARNINGS -eq 'true') { $reportWarnings = 'true' } -$exitCode = 0 -$counter = 0 +$maxParallel = [Math]::Max(1, [int]$Env:MAX_PARALLEL) +if ($maxParallel -eq 1) { + # --- Sequential path: historic behavior, reset-script between frameworks --- + $testProjectsSorted = $testProjects.GetEnumerator() | Sort-Object Name + $testFrameworksSorted = $testFrameworks.GetEnumerator() | Sort-Object + + $exitCode = 0 + $counter = 0 + + foreach ($framework in $testFrameworksSorted) { + + $counter = $counter + 1 + + if (($PSVersionTable.Platform -eq 'Unix') -and ($framework.StartsWith("net4") -or $framework.Contains("-windows"))) { + continue + } + + foreach ($project in $testProjectsSorted) { + + if (-not $project.Value.Contains($framework)) { + continue + } + + Write-Output "::group::Running $(Split-Path $project.Name -leaf) ($framework)" + + $targetPlatformParam = "RunConfiguration.TargetPlatform=$($Env:TARGET_PLATFORM)" + + $Env:PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX = '0' + + dotnet test $project.Name --configuration Release --no-build --framework $framework --logger "GitHubActions;report-warnings=$reportWarnings" -- RunConfiguration.TreatNoTestsAsError=true $targetPlatformParam + + Write-Output "::endgroup::" -foreach ($framework in $testFrameworks) { + if ($LASTEXITCODE -ne 0) { + Write-Output "::error::Exit code = $LASTEXITCODE" + $exitCode = 1 + } + } + + if (($counter -lt $testFrameworksSorted.Count) -and ($Env:HAS_RESET_SCRIPT -eq 'true')) { + Write-Output "::group::Running reset script" + Invoke-Expression $Env:RESET_SCRIPT + Write-Output "::endgroup::" + + if ($LASTEXITCODE -ne 0) { + Write-Output "::error::Exit code = $LASTEXITCODE" + $exitCode = 1 + } + } + } - $counter = $counter + 1 + exit $exitCode +} +else { + # --- Parallel path: ported from ServiceControl tools/run-tests.ps1 --- + Write-Output "Max parallel test runs = $maxParallel" - if (($PSVersionTable.Platform -eq 'Unix') -and ($framework.StartsWith("net4") -or $framework.Contains("-windows"))) { - continue + if ($Env:HAS_RESET_SCRIPT -eq 'true') { + Write-Output "::warning::reset-script is ignored when max-parallel > 1. Running it concurrently with in-flight test processes is unsafe, and 'between frameworks' has no meaning once runs are flattened." } - foreach ($project in $testProjects) { + $isUnix = $PSVersionTable.Platform -eq 'Unix' + + $testProjectsSorted = $testProjects.GetEnumerator() | Sort-Object Name + $testFrameworksSorted = $testFrameworks.GetEnumerator() | Sort-Object + + # Flatten into a list of (Label, Project, Framework) runs, ordered by framework then project, + # skipping frameworks that cannot run on this platform. Each run is tagged with its 0-based + # Index in this flattened list, which is exposed to consumers via + # PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX (see D3) so concurrent runs can derive distinct + # per-run resources (ports, temp dirs, etc.) from it. + $runs = [Collections.Generic.List[object]]::new() + $runIndex = 0 - if (-not $project.Value.Contains($framework)) { + foreach ($framework in $testFrameworksSorted) { + if ($isUnix -and ($framework.StartsWith('net4') -or $framework.Contains('-windows'))) { continue } - Write-Output "::group::Running $(Split-Path $project.Name -leaf) ($framework)" + foreach ($project in $testProjectsSorted) { + if (-not $project.Value.Contains($framework)) { + continue + } - $targetPlatformParam = "RunConfiguration.TargetPlatform=$($Env:TARGET_PLATFORM)" + $runs.Add([pscustomobject]@{ + Label = "$(Split-Path $project.Name -Leaf) ($framework)" + Project = $project.Name + Framework = $framework + Index = $runIndex + }) + $runIndex++ + } + } + + if ($runs.Count -eq 0) { + throw 'No test projects were runnable on this platform.' + } - dotnet test $project.Name --configuration Release --no-build --framework $framework --logger "GitHubActions;report-warnings=$reportWarnings" -- RunConfiguration.TreatNoTestsAsError=true $targetPlatformParam + # Per-run parallel index. Concurrent runs need to distinguish themselves (e.g. so suites that + # bind a fixed port like ServiceControl's RavenDB.Embedded tests do not all probe-and-bind the + # same port). Rather than bake port arithmetic into the action, each run is tagged with its + # 0-based position in the flattened run list (unique across the whole invocation) and that value + # is exposed via PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX immediately before spawning, so the + # child inherits it. Consumers derive whatever per-run distinct resource they need from the + # index. The action itself does no port arithmetic. (See design decision D3.) + $exitCode = 0 - Write-Output "::endgroup::" + function Complete-Run($run) { + Write-Output "::group::Running $($run.Label)" + foreach ($stream in @($run.OutFile, $run.ErrFile)) { + if ((Test-Path $stream) -and (Get-Item $stream).Length -gt 0) { + Get-Content -Path $stream | Write-Output + } + Remove-Item -Path $stream -Force -ErrorAction SilentlyContinue + } + Write-Output '::endgroup::' - if ($LASTEXITCODE -ne 0) { - Write-Output "::error::Exit code = $LASTEXITCODE" - $exitCode = 1 + if ($run.Process.ExitCode -ne 0) { + Write-Output "::error::$($run.Label) exit code = $($run.Process.ExitCode)" + $script:exitCode = 1 } } - if (($counter -lt $testFrameworks.Count) -and ($Env:HAS_RESET_SCRIPT -eq 'true')) { - Write-Output "::group::Running reset script" - Invoke-Expression $Env:RESET_SCRIPT - Write-Output "::endgroup::" + $pending = [Collections.Generic.Queue[object]]::new($runs) + $active = [Collections.Generic.List[object]]::new() + + while ($pending.Count -gt 0 -or $active.Count -gt 0) { + while ($active.Count -lt $maxParallel -and $pending.Count -gt 0) { + $run = $pending.Dequeue() + $run | Add-Member -NotePropertyName OutFile -NotePropertyValue ([IO.Path]::GetTempFileName()) + $run | Add-Member -NotePropertyName ErrFile -NotePropertyValue ([IO.Path]::GetTempFileName()) + + $arguments = @( + 'test', $run.Project + '--configuration', 'Release' + '--no-build' + '--framework', $run.Framework + '--logger', "GitHubActions;report-warnings=$reportWarnings" + '--' + 'RunConfiguration.TreatNoTestsAsError=true' + "RunConfiguration.TargetPlatform=$($Env:TARGET_PLATFORM)" + ) + + # Expose this run's 0-based index to the spawned process so consumers can derive + # per-run distinct resources from it (e.g. a unique port). Unique across all runs in + # the invocation; set immediately before spawning so the child inherits it. + $Env:PARTICULAR_RUN_TESTS_ACTION_PARALLEL_INDEX = "$($run.Index)" - if ($LASTEXITCODE -ne 0) { - Write-Output "::error::Exit code = $LASTEXITCODE" - $exitCode = 1 + Write-Output "Starting $($run.Label)" + + $run | Add-Member -NotePropertyName Process -NotePropertyValue ( + Start-Process -FilePath 'dotnet' -ArgumentList $arguments -NoNewWindow -PassThru ` + -RedirectStandardOutput $run.OutFile -RedirectStandardError $run.ErrFile) + $active.Add($run) + } + + $finished = $active | Where-Object { $_.Process.HasExited } + + if (-not $finished) { + Start-Sleep -Milliseconds 500 + continue + } + + foreach ($run in @($finished)) { + # Bounded on purpose. The parameterless WaitForExit() also waits for the redirected streams to + # reach EOF, and on Linux Start-Process pumps them through a pipe, so a test that leaves behind + # a child holding the inherited handle blocks it forever. HasExited has already told us the + # test process itself is done; this only gives the pump a moment to drain. + [void]$run.Process.WaitForExit(5000) + Complete-Run $run + [void]$active.Remove($run) } } -} -exit $exitCode + exit $exitCode +}