Flaky test: AzureEventHubsExtensionsTests.VerifyAzureEventHubsEmulatorResource times out under CI load

[davidfowl]

Test

Aspire.Hosting.Azure.Tests.AzureEventHubsExtensionsTests.VerifyAzureEventHubsEmulatorResource(referenceHub: True, hubName: null)

Error

Polly.Timeout.TimeoutRejectedException : The operation didn't complete within the allowed timeout of '00:00:20'.
---- System.Threading.Tasks.TaskCanceledException : A task was canceled.

Build

https://github.com/dotnet/aspire/actions/runs/22538430952/job/65289833249

Analysis

The test times out waiting for the EventHub emulator health check to pass. CI heartbeat logs show the machine was under extreme resource contention at the time:

• CPU: ~96% sustained
• VBCSCompiler: 170% CPU / 988 MB
• bicep: ~100-120% CPU
• 11-16 DCP processes running simultaneously (133-170% aggregate CPU)
• Multiple node processes at 30-60% CPU each

The 20-second Polly timeout is not sufficient when the CI machine is this saturated. The emulator container was starting up (Microsoft.Cloud.EventHub.Emulator.Host visible at ~18-30% CPU) but the health check at /health did not return 200 within the timeout window.

This is a different root cause than the previously closed #6751 (which was an AMQP entity-not-found error). This failure is purely a resource contention timeout.

Suggested Fix

Consider increasing the Polly timeout for this test or marking it as a quarantined test if it continues to be flaky under CI load.

[davidfowl]

CI Heartbeat Logs (around failure time)

The test ran from ~07:24:43 to ~07:25:08 (24.9s, 20s Polly timeout + overhead). Here are the heartbeat snapshots showing sustained resource contention:

[07:23:35] CPU: calculating | DCP: 2 procs (136.4%) | Top: VBCSCompiler 219% CPU/988MB
[07:23:40] CPU: 98.5% | DCP: 9 procs (148.6%) | Top: VBCSCompiler 202%, Tests 122%
[07:23:45] CPU: 99.5% | DCP: 9 procs (111.8%) | Top: VBCSCompiler 188%
[07:23:51] CPU: 96.3% | DCP: 11 procs (133.8%) | Docker: 2 containers (CPU: 140.7%) | Top: VBCSCompiler 170%, bicep 99.5%, node 97.2%, dcp 86.5%
[07:23:58] CPU: 88.4% | DCP: 12 procs (144.5%) | Docker: 3 containers (CPU: 8.8%) | Top: VBCSCompiler 155%, bicep 106%, bicep 102%, dcp 85.3%
[07:24:06] CPU: 96.7% | DCP: 16 procs (170.9%) | Docker: 2 containers (CPU: 3.1%) | Top: VBCSCompiler 143%, bicep 120%
[07:24:14] CPU: 95.4% | DCP: 14 procs (69.5%)  | Docker: 3 containers (CPU: 1.0%) | Top: VBCSCompiler 132%, bicep 120%
[07:24:28] CPU: 68.7% | DCP: 11 procs (62.6%)  | Docker: 2 containers (CPU: 97.2%)

Test Error Output

failed VerifyAzureEventHubsEmulatorResource(referenceHub: True, hubName: null) (24s 890ms)
  Polly.Timeout.TimeoutRejectedException : The operation didn't complete within the allowed timeout of '00:00:20'.
  ---- System.Threading.Tasks.TaskCanceledException : A task was canceled.
    at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](...)
    ...
    at Aspire.Hosting.Dcp.KubernetesService.<>c__DisplayClass28_0`1.<<ExecuteWithRetry>b__0>d.MoveNext()
      in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:482
    at Aspire.Hosting.Dcp.KubernetesService.EnsureKubernetesAsync(CancellationToken cancellationToken)
      in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:586

Resource Notification State at Failure

Resource eventhubns/eventhubns-pneebbet: State = (null), ResourceReady = False
  HealthReports = { eventhubns_emulatorhealth_/health_200_check = (null) }

Resource eventhubns-storage/eventhubns-storage-tzwsbkjp: State = (null), ResourceReady = False
  HealthReports = { eventhubns-storage_check = (null) }

Resource hub/hub: State = (null), ResourceReady = False
  HealthReports = { eventhubns_emulatorhealth_/health_200_check = (null) }

All three resources were still in initial state with null health reports when the timeout fired. The emulator process (Microsoft.Cloud.EventHub.Emulator.Host) was running at 18-30% CPU but hadn't finished initializing within the 20s window.

[davidfowl]

DCP Logs (session aspire.hosting.tests-38b51df2)

Complete timeline from the DCP run-controllers log for the failing test:

07:25:10.951  ContainerOrchestrator: Runtime status - docker installed & running
07:25:11.179  ServiceReconciler: Added finalizer for eventhubns-emulatorhealth
07:25:11.382  ContainerReconciler: Added finalizer for eventhubns-fzkwxfba
07:25:11.389  ContainerReconciler: Container health status changed → Caution (both containers)
07:25:11.405  docker create eventhubns-fzkwxfba (eventhubs-emulator:2.2.0)
07:25:11.412  docker create eventhubns-storage-dabgemkt (azurite:3.35.0)
07:25:11.846  Container created: eventhubns-storage-dabgemkt (ContainerID: 36808127f358)
07:25:11.855  Container created: eventhubns-fzkwxfba (ContainerID: 2e64d16943b8)
07:25:11.856  docker cp config files to eventhubns-fzkwxfba
07:25:12.171  Files copied to /Eventhubs_Emulator/ConfigFiles
07:25:12.987  ContainerNetworkConnection added for eventhubns-fzkwxfba
07:25:13.039  Container not connected to expected number of networks (Expected:1, Connected:0)
07:25:13.141  Still waiting for network connection (Expected:1, Connected:0)
07:25:13.185  docker container start eventhubns-fzkwxfba (2e64d16943b8)
07:25:13.651  Container start completed
07:25:14.996  docker container start eventhubns-storage-dabgemkt (36808127f358)
07:25:15.310  Container start completed

              *** ~11 seconds gap - no health probe activity ***
              *** Test timeout fires at 07:25:08 + 20s = ~07:25:28 ***

07:25:26.724  Container eventhubns-fzkwxfba being deleted (CurrentState: Stopping)
07:25:26.748  Container eventhubns-storage-dabgemkt being deleted (CurrentState: Stopping)
07:25:31.595  Container eventhubns-fzkwxfba removed
07:25:32.955  HealthProbeSet: Shutting down health probe set...

Key observations from DCP logs:

1. The containers started at 07:25:13-15 but the health probe set was shut down at 07:25:32 without ever recording a successful health check
2. The HealthProbeSet only logged a shutdown message — no health probe attempts were logged, suggesting the health probes never had a chance to run before the test was cancelled
3. The emulator container took ~2s just for docker create + file copy + network setup before docker start
4. The 20s Polly timeout started ticking while DCP was still setting up containers and networks, leaving very little time for the emulator to initialize and pass health checks