The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2807312	dotnet-source-indexer	🟣Clone Stage1 data	Log
2807289	dotnet-machinelearning	Install build dependencies	Log
2807264	dotnet-machinelearning	Build Packages	Log
1164306	dotnet/runtime	Build	Log	dotnet/runtime#119432
2807210	dotnet-machinelearning	Build Packages	Log
2807240	dotnet-machinelearning	Install build dependencies	Log
1164127	dotnet/runtime	Send to Helix	Log
2807165	dotnet-machinelearning	Install build dependencies	Log
1164108	dotnet/aspire	Run helix tests	Log
2807049	dotnet-dotnet	Build (centOSStreamContainer)	Log
1163568	dotnet/runtime	Send tests to Helix (Unix)	Log
1163981	dotnet/runtime	Send job to Helix (Unix)	Log
2806982	dotnet-runtime	Run performance job script	Log
1163569	dotnet/runtime	Send to Helix	Log
1163702	dotnet/roslyn	Test	Log	dotnet/roslyn#80540
1163693	dotnet/roslyn	Run Unit Tests	Log
1163518	dotnet/runtime	Send to Helix	Log
1163632	dotnet/runtime	Send job to Helix (Unix)	Log
1163589	dotnet/runtime	Send to Helix	Log
1163592	dotnet/runtime	Send to Helix	Log
1163457	dotnet/runtime	AOT compile CoreCLR tests	Log
1163362	dotnet/aspire	Run helix tests	Log
1163356	dotnet/aspire	Run helix tests	Log
1163281	dotnet/roslyn	Run Unit Tests	Log
2806495	dotnet-aspire	🟣Restore	Log
2806428	dotnet-aspire	🛡 Guardian: TSAUpload	Log
1162998	dotnet/runtime	Send to Helix	Log
2806379	dotnet-aspire	🔒 Start Network Isolation	Log
2806402	dotnet-roslyn	Run Unit Tests	Log
2806328	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2806276	dotnet-runtime	🛡 Generate SBoM Manifest	Log
2806337	dotnet-runtime	Send job to Helix (Unix)	Log
2806281	dotnet-dotnet	Run Binary Analysis Scan Optimization (Expand Archives) (1ES PT)	Log
1162951	dotnet/runtime	Send tests to Helix (Unix)	Log
2806164	dotnet-efcore	Send job to helix	Log	#53860
1162957	dotnet/runtime	Send to Helix	Log
2806222	dotnet-dotnet	Run Binary Analysis Scan Optimization (Expand Archives) (1ES PT)	Log
1162837	dotnet/runtime	Build product	Log	dotnet/runtime#119432
2806219	dotnet-dotnet-monitor	Build (build_linux_musl_arm64_cross)	Log	#53919
1162838	dotnet/runtime	Build product	Log	dotnet/runtime#119432
2806140	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2805896	dotnet-efcore	Send job to helix	Log
1162385	dotnet/runtime	Send tests to Helix (Unix)	Log
2806156	dotnet-aspnetcore	🛡 Guardian: ESLint (TypeScript)	Log	#53905
2806098	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2806084	dotnet-runtime	Send job to Helix (Unix)	Log
1162306	dotnet/runtime	Send tests to Helix (Unix)	Log
2806045	dotnet-runtime	Send job to Helix (Unix)	Log
1162525	dotnet/runtime	Send to Helix	Log
1162436	dotnet/runtime	Send to Helix	Log
1162526	dotnet/runtime	Send to Helix	Log
2806012	dotnet-efcore	Send job to helix	Log
1162509	dotnet/performance	Send job to Helix (Windows)	Log	dotnet/performance#4982
1162483	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51063
2805923	dotnet-runtime	Send job to Helix (Windows)	Log
1162335	dotnet/runtime	Build	Log	dotnet/runtime#119432
1162351	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115502
2805895	dotnet-efcore	Send job to helix	Log
2805894	dotnet-efcore	Send job to helix	Log
1162334	dotnet/runtime	Build product	Log	dotnet/runtime#119432
2805744	dotnet-sdk	Run Tests in Helix	Log	#53890
2805798	dotnet-efcore	Send job to helix	Log	#53860
2805789	dotnet-roslyn	Test	Log
2805745	dotnet-sdk	Run Tests in Helix	Log	#53891
2805746	dotnet-sdk	Run Tests in Helix	Log	#53892
2805661	dotnet-performance	Send job to Helix (Windows)	Log
1162155	dotnet/performance	Send job to Helix (Windows)	Log
1162158	maestro-auth-test/helix-queue-insights-test	Run Tests (Windows)	Log
1162125	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1162004	dotnet/runtime	Send to Helix	Log
1162121	dotnet/roslyn	Test	Log	dotnet/roslyn#80498
1162082	dotnet/runtime	Send to Helix	Log
2805647	dotnet-sdk	Checkout dotnet-sdk@refs/pull/53883/merge to s	Log	#53883
1159634	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115502
1161893	dotnet/runtime	Send tests to Helix (Unix)	Log
2805540	dotnet-arcade-validation	Run Tests (Windows)	Log
1161894	dotnet/runtime	Send tests to Helix (Unix)	Log
1161700	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#119803
2805431	dotnet-dotnet	🛡 Guardian: Post Analysis	Log
2805448	dotnet-aspnetcore	Run build.cmd helix target	Log	#53878
1161866	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#63758
2805435	dotnet-performance	Send job to Helix (Windows)	Log
1161833	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120266
2805417	dotnet-aspnetcore	Run build.cmd	Log
2805382	dotnet-aspnetcore	Build x64	Log
2805344	dotnet-runtime	Send job to Helix (Unix)	Log
2805346	dotnet-dotnet-monitor	NuGetAuthenticate@1	Log
1161252	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120239
2805272	dotnet-dotnet	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2805262	dotnet-dotnet	Prepare BuildLogs staging directory	Log
1161562	dotnet/runtime	Send to Helix	Log	dotnet/runtime#120249
1161578	dotnet/aspnetcore	Run build.cmd helix target	Log
2805220	dotnet-performance	Send job to Helix (Windows)	Log
1161581	dotnet/performance	Send job to Helix (Windows)	Log
2805180	dotnet-dotnet	Build (centOSStreamContainer)	Log
1161555	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119815
1161538	dotnet/runtime	Send to Helix	Log
1161549	dotnet/runtime	Send to Helix	Log
1161545	dotnet/runtime	Send to Helix	Log
1161477	dotnet/runtime	Send to Helix	Log
Displaying 100 of 1178 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
22	202	1178

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez