slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2813742	dotnet-install-scripts	MacOSTests	Log
1171424	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1171379	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1171118	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1171078	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1171074	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1170856	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
2813498	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Release	Log
2813416	dotnet-sdk	TestBuild: macOS (x64)	Log	#54241
1170786	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119432
1170840	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1170844	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1170846	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2813264	dotnet-sdk	Darwin	Log	#54231
2813263	dotnet-sdk	Darwin	Log	#54233
2813262	dotnet-sdk	Darwin	Log	#54230
2813249	dotnet-sdk	AoT: macOS (x64)	Log	#54237
2813245	dotnet-sdk	AoT: macOS (x64)	Log	#54241
2813240	dotnet-sdk	AoT: macOS (x64)	Log	#54236
1170695	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#120118
1170679	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1170538	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1170515	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1170518	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1165389	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120396
1170211	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1170206	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1170109	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1169765	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119432
2812346	dotnet-sdk	TestBuild: macOS (x64)	Log	#54189
1169718	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1169629	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120409
1169598	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120408
1169618	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1169393	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1169161	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1169154	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1169005	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#120549
1168927	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#120556
1168895	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120542
2811530	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1168964	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51162
1168963	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#51155
1168953	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#51028
1168951	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#50996
2811381	dotnet-sdk	AoT: macOS (x64)	Log	#54089
2811373	dotnet-sdk	Darwin	Log	#53890
2811372	dotnet-sdk	Darwin	Log	#53891
2811371	dotnet-sdk	Darwin	Log	#53892
1167654	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120474
1168639	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51028
1168753	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1168663	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#120543
1168629	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#120546
1168691	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1168605	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#120545
1168591	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#120541
1168651	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#50898
1168646	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50920
1168644	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#50917
1168642	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#50973
1168640	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#50994
1168556	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2811141	dotnet-aspire	osx_arm64	Log
1168528	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120409
1168514	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120408
1168303	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1168288	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1168232	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
2810745	dotnet-sdk	Darwin	Log	#53892
2810744	dotnet-sdk	Darwin	Log	#53891
2810743	dotnet-sdk	Darwin	Log	#53890
2810691	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2810570	dotnet-sdk	TestBuild: macOS (x64)	Log	#54089
1167981	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1167982	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1167839	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#50946
2810536	dotnet-sdk	Darwin	Log	#53890
2810535	dotnet-sdk	Darwin	Log	#53891
2810534	dotnet-sdk	Darwin	Log	#53892
1167649	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#120473
1167590	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1167490	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119432
1167337	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1167325	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1167114	dotnet/runtime	osx-x64 Debug PALTests	Log
1167010	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#119432
1166894	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#120453
1166881	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1166867	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1166747	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2809549	dotnet-performance	Performance scenarios windows Win11 x64 Tiger 9.0	Log
1166704	dotnet/performance	Performance mlnet ubuntu 2204 x64 Open 9.0	Log
1166699	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51074
2809487	dotnet-install-scripts	MacOSTests	Log
1166440	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
1166434	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
2809298	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT osx x64 perfiphone12mini net10.0	Log
1166362	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51050
1166120	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono	Log
Displaying 100 of 507 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
10	105	507

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.