CoreSimulator folder permissions issue on some Mac machines

[ivanpovazan]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=822026

Build leg reported

E2E Apple - Simulator Commands Helix Tests Build_Debug

Pull Request

dotnet/xharness#1282

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

{
  "ErrorPattern": "permission to save the file.*in the folder.*CoreSimulator",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

Build error leg or test failing:

• Failed job: E2E Apple - Simulator Commands Helix Tests Build_Debug
• Failed test: System.Numerics.Vectors.Tests

When testing simulator commands on some machines we get:

[08:21:48] dbug: Running /Applications/Xcode_14.3.app/Contents/Developer/usr/bin/simctl boot 4B1F47EA-7853-45DF-B459-37A3E6BCABF8
[08:21:49] dbug: An error was encountered processing the command (domain=NSCocoaErrorDomain, code=513):
[08:21:49] dbug: You don’t have permission to save the file “4B1F47EA-7853-45DF-B459-37A3E6BCABF8” in the folder “CoreSimulator”.
[08:21:49] dbug: You don’t have permission.
[08:21:49] dbug: To view or change permissions, select the item in the Finder and choose File > Get Info.
[08:21:49] dbug: Underlying error (domain=NSPOSIXErrorDomain, code=13):
[08:21:49] dbug: The operation couldn’t be completed. Permission denied
[08:21:49] dbug: Permission denied
[08:21:49] dbug: Process simctl exited with 1

Full log for context as it repeats the permission denied throughout the sequence: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-xharness-refs-pull-1282-merge-6620621afd364a37b1/System.Numerics.Vectors.Tests.Attempt.3/1/console.fd153acd.log?helixlogtype=result

Additional info

This was occurring before as reported in: #1878

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=822026
Error message validated: [permission to save the file.*in the folder.*CoreSimulator]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 10/1/2024 2:44:39 PM UTC

Report

Build	Definition	Test	Pull Request
1165776	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution
1164794	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1475
1163560	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1475
1158342	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1466
1158067	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution
1156453	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1465
1146463	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1469
1146322	dotnet/xharness	System.Numerics.Vectors.Tests.WorkItemExecution	dotnet/xharness#1468

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	3	8

[ivanpovazan]

This is happening on:

• dci-mac-build-250
• dci-mac-build-240
• dci-mac-build-225
• dci-mac-build-221
• dci-mac-build-220
• dci-mac-build-234
• dci-mac-build-224
• dci-mac-build-223
• dci-macm2-build-033
• dci-macm2-build-038
• dci-macm2-build-032

This is similar to #1878

@dotnet/dnceng can someone please log onto the machines from above and check/set the right folder permissions?

[dougbu]

full path should be /Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator

[ilyas1974]

@premun could you take a look at this and see if there is anything else that we need to do here? I know we have a work item (#5288) but that is for future deployments and should not affect existing systems

[premun]

Is it a safe operation to do chmod -R on this folder?

@ivanpovazan FYI there is a full passwordless sudo available so this is possible to be done via a Helix job - no manual logon needed.
We can do it quite quickly but I don't know if it won't break Xcode?

I understand you've done something similar as part of #5024? Do you know what exactly was done?

[premun]

It would also be good to understand how the permissions went wrong in the first place. Is it just wrong user?

[dellis1972]

I am seeing this issue on one of our internal pipelines which use helix.
I'm not sure if I should be using sudo to run the various xcrun simctl commands we have to start the simulator?
Is that the best work around for now?

[premun]

We never really run these commands manually but instead use XHarness. Did Xamarin/MAUI start using Helix recently?

I am curious if that would be the source of the recent problems - maybe things like running the provisionator or similar on the on-prem machines.

I am asking because I know MAUI have a much different approach to running iOS/Android workloads to Helix.

[dellis1972]

We can't use xharness. This is not for maui or xamarin. I'm adding mobile tests for the debugger. So I have to call those commands manually.

[premun]

Ah, okay. Running xcrun simctl is fine, I was just doublechecking that things like provisionator are not run. Only because I know some repo's workloads are targeting VMs mostly and always prepare the Xcode and all dependencies from ground up while Helix usually treats the on-prem machines more carefully as they are not VMs.

[premun]

I see the previous IcM ticket and I will reset the owner of these directories to helix-runner