[Question]: Crashing Compile-AppInBcContainer/Publish-NavContainerApp in a PR pipeline - how to keep the container alive after crash for further investigation?

[KristofKlein]

Question

Hi,

I am about to add a test app to one of our PTE apps. In order to make this work I needed to get rid of "use CompilerFolder:true" and "doNotPublish: true".
So I can see now, the System is spinning a container, once it wants to compile it just crashes the alc.exe :

Compiling...
  .\alc.exe /project:"c:\sources\App1" /packagecachepath:"c:\sources\.packages" /out:"c:\sources\.output\Me_App1_13.0.2147483647.0.app" /SourceRepositoryUrl:"https://github.com/xxx" /SourceCommit:"12345678912345678912345678912345678969c0" /BuildBy:"AL-Go for GitHub,v6.0" /BuildUrl:"https://github.com/xxx/actions/runs/123456789" /assemblyprobingpaths:"C:\Program Files\dotnet\shared","c:\sources\App1\.netpackages","C:\ProgramData\BcContainerHelper\Extensions\app111681306045\.netPackages\Service"
  **Processing data from remote server app111681306045 failed with the following error message: An internal error occurred.** For more information, see the about_Remote_Troubleshooting Help topic.
  
  Exception Script Stack Trace:
  at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 61
  at Compile-AppInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Compile-AppInNavContainer.ps1: line 584
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 914
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2079
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1608
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1601
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057
  at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391
  at <ScriptBlock>, C:\4\_work\_temp\56037fb6-0832-4fcd-ab9b-5edea670ea97.ps1: line 3
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17
  at <ScriptBlock>, C:\4\_work\_temp\56037fb6-0832-4fcd-ab9b-5edea670ea97.ps1: line 2
  at <ScriptBlock>, <No file>: line 1
  
  PowerShell Call Stack:
  at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 71
  at Compile-AppInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Compile-AppInNavContainer.ps1: line 584
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 914
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 2079
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1608
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1601
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057
  at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391
  at <ScriptBlock>, C:\4\_work\_temp\[560](https://github.com/xxx/actions/runs/11681306045/job/32526115746#step:8:570)37fb6-0832-4fcd-ab9b-5edea670ea97.ps1: line 3
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17
  at <ScriptBlock>, C:\4\_work\_temp\56037fb6-0832-4fcd-ab9b-5edea670ea97.ps1: line 2
  at <ScriptBlock>, <No file>: line 1
  Compile-AppInBcContainer Telemetry Correlation Id: 3cb51780-521d-42f4-8c14-817af6ab1b49

This internal error is not visible anywhere to me, I do get - thanks for that, the container logs, but those are looking quite strange "out of the box".
(I have to flip to the Details tab and XML view and read the EventData there as the "General" tab shows quite often nonsense)

Strange as I am, I connected to the Build Agents machine, started a container with the same Image, and called Compile-AppInBcContainer manually from the prompt...and "surprise" that works just fine ...

IF I now reenable CompileInFolder, I actually pass the compiling in container issue , however, once it wants to install the armada of tests apps from MS, it just crashes there on publishing apps ( feels a bit like "out of memory" - here the Events in the container log shows Warnings , EventID 705 and 712 ). However, If I do this manually from the powershell, against my selfmade container it just works :/

The Agent was build with the current ARM template. I had a lot of noise and memory consumption from the "Antimalware Service Executable" (real time protection (turned it off)). I also resized the VM to something more memory friendly (B8ms 32gig). that helped me to get "a bit further" but still crashing while publishing test apps:

Importing test toolkit
  
  Synchronizing Permissions Mock on default
  App successfully synchronized
  Installing Permissions Mock on default
  App successfully installed
  Synchronizing Test Runner on default
  App successfully synchronized
  Installing Test Runner on default
  App successfully installed
  Synchronizing Any on default
  App successfully synchronized
  Installing Any on default
  App successfully installed
  Synchronizing Library Assert on default
  App successfully synchronized
  Installing Library Assert on default
  App successfully installed
  Skipping app 'C:\Applications.US\Microsoft_Permissions Mock_23.0.12034.12802.app' as it is already installed
  Synchronizing Library Variable Storage on default
  App successfully synchronized
  Installing Library Variable Storage on default
  App successfully installed
  Publishing C:\ProgramData\BcContainerHelper\Extensions\bcbase11680932926\4b7c9915-1e6b-4d1c-8085-39781cacf26e\Microsoft_System Application Test Library_23.0.12034.12802.app
  Synchronizing System Application Test Library on tenant default
  Installing System Application Test Library on tenant default
  App Microsoft_System Application Test Library_23.0.12034.12802.app successfully published
  Publishing C:\ProgramData\BcContainerHelper\Extensions\bcbase11680932926\900f8564-2626-4a4d-b738-f663bfef6fff\Microsoft_Tests-TestLibraries_23.0.12034.12802.app
  Processing data from remote server bcbase11680932926 failed with the following error message: An internal error occurred. For more information, see the about_Remote_Troubleshooting Help topic.
  
  Exception Script Stack Trace:
  at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 61
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 379
  at Publish-BcContainerApp, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Publish-NavContainerApp.ps1: line 154
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ObjectHandling\Import-TestToolkitToNavContainer.ps1: line 232
  at Import-TestToolkitToBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ObjectHandling\Import-TestToolkitToNavContainer.ps1: line 195
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 908
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1650
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1628
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1608
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1601
  at <ScriptBlock>, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1057
  at Run-AlPipeline, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Run-AlPipeline.ps1: line 1019
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\RunPipeline\RunPipeline.ps1: line 391
  at <ScriptBlock>, C:\4\_work\_temp\141486ba-896a-455c-b625-672f3e835ba5.ps1: line 3
  at <ScriptBlock>, C:\4\_work\_actions\microsoft\AL-Go-Actions\v6.0\Invoke-AlGoAction.ps1: line 17
  at <ScriptBlock>, C:\4\_work\_temp\141486ba-896a-455c-b625-672f3e835ba5.ps1: line 2
  at <ScriptBlock>, <No file>: line 1

So, I am a bit puzzled why the pipeline container just crashes the alc.exe (and can't handle the import) while basically the same flow from powershell works. Could this be a permission/security thing? I mean, powershell runs with the vmadmin, while pipelines are executed by a windows service. (still, why would that hit the alc.exe within a container?)

Can I keep the container that the pipeline creates? I recall a parameter ... maybe?
Also some other hint to get this working would be fine :) (I think I will try to make use of another host VM...just to see what will happen.)

I know, this is not made for on Prem, but still the import of MS TestApps should just work, it has not even started to publish my apps. Something is fishy with this container ....

some more details: Al.Go: v6.0
PTE Project "for OnPrem" BC v23
container isolation process
as long as no container/testApp is involved, I can compile and deploy to hosted environments.
Agent runs in a VM on Azure made with the ARM template
can Repro? Yes - can I share? no - at least not public :/

[KristofKlein]

So I have redone my build agent machine, the problem persists.

Error: Unexpected error when running action. Error Message: Processing data from remote server bcbase11955127532 failed with the following error message: An internal error occurred. For more information, see the about_Remote_Troubleshooting Help topic., StackTrace: at Invoke-ScriptInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\ContainerHandling\Invoke-ScriptInNavContainer.ps1: line 113 <- at Compile-AppInBcContainer, C:\ProgramData\BcContainerHelper\6.0.28\BcContainerHelper\AppHandling\Compile-AppInNavContainer.ps1: line 280 <- at <ScriptBlock>,

If I check line 280 - this is the Invoke that should handle the alc.zip

Not sure this is a good idea, but as the bccontainer folder gets mounted, I will try to get some more output from that script - so I will change it a bit :)

[freddydk]

Sorry, didn't see this - could you include the full log?
I can probably see the problem from that.

[KristofKlein]

all good - what confuses me is this $tempZip is a path to a file (alc.zip), the Copy-Item uses this as Destination?!
Can I mail you the log ?

ah now!
so you just build a "temp" incl. alc.zip as the name such that expand can handle it!

[KristofKlein]

when I add some -verbose to this currently I see:
it will copy the vsix found in the run folder to -> C:\Users\winrm\AppData\Local\Temp\alc.zip

than it wants to extract:

VERBOSE: Performing the operation "Create Directory" on target "Destination: C:\build\vsix".
VERBOSE: Preparing to expand...
Exception calling "ShouldProcess" with "1" argument(s): "Object reference not set to an instance of an object."
Exception Script Stack Trace:
at Expand-Archive, C:\program files\powershell\7\Modules\Microsoft.PowerShell.Archive\Microsoft.PowerShell.Archive.psm1: line 375
at , : line 11
PowerShell Call Stack:

is this some winrm, process isolation thing, pwsh7 thing?

[KristofKlein]

this looks like it is getting somewhere: I added $bcContainerHelperConfig.useWinRmSession = "never" to my config, and now the compiler folder was made and also the compilation process with alc.exe started to execute!

what would be the best spot to place this setup in AL:GO if it is really fixing my issues?

"fingers crossed"

[KristofKlein]

that helped! without winrm the process runs. So looks like host to container have issues to keep the connection alive...

[KristofKlein]

so @freddydk - this winrm is the thing that makes it go boom. All runs I triggered now without it worked flawless.

About the build agent: it is based on the Create a self hosted runner runner document. Right now the basic D4x, choco, and the installation steps mentioned in the howto.
Agent setup based on GitHubs howto + the things from the ARM template script
( I also tried to create myself a template but that didn't worked so well :D - I think there is a mistake in the description text about how to setup the URL you should make use of [https://freddyk.azurewebsites.net/api/... ] )

To verify this once more: what is the most appropriate place for the setup to tell the whole thing: make not use of winrm?

[freddydk]

You can add "useWinRmSession": "never" to .github/AL-Go-Settings.json

but... - I think the problem might be something else - if you could include the full log, I might be able to spot the problem.

Also, which user is running the agent on the runner?

[KristofKlein]

this is what the agent uses as Service Account: (i think I got this from the ARM template when it start the agents)

If I can mail you the log rather sharing it here, that would be awesome :) You know - Company rules... It also sounds that you might have a spot you are very interested in - I can also go and cut that for you ;)

[freddydk]

System is fine - please email a link to where i can download the log - I cannot say a specific thing - there is a ton of info in the log

[SvenPouwelsBli]

Did anything ever come out of this? We're currently seeing the same error in multiple repositories. The WinRM setting does seem to fix it, but I can't imagine that's the actual fix.

[freddydk]

The real fix is that BcContainerHelper itself figures out that it shouldn't use WinRM - will add that to the list of things to get done

[freddydk]

Looking at this.
The strange thing here is that WinRM works fine for a number of calls (when looking at the logs) - but suddenly, it stops working during the compile. I would expect that if WinRM doesn't work, it wouldn't work at all - but that's not the case.
Will investigate a bit more and maybe change the default of WinRM to never - then people can change it if needed.

[KristofKlein]

what is the benefit of WinRM? quicker?

[freddydk]

Much - ms vs. sec