Memory exhaustion during data exports in pillars that rely on the User export

[jamesmor]

See also (#1236 (comment))

In running the assessment in my tenant with ~1.7M objects, 1.1M of which are users, I have run into a problem where all available memory is exhausted usually resulting in the entire system being hung. As a result, I've not been able to create PSF packages for this issue.

I've run the assessment on devices or VMs with up 128GB and 16 vCPUs and continue to see the problem in version 2.3.0. Version 2.4.0 provided by @kodevza improved the memory usage early on in exports, but eventually the system still hung. The user export is where I've typically seen the memory exhaustion.

[kodevza]

@jamesmor Just to clarify: version 2.4.0 was not an official upstream release. It was a test build from my fork, prepared to validate memory-related changes.

Based on my analysis, the main pressure in your tenant is likely the very large user population, plus the high number of user-to-role assignments that need to be exported and processed. With ~1.1M users, the user export can still exhaust memory even if the earlier service principal/application export path behaves better.

I have prepared another version that specifically targets the memory problem in the user export path:

https://github.com/kodevza/zerotrustassessment/releases/tag/v2.4.99-memory_leak_gc

Could you please test this version and run the assessment again? If it still hangs, the most useful next signal would be where memory starts growing fastest: user export, group membership export, or report generation.

Changes: kodevza@866ea01

[jamesmor]

2.4.99-memory_leak_gc got to over 78GB in use before the assessment errored out, so the memory usage is definitely improving