System.OutOfMemoryException error when running the visio scanner tool

[hsugianto]

We are running into a System.OutOfMemoryException error when running the SharePoint Visio Scanner Tool ( https://github.com/pnp/PnP-Tools/tree/master/Solutions/SharePoint.Visio.Scanner). We have attempted to run on the servers where it has larger memory (32gb), running against smaller set of sites, and reducing the threads parameter from the default of 10 to 5 with the same error. Have anyone run into similar behavior? Any pointer that you can provide would be much appreciated.

Full error information:

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType)
at Microsoft.SharePoint.Client.JsonReader.ReadKeyValue()
at Microsoft.SharePoint.Client.JsonReader.ReadDictionary()
at Microsoft.SharePoint.Client.JsonReader.ReadDictionaryT

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.SharePoint.Client.JsonReader.ReadT
at Microsoft.SharePoint.Client.JsonReader.ReadListT
at Microsoft.SharePoint.Client.Search.Query.ResultTable.InitOnePropertyFromJson(String peekedName, JsonReader reader)
at Microsoft.SharePoint.Client.ClientValueObject.FromJson(JsonReader reader)
at Microsoft.SharePoint.Client.JsonReader.ReadJsonObject(Type fallbackType)
at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType)
at Microsoft.SharePoint.Client.JsonReader.ReadT
at Microsoft.SharePoint.Client.JsonReader.ReadListT
at Microsoft.SharePoint.Client.ClientValueObjectCollection1.InitOnePropertyFromJson(String peekedName, JsonReader reader) at Microsoft.SharePoint.Client.Search.Query.ResultTableCollection.InitOnePropertyFromJson(String peekedName, JsonReader reader) at Microsoft.SharePoint.Client.ClientValueObject.FromJson(JsonReader reader) at Microsoft.SharePoint.Client.JsonReader.ReadJsonObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.ReadObject(Type fallbackType) at Microsoft.SharePoint.Client.JsonReader.Read[T]() at Microsoft.SharePoint.Client.ClientResult1.Microsoft.SharePoint.Client.IFromJson.FromJson(JsonReader reader)
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
at Microsoft.SharePoint.Client.ClientRequest.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SharePoint.Client.ClientRequest.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SharePoint.Client.ClientRuntimeContext.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SharePoint.Client.ClientContext.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SharePoint.Client.ClientContextExtensions.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount, Int32 delay, String userAgent)
at SharePoint.Scanning.Framework.ScanJob.ProcessQuery(Web web, String keywordQueryValue, List1 propertiesToRetrieve, List1 sites, KeywordQuery keywordQuery) in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 381
at SharePoint.Scanning.Framework.ScanJob.Search(Web web, String keywordQueryValue, List1 propertiesToRetrieve, Boolean trimDuplicates) in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 359 at SharePoint.Visio.Scanner.VisioScanJob.ResolveAddedSites(List1 addedSites)
at OfficeDevPnP.Core.Framework.TimerJobs.TimerJob.Run()
at SharePoint.Scanning.Framework.ScanJob.Execute() in C:\github\BertPnPTools\Solutions\SharePoint.Scanning\SharePoint.Scanning.Framework\ScanJob.cs:line 118
at SharePoint.Visio.Scanner.VisioScanJob.Execute()
at SharePoint.Visio.Scanner.Program.Main(String[] args)

[cshin80]

Mine ran for around 12 days on a server, then got the memory exception, but without the extensive stack trace, followed by several timout errors before freezing out.

[hsugianto]

Mine ran for around 12 days on a server, then got the memory exception, but without the extensive stack trace, followed by several timout errors before freezing out.

Thank you for the comment and providing your experience running the tool. Our Microsoft DSE had reached out to the author of the visio scanner tool and learned that it's an older open source tool that is no longer maintained with the suggestion to recompile it to 64bit version, limit sites with -r parameter, and possibly exclude the search piece to alleviate the performance overhead. I don't think the tool is meant for scanning a large tenant as I was able to successfully running it in our POC with only about 500 sites without issue.

In the meantime and as the workaround, we use search's graph API to find and search the .vsdw file extension. I hope that helps for your scenario as well.