feat(crashdump): Add crash dump functionality for fatal errors #1594

slurmlord · 2025-09-18T09:33:24Z

Merge after bugfix(system): Prevent AMD/ATI driver crash on game launch by front loading the system dbghelp.dll #1066
Merge after fix(debug): Implement reference counter and thread safety in DbgHelpLoader to allow for independent users #1804

This change introduces the option to generate crash dumps, aka. mini dumps, on fatal errors.
The main minidump functionality is done by explicitly loading the dbghelp.dll from the system directory, as the dbghelp.dll that is bundled with the game is an older version that does not include this functionality. There is an option to create small dumps or extended info dumps, currently both are created.

Small dumps

These mostly contain stacks for the process threads and some stack variables, or to create dumps with extended info. The use case for these is to quickly determine where a crash occured, the type of crash, if it was already fixed etc. In addition, if the memory allocation structures are corrupted enough, an extended info dump might not succeed while the small dump should. The size of these dumps are typically on the order of 250kB.

Extended info

These contain global values, along with the memory regions allocated via the memory pool factory and the dynamic memory allocator. This makes all in-game objects available to the person debugging the crash dump, so for example dt generalszh!TheWritableGlobalData in WinDbg will show the state at the time the dump was created.

An alternative option could be to not traverse the memory structures "manually" to get to the allocations and instead just specify the MiniDumpWithFullMemory flag to MiniDumpWriteDump, but that increases the file size considerably.

As an example, dump of the generalszh process in the main menu with the shell map in the background yields a ~140MB dump when traversing and ~420MB with MiniDumpWithFullMemory. Beyond that, the ~140MB file compresses to ~20MB with 7Z, so should be relatively easily transferable.

Storage Location

Crash dumps are stored in a new folder called 'CrashDumps' under the userDir ("Documents\Command and Conquer Generals Zero Hour Data"), and on startup it will create this directory if it doesn't exist and delete any older dumps so only the 10 newest small and 2 newest extended info dumps are left. This is to preserve disk space, as the extended info files can be several hundred MB.

Integration points

For VS2022 builds, unhandled exceptions end up in the UnhandledExceptionFilter in WinMain, which then get a reference to the actual exception that occurred and includes that in the dump.
For VC6 builds, unhandled exceptions are caught in the catch(...) blocks of GameEngine::execute which then calls RELEASE_CRASH. As there is no exception data available in this case to populate _EXCEPTION_POINTERS from, an intentional exception is triggered to get the trace of the current thread. This makes the stack traces for VC6 a bit more cryptic than VS2022 builds as the C++ exception handling gets included in the trace.

Limitations

In the longer run we'll probably want to replace this code with a more mature solution, like CrashPad, but that currently depends on a newer compiler than VC6.
As the code is intended to be temporary, it's kept behind a new CMake feature so it can be easily removed. There are also some other decisions made with this in mind:

Minidump is created in-process. Ideally, the dump should be performed by a process external to the crashing/failing process, but to avoid having to ship an extra binary, in-process was chosen instead. It's being performed in a separate thread to hopefully have a clean stack to work with.
Depends on RTS_BUILD_OPTION_VC6_FULL_DEBUG for VC6 builds. The PDBs generated with the default VC6 compile options are lacking in information, making the mini dumps less useful. The option RTS_BUILD_OPTION_VC6_FULL_DEBUG should be enabled for VC6 builds to ensure maximum usability. VS2022 builds produce better PDBs and require no extra options.
Directory management code is contained within the MiniDumper class, not re-usable by other components.
Many Win32-specific types and functions are used directly without regards for portability.
As the MiniDump feature is not available for VC6, a lot of headers have been borrowed from minidumpapiset.h and included in the MiniDumper_compat.h file.
Globals are used for storing the current exception info.
Only enabled for the games, tools are currently not included.

OmniBlade

Looks good, tested dump generation with a 2022 build.

slurmlord · 2025-10-09T12:41:24Z

Another approach I realized after publishing could be to move the GameMemory allocations from using GlobalAlloc to instead create a separate GameMemory heap and use HeapAlloc.
The benefit would be a lot less code required to traverse the allocations for inclusions the dump, as it could be done with HeapWalk instead for only the GameMemory heap.

xezon

Looks good overall. Just a bunch of small comments.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Source/Common/System/Debug.cpp

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/GameMemory.h

Core/GameEngine/Source/Common/System/GameMemory.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

slurmlord · 2025-10-14T19:54:04Z

Pushed a commit attempting to address most of the comments in the previous review:

Made TheMiniDumper a pointer, declared in MiniDumper.cpp
Moved extern TheMiniDumper to MiniDumper.h
Moved MiniDumper ctor body to MiniDumper.cpp
(Attempted to) find and correct all the stylistically misplaced "{"
Added enum for dump type
Fall back to DUMP_TYPE_FULL when DUMP_TYPE_GAMEMEMORY is requested and game memory implementation is turned off
Removed cmake feature dependency on RTS_GAMEMEMORY_ENABLE
Removed thread Id from dump file name
Replace hard-coded list of executable names to match with GetModuleFileNameW
(Attempted to) clean up log format
Added Enum values for dump thread exit codes
Removed MiniDumper::m_endRangeIter

In addition:

Added new static init/shutdown methods, using placement new on the process heap
Prefixed win32 api function calls with global namespace ::
Added asserts for dumping thread active state

Remaining:
Merge with #1066 when merged.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Skyaero42 · 2025-11-01T12:45:39Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

slurmlord · 2025-11-04T19:48:09Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

Agreed, that's the main use case I was hoping this could address. Once this is merged, the remaining hurdle would be to get the weekly builds to enable the Vc6FullDebug cmake option.

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

xezon

I think the cleanup in MiniDumper class needs another look. Make it simple and robust.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

cmake/config-memory.cmake

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon

Code looks good to me. A few minor comments that could be looked into.

Core/GameEngine/Source/Common/System/Debug.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Source/Common/System/Debug.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Caball009 · 2025-11-21T20:16:54Z

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader_minidump.h

+	MiniDumpFilterWriteCombinedMemory = 0x01000000,
+	MiniDumpValidTypeFlags = 0x01ffffff,
+	MiniDumpNoIgnoreInaccessibleMemory = 0x02000000,
+	MiniDumpValidTypeFlagsEx = 0x03ffffff,


Is this value correct?

From https://learn.microsoft.com/en-us/windows/win32/api/minidumpapiset/ne-minidumpapiset-minidump_type I'd expect this to be 0x02000001.

There does seem to be some discrepancy between the doc and the SDK headers. The SDK headers defines it explicitly as 0x03ffffff, you can see this in the original file at C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um\minidumpapiset.h
My money would be on the docs being wrong, and as we don't use this value directly it should be fine for our purpose. Does that seem reasonable?
And just out of curiosity, how did you notice this?

Include\10.0.26100.0\

I only have two older versions and MiniDumpValidTypeFlags is the last value for both versions.

Does that seem reasonable?

Yeah, seems reasonable to me.

And just out of curiosity, how did you notice this?

I was wondering why we're defining all these structs / enums, so I happened to check a couple or maybe just this one to see where they were coming from.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

EndsWithCaseInsensitive needs looking

slurmlord · 2025-11-26T18:35:12Z

Rebased on main.

Added use of endsWithNoCase from refactor(string): Implement generic startsWith, endsWith functions for C strings #1898 in MiniDumper::ShouldWriteDataSegsForModule.
Reordered wanted_modules in MiniDumper::ShouldWriteDataSegsForModule to static constexpr const.

xezon · 2025-11-26T19:06:13Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+	case ThreadCallback:
+		retVal = TRUE;
+		break;
+	case ThreadExCallback:


Can merge ThreadCallback, ThreadCallback, ThreadExCallback cases.

xezon · 2025-11-26T19:07:44Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+	switch (input.CallbackType)
+	{
+	case IncludeModuleCallback:
+		retVal = TRUE;


retVal already starts as TRUE so this is not necessary.

xezon · 2025-11-26T19:09:08Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+// This is where the memory regions and things are being filtered
+BOOL MiniDumper::CallbackInternal(const MINIDUMP_CALLBACK_INPUT& input, MINIDUMP_CALLBACK_OUTPUT& output)
+{
+	BOOL retVal = TRUE;


What does the retVal signal to the caller? Perhaps give it a name that describes it, because it is not obvious what return TRUE and FALSE will do now.

This one is a bit of a mess - it's returned in the minidump callback and its meaning varies between callback types.
The documentation for the callback function says If the function succeeds, return TRUE; otherwise, return FALSE., but the individual callback types sometimes uses it to indicate if something should be included or not, if an enumeration should continue, etc. See doc for reference.
I tried coming up with a fitting name earlier, but didn't found something that seemed right. Suggestions welcome :)

Hmm that sucks. Maybe ask Chat for name idea.

Reading through the docs, it seems like we can always return TRUE and still convey the same exclusions and conditions via the output parameter.
Ended up renaming it to success, which is always TRUE.
Wasn't sure if declaring it as const BOOL success = TRUE; was too bold ;)

xezon · 2025-11-26T19:20:03Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		DynamicMemoryAllocator* allocator = TheDynamicMemoryAllocator;
+
+		//m_dumpObjectsSubState is used to track the index of the allocator we are currently traversing
+		for (int i = 0; i < m_dumpObjectsSubState; ++i)


Is this loop expensive? Does m_dumpObjectsSubState become large? If yes, could cache allocator maybe.

Cached it in a new class member and repurposed the m_dumpObjectsSubState variable to track the index of the raw block in the current DMA.

slurmlord · 2025-11-27T17:50:57Z

Refactored a bit, introducing an enum for the memory range phases and modified the output variable of the callback routine so that our callback can always return TRUE.

xezon

To me it looks like we can get rid of the concept of m_dumpObjectsSubState by caching the actual pointers to memory structures and keep iterating until they reach NULL and then go to the next until all lists reached the end.

Performance wise it is probably not the biggest of deals but it should make the logic cleaner.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon · 2025-11-27T19:44:31Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		{
+			// DumpMemoryObjects will set outputMemorySize to 0 once it's completed, signalling the end of memory callbacks
+			DumpMemoryObjects(output.MemoryBase, output.MemorySize);
+		} while ((output.MemoryBase == 0 || output.MemorySize == 0) && m_dumpObjectsState != COMPLETED);


Are all these parts of the condition required and robust?

So for example it is legal to allocate a 0 size array and get a pointer for it, in which case one could technically track a pointer with zero size.

It's valid programming wise, but not as a callback-value. Returning a region with size 0 means "done with memory regions", so no more MemoryCallbacks will be sent after that.
That's why, when transitioning between the phases in DumpMemoryObjects we need to call it again if output.MemorySize was 0 so we don't stop getting MemoryCallbacks. The check for COMPLETED is so we know when we should actually return the 0 sized region.
Checking MemoryBase is not needed, I'll remove that.

xezon · 2025-11-27T19:51:38Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		//m_dumpObjectsSubState contains the index in TheMemoryPoolFactory of the MemoryPool that is being processed
+		if (m_dumpObjectsSubState < poolCount)
+		{
+			MemoryPool* pool = TheMemoryPoolFactory->getMemoryPoolN(m_dumpObjectsSubState);


getMemoryPoolN also has a loop inside.

Can we perhaps cache the MemoryPool* pointer and just keep going to next one until it is NULL, without using getMemoryPoolCount, getMemoryPoolN?

xezon · 2025-11-27T19:53:40Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		if (m_dumpObjectsSubState == 0)
+		{
+			m_rangeIter = TheMemoryPoolFactory->cbegin();
+			++m_dumpObjectsSubState;


Perhaps can get rid of m_dumpObjectsSubState by simply initializing m_rangeIter = TheMemoryPoolFactory->cbegin() before this callback?

xezon · 2025-11-27T19:56:12Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		if (m_dumpObjectsSubState < rawBlocksInDma)
+		{
+			// Dump this block
+			m_currentAllocator->fillAllocationRangeForRawBlockN(m_dumpObjectsSubState, rawBlockRange);


fillAllocationRangeForRawBlockN also has a loop inside.

Can we perhaps cache the MemoryPoolSingleBlock* pointer and just keep going to next one until it is NULL, without using getRawBlockCount, fillAllocationRangeForRawBlockN?

slurmlord · 2025-11-27T22:36:01Z

Got rid of the m_dumpObjectsSubState and cached the various objects that's being iterated. This also reduced the changes required in GameMemory. Looks better!

xezon

This looks much cleaner indeed.

xezon · 2025-11-27T22:42:56Z

Core/GameEngine/Include/Common/MiniDumper.h


 enum DumpObjectsState CPP_11(: Int)
 {
+	BEGIN,


I suggest use DumpObjectsState_Begin enum name style to avoid potential conflicts with MACRO_CASE.

xezon · 2025-11-27T22:43:59Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 			// DumpMemoryObjects will set outputMemorySize to 0 once it's completed, signalling the end of memory callbacks
 			DumpMemoryObjects(output.MemoryBase, output.MemorySize);
-		} while ((output.MemoryBase == 0 || output.MemorySize == 0) && m_dumpObjectsState != COMPLETED);
+		} while (output.MemorySize == 0 && m_dumpObjectsState != COMPLETED);


Shouldn't this be != 0 then?

xezon · 2025-11-27T22:44:22Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 #ifndef DISABLE_GAMEMEMORY
 		do
 		{
 			// DumpMemoryObjects will set outputMemorySize to 0 once it's completed, signalling the end of memory callbacks


"memory size"

xezon · 2025-11-27T22:46:10Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+			if (TheMemoryPoolFactory)
 			{
-				m_dumpObjectsSubState = poolCount;
+				m_rangeIter = TheMemoryPoolFactory->cbegin();


Maybe also move that to BEGIN state?

xezon · 2025-11-27T22:47:30Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 		{
 			m_dumpObjectsState = DMA_ALLOCATIONS;
+			m_currentAllocator = TheDynamicMemoryAllocator;
+			m_currentSingleBlock = m_currentAllocator->getFirstRawBlock();


Maybe also move this to BEGIN state?

xezon · 2025-11-27T22:48:13Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 		{
 			m_dumpObjectsState = DMA_ALLOCATIONS;
+			m_currentAllocator = TheDynamicMemoryAllocator;
+			m_currentSingleBlock = m_currentAllocator->getFirstRawBlock();


Test m_currentAllocator or TheDynamicMemoryAllocator before dereference incase TheDynamicMemoryAllocator was null?

xezon · 2025-11-27T22:51:13Z

Core/GameEngine/Include/Common/MiniDumper.h

+	DUMP_TYPE_FULL = 'F',
+};
+
+enum MiniDumperExitCode CPP_11(: Int)


Can this enum be private in MiniDumper class?

xezon · 2025-11-27T22:51:37Z

Core/GameEngine/Include/Common/MiniDumper.h

+	DUMPER_EXIT_FORCED_TERMINATE = 0x158B1154,
+};
+
+enum DumpObjectsState CPP_11(: Int)


Can this enum be private in MiniDumper class?

xezon · 2025-11-27T22:56:12Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+
+		if (m_currentSingleBlock == NULL)
+		{
+			m_currentAllocator = m_currentAllocator->getNextDmaInList();


Add

if (m_currentAllocator) m_currentSingleBlock = m_currentAllocator->getFirstRawBlock();

xezon · 2025-11-27T22:57:34Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+		if (m_currentSingleBlock == NULL)
+		{
+			// Iterated to a new allocator, start iterating over its blocks
+			m_currentSingleBlock = m_currentAllocator->getFirstRawBlock();


Remove this.

OmniBlade approved these changes Sep 23, 2025

View reviewed changes

xezon added Major Severity: Minor < Major < Critical < Blocker Gen Relates to Generals ZH Relates to Zero Hour Debug Is mostly debug functionality System Is Systems related labels Oct 8, 2025

xezon reviewed Oct 9, 2025

View reviewed changes

xezon reviewed Oct 16, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/MiniDumper.cpp Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

xezon added the Approved Pull Request was approved label Oct 21, 2025

slurmlord force-pushed the feature-crashdump branch from 7bb8e8c to 90a0b49 Compare November 4, 2025 19:43

slurmlord mentioned this pull request Nov 5, 2025

build(weekly): Enable complete debug symbols for VC6 weekly builds #1803

Merged

xezon reviewed Nov 5, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

slurmlord commented Nov 12, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Show resolved Hide resolved

xezon reviewed Nov 16, 2025

View reviewed changes

xezon previously approved these changes Nov 19, 2025

View reviewed changes

xezon reviewed Nov 21, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/Debug.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/Debug.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Caball009 reviewed Nov 21, 2025

View reviewed changes

xezon reviewed Nov 22, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

slurmlord mentioned this pull request Nov 23, 2025

refactor(string): Implement generic startsWith, endsWith functions for C strings #1898

Merged

feat(crashdump): Add crash dump functionality

a241b80

slurmlord force-pushed the feature-crashdump branch from 9cbf93d to a241b80 Compare November 26, 2025 18:31

xezon reviewed Nov 26, 2025

View reviewed changes

Refactor CallbackInternal and DumpMemoryObjects

5e543ba

xezon reviewed Nov 27, 2025

View reviewed changes

Improve iteration in DumpMemoryObjects

7399cce

xezon reviewed Nov 27, 2025

View reviewed changes

feat(crashdump): Add crash dump functionality for fatal errors #1594

Are you sure you want to change the base?

feat(crashdump): Add crash dump functionality for fatal errors #1594

Uh oh!

Conversation

slurmlord commented Sep 18, 2025 • edited by xezon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Small dumps

Extended info

Storage Location

Integration points

Limitations

Uh oh!

OmniBlade left a comment

Choose a reason for hiding this comment

Uh oh!

slurmlord commented Oct 9, 2025

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Skyaero42 commented Nov 1, 2025

Uh oh!

slurmlord commented Nov 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

slurmlord commented Sep 18, 2025 •

edited by xezon

Loading