GS:HW: Reduce primid max value and add primid dumping.

[TJnotJT]

Description of Changes

Reduce the maximum value of the primid texture to 0xFFFFFF (~ 17 million).

Add dumping of primid texture for debugging when RT dumping is enabled.

Rationale behind Changes

Fixes #14818 on Mesa OpenGL. Although that's the only driver known to have an issue with large primid values, 0xFFFFFF primitives should be large enough for all use cases (we limit draws to roughly 0xFFFF vertices).

Suggested Testing Steps

Follow the steps in the linked issue.

Enable RT dumping in Settings>Debug>GS and run a game/dump that uses primid DATE. The primid textures should be dumped with names like 00045_f00000_primid.png.

Did you use AI to help find, test, or implement this issue or feature?

No.

[TellowKrinkle]

What's the use for primid dumping? I would have expected this to be better debugged in renderdoc, since it has nothing to do with what the game is doing and is entirely something the emulator is doing for itself. But if you do include it, I would much prefer dumping raw floats, not floats converted to ints.

[TJnotJT]

What's the use for primid dumping? I would have expected this to be better debugged in renderdoc, since it has nothing to do with what the game is doing and is entirely something the emulator is doing for itself.

OGL renderdoc debugging is a bit limited on my system (e.g. no viewing of textures) so I had to resort to dumping the primid textures. Iirc, the problem draw was failing all fragments incorrectly because the primid texture had all negative values.

But if you do include it, I would much prefer dumping raw floats, not floats converted to ints.

Sure, I can change it to dump the floating point bits.

[TJnotJT]

Smoke tested the primid dumping on DX11/12, VK, and GL. It turns out that the Mesa GL primid texture looks fine when dumped even with 0x7fffffff (2^31-1) as the max primid value. It appears to get the wrong value only when it samples the texture in the main draw.