Questions regarding GGUF handling

[danishblunt]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Your question

Hello there, first of all, great fork, love how well it works thus far, but do want to ask regarding GGUF's.

When using a qwen 2.5 vl scaled fp8 text model in combination with an qwen image edit 2511 Q4 guff, the memory footprint in my vRAM is significantly smaller than when using a qwen 2.5 vl Q2 model, which in theory should be much smaller, however I still ended up with a notably larger footprint when using the GGUF vairant, around 2.3GB vRAM less to be precise.

I understand this has a lot do with how ROCM generally handles GGUF's poorly and AMD natively seems to lack features to do quantization efficiently.

Are there any recommendations and or explanations from your side to possibly fix this behaviour or is it simply expected when using GGUF formats?

Logs

Other

No response

[patientx]

was answering to a question similar to this in an unrelated discord conversation, the gguf support getting worse isn't caused by amd or rocm but comfyui overall. The official standing is : "stop using gguf's , use our (their) premade, prequantized fp8, fp8mixed (and now int8-convrot since they "officially" started supporting it) models.

gguf's are used with two main nodepacks neither has been updated for months now ... Also comfy's handling of vram is constantly changing , and it is mainly favoring the models their premade , selected quantization types. You should know even int8's now are bad with amd unless you use the custom node, so forget about getting proper support for gguf's, unless comfy's devs change their mind.

TLDR; gguf's were better choice like 5-6 months ago , compared to fp8_e4m3fn ones which were the norm then. This was the case for both models and clips. Now comfy has moved to specific memory optimizations mainly for nvidia and mainly for their official supported quantizations. And gguf isn't part of that. Unless you use old comfy build's or try to monkey patch gguf nodes , again won't work with all the memory optimizations, that can't be fixed....

[danishblunt]

was answering to a question similar to this in an unrelated discord conversation, the gguf support getting worse isn't caused by amd or rocm but comfyui overall. The official standing is : "stop using gguf's , use our (their) premade, prequantized fp8, fp8mixed (and now int8-convrot since they "officially" started supporting it) models.

gguf's are used with two main nodepacks neither has been updated for months now ... Also comfy's handling of vram is constantly changing , and it is mainly favoring the models their premade , selected quantization types. You should know even int8's now are bad with amd unless you use the custom node, so forget about getting proper support for gguf's, unless comfy's devs change their mind.

TLDR; gguf's were better choice like 5-6 months ago , compared to fp8_e4m3fn ones which were the norm then. This was the case for both models and clips. Now comfy has moved to specific memory optimizations mainly for nvidia and mainly for their official supported quantizations. And gguf isn't part of that. Unless you use old comfy build's or try to monkey patch gguf nodes , again won't work with all the memory optimizations, that can't be fixed....

That seems rather bleak, what about folks who don't have 48GB VRAM? Its not like they can load gigantic fp8 models and pray that comfyUI somehow doesn't manage it poorly. What is the route for us from here on out? How would I go on about using large diffusion models efficiently with comfyUI?

[patientx]

https://www.reddit.com/r/comfyui/comments/1v167ix/why_cant_we_get_a_fix_for_dynamic_vram/ ; not directly related but read the top comment from comfyuianonymous himself and the replies to it.

Regarding gguf's ; try int8-convrot models. With the included int8-fast-rocm node pack or if you have rdna4, native loaders work OK I guess. In my case for example , gguf's which were faster are slower than int8 even though they are half the size...

Regarding model usage, with amd at least pinned-memory is harmful , I have an rx 6800 with 16 gb vram, I use this startup :

--use-quad-cross-attention --disable-api-nodes --cache-lru 20 --disable-smart-memory --disable-pinned-memory

of course quad-cross is a choice but the caching method and those two memory enhancements work better this way.

(also keep in mind, dynamic-vram isn't enabled here, even if I didn't specifically disable it, to enable it I have to add : --use-dynamic-vram ; which works more or less the same for me BUT it could be better for newer gpu models from amd.)

And another note ; with ltxvideo2.3 and wan 2.2 especially this combo helps a lot :

--disable-dynamic-vram --reserve-vram 16 It looks counterintuitive but yes, reserving whole of vram actually makes ltxvideo and wan work way faster than without it. I even saw people with nvidia doing this.

[danishblunt]

https://www.reddit.com/r/comfyui/comments/1v167ix/why_cant_we_get_a_fix_for_dynamic_vram/ ; not directly related but read the top comment from comfyuianonymous himself and the replies to it.

Regarding gguf's ; try int8-convrot models. With the included int8-fast-rocm node pack or if you have rdna4, native loaders work OK I guess. In my case for example , gguf's which were faster are slower than int8 even though they are half the size...

Regarding model usage, with amd at least pinned-memory is harmful , I have an rx 6800 with 16 gb vram, I use this startup :

--use-quad-cross-attention --disable-api-nodes --cache-lru 20 --disable-smart-memory --disable-pinned-memory

of course quad-cross is a choice but the caching method and those two memory enhancements work better this way.

(also keep in mind, dynamic-vram isn't enabled here, even if I didn't specifically disable it, to enable it I have to add : --use-dynamic-vram ; which works more or less the same for me BUT it could be better for newer gpu models from amd.)

And another note ; with ltxvideo2.3 and wan 2.2 especially this combo helps a lot :

--disable-dynamic-vram --reserve-vram 16 It looks counterintuitive but yes, reserving whole of vram actually makes ltxvideo and wan work way faster than without it. I even saw people with nvidia doing this.

I also have an RDNA 2 card, so out setup should be somewhat similar, are you using --cache-lru 20 to prevent spilling into iGPU?

I do want to note that I'm trying to make qwen image edit work, but for some reason it seems to want more than 20GB vRAM, which seems way to much especially when using GGUFS as low as Q3 and it still wants to use 20gb vram.

[patientx]

rx 6800 - w10 - 32 gb system ram
no igpu here, was using cache-none but tested all available options, this gives the better overall performance. mind you I have a 120gb swap file setup with windows.

[danishblunt]

rx 6800 - w10 - 32 gb system ram no igpu here, was using cache-none but tested all available options, this gives the better overall performance. mind you I have a 120gb swap file setup with windows.

Why would you swap that much? Wouldn't that completely ruin your speed to umbearibly low it/s?

[patientx]

rx 6800 - w10 - 32 gb system ram no igpu here, was using cache-none but tested all available options, this gives the better overall performance. mind you I have a 120gb swap file setup with windows.

Why would you swap that much? Wouldn't that completely ruin your speed to umbearibly low it/s?

These models eventually go into swap with my system ram sooner or later. Yes at first one would think it would be much slower but I think unlike nvidia even a small amount of system ram or swap usage makes the speed slower already. Before changing swap size, first try the --reserve-vram 16 ; do something like 832x480x241 (10 seconds, 24 fps) ltxvideo gen, make sure to do it at least three times and then try the same with the --disable-dynamic-vram --reserve-vram 16 added. It is almost 2x speedup most of the time, specially at higher framecounts. (I can do something like 1280x960x241 with this , no oom)

[danishblunt]

rx 6800 - w10 - 32 gb system ram no igpu here, was using cache-none but tested all available options, this gives the better overall performance. mind you I have a 120gb swap file setup with windows.

Why would you swap that much? Wouldn't that completely ruin your speed to umbearibly low it/s?

These models eventually go into swap with my system ram sooner or later. Yes at first one would think it would be much slower but I think unlike nvidia even a small amount of system ram or swap usage makes the speed slower already. Before changing swap size, first try the --reserve-vram 16 ; do something like 832x480x241 (10 seconds, 24 fps) ltxvideo gen, make sure to do it at least three times and then try the same with the --disable-dynamic-vram --reserve-vram 16 added. It is almost 2x speedup most of the time, specially at higher framecounts. (I can do something like 1280x960x241 with this , no oom)

Had to adjust the reserve ram parameter somewhat but yeah saw some speedups as well. The int8 recommendation was however the best one, it lowered vRAM and increased performance quite a bit.