fix(server): give auto-tune real VRAM usage on Windows - #3486
Open
anditherobot wants to merge 1 commit into
Open
fix(server): give auto-tune real VRAM usage on Windows#3486anditherobot wants to merge 1 commit into
anditherobot wants to merge 1 commit into
Conversation
get_vram_usage_gb() returned -1.0, which means auto-tune measured system RAM instead of video RAM and subtracted that from the card's total. Windows GPUs are affected. Because of that inaccurate reading the auto-tuner sees little or no memory free, and by default gives models the smallest context size. This fix reads the real figure from the Windows GPU performance counters instead. Fixes lemonade-sdk#3461
anditherobot
force-pushed
the
fix/3461-windows-vram-metrics
branch
from
September 3, 2026 16:11
16b8fb6 to
1550c86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3461.
When Lemonade loads a model, auto-tune uses the available GPU memory to choose its context size. On Windows, it mistakenly used system RAM usage instead of VRAM usage. In testing, this made an 8 GB graphics card appear to have 0.00 GB free, so auto-tune selected the minimum context size of 4,096 tokens.
Technically,
get_vram_usage_gb()returned-1.0, causing auto-tune to fall back toget_memory_usage_gb(). Windows now reads the actual dedicated-memory usage from its GPU performance counters.Measured on an 8 GB RX 5700 XT with
Llama-3.2-1B-Instruct-GGUFand automaticctx_size:Implementation notes
The fix reads current VRAM usage from Windows and only returns a value when it can be safely attributed to a single GPU.
Testing details
lemondon Windows 11 with MSVC in Release mode, with no new warnings.Llama-3.2-1B-Instruct-GGUFwith automaticctx_size. Available VRAM increased from 0.00 GB to 6.81 GB, and the selected context increased from 4,096 to the model's 131,072-token limit.test_gpu_adapter_memorycoverage for valid single-GPU readings and fallback cases. The test passes under MSVC and g++ with-Wall -Wextra.Scope
Testing
Documentation
Breaking Changes
On supported Windows systems,
vram_gbin/system-statsand/metricsnow reports a value where it previously returnednull.AI-assisted contribution