Skip to content

fix(server): give auto-tune real VRAM usage on Windows - #3486

Open
anditherobot wants to merge 1 commit into
lemonade-sdk:mainfrom
anditherobot:fix/3461-windows-vram-metrics
Open

fix(server): give auto-tune real VRAM usage on Windows#3486
anditherobot wants to merge 1 commit into
lemonade-sdk:mainfrom
anditherobot:fix/3461-windows-vram-metrics

Conversation

@anditherobot

Copy link
Copy Markdown
Contributor

Summary

Fixes #3461.

When Lemonade loads a model, auto-tune uses the available GPU memory to choose its context size. On Windows, it mistakenly used system RAM usage instead of VRAM usage. In testing, this made an 8 GB graphics card appear to have 0.00 GB free, so auto-tune selected the minimum context size of 4,096 tokens.

Technically, get_vram_usage_gb() returned -1.0, causing auto-tune to fall back to get_memory_usage_gb(). Windows now reads the actual dedicated-memory usage from its GPU performance counters.

Measured on an 8 GB RX 5700 XT with Llama-3.2-1B-Instruct-GGUF and automatic ctx_size:

Before After
Reported in use 11.30 GB (system RAM) 1.19 GB (VRAM)
Available VRAM 0.00 GB 6.81 GB
Context size chosen 4,096 131,072 (the model's own limit)

Implementation notes

The fix reads current VRAM usage from Windows and only returns a value when it can be safely attributed to a single GPU.

  • Windows can return multiple memory-counter rows, including zero-use software adapters, so the implementation selects the largest value instead of adding them together.
  • Windows does not reliably identify which physical GPU owns each row. If more than one GPU is detected, the existing fallback behavior is preserved.
  • The result is not cached because auto-tune must observe VRAM released when another model is unloaded.

Testing details

  • Built lemond on Windows 11 with MSVC in Release mode, with no new warnings.
  • Tested end to end on an 8 GB AMD Radeon RX 5700 XT by loading Llama-3.2-1B-Instruct-GGUF with automatic ctx_size. Available VRAM increased from 0.00 GB to 6.81 GB, and the selected context increased from 4,096 to the model's 131,072-token limit.
  • Added test_gpu_adapter_memory coverage for valid single-GPU readings and fallback cases. The test passes under MSVC and g++ with -Wall -Wextra.

Scope

  • This PR addresses one clear issue or change.
  • I reviewed the full diff myself before submitting.
  • I removed unrelated local changes.
  • I kept refactoring separate unless it is required for this change.

Testing

  • Code builds without errors locally.
  • I tested this change locally.
  • I described the testing performed above.

Documentation

  • Documentation is not affected by this change.

Breaking Changes

  • This PR does not introduce breaking changes.

On supported Windows systems, vram_gb in /system-stats and /metrics now reports a value where it previously returned null.

AI-assisted contribution

  • I used AI tools for this PR.
  • I verified that I understand the changes.
  • I checked for hallucinated APIs, unrelated changes, and incorrect assumptions.

@github-actions github-actions Bot added the bug Something isn't working label Sep 2, 2026
@jeremyfowers jeremyfowers added this to the v11.9.1 milestone Sep 2, 2026
get_vram_usage_gb() returned -1.0, which means auto-tune measured system RAM
instead of video RAM and subtracted that from the card's total. Windows GPUs
are affected.

Because of that inaccurate reading the auto-tuner sees little or no memory
free, and by default gives models the smallest context size.

This fix reads the real figure from the Windows GPU performance counters
instead.

Fixes lemonade-sdk#3461
@anditherobot
anditherobot force-pushed the fix/3461-windows-vram-metrics branch from 16b8fb6 to 1550c86 Compare September 3, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows GPU auto-tune uses system RAM instead of VRAM usage

2 participants