Add Video Super Resolution for Windows x64 (AMD, Intel and NVIDIA), Windows ARM (Snapdragon), Linux and MacOS - #1557
Add Video Super Resolution for Windows x64 (AMD, Intel and NVIDIA), Windows ARM (Snapdragon), Linux and MacOS#1557linckosz wants to merge 7 commits into
Conversation
|
Here is a summary of my observations following many tests conducted with different settings and multiple GPUs. AMD delivers the best results, followed by Nvidia, then Intel. VSR (Video Super Resultion) performs exceptionally well in SDR, but HDR setup (in the code) is challenging, and some specifics feature (like sharpening) aren't yet avaible due to drivers limitation. I would advise using the Video Enhancement feature without HDR. 1) Nvidia RTX 4070 Ti: 2) Intel Arc A380: 3) AMD RX 7600: 4) Intel UHD Graphics Xe-LP (iGPU) Does the feature "Video Enhancement" have an impact on the performance? Does HDR work with Video Enhancement? Below are my suggestions for different scenarios with varying settings from host to display (Host->Stream->Display): My suggestion for Office: 1440->1440->1440 My suggestion for Games: 1080->1080->1440 GPU comparison (1440->720/1080->1440 / 150 Mbps / Static picture):
In-motion block artifact comparison at low bitrate (1440->1080->1440 / 20 Mbps):
In-motion GPU comparison: When upscaling (stream at 720 or 1080 to render on a 1440 display), in all scenarios AMD provides the best result. When we use native resolution (stream at 1440 to render on a 1440 display), v5.0.1, AMD and Nvidia are all similar, only Intel still give a pixelized picture.
Resolution comparison (Nvidia Quality 4):
720p GPU comparison (1440->720->1440 / 150 Mbps):
1080p GPU comparison (1440->1080->1440 / 150 Mbps):
|
|
Here is a comparison on a Macbook M1 Pro.
|
|
There is a lot of reformatting going on and it makes it difficult to review the changes. Can you disable whatever auto-formatting it is that you're running? Removing trailing spaces if there are any is OK though. |
|
@andygrundman , |
|
Sure, stuff like this, where there is no actual change, just reformatting. It looks like it's trying to line things up to the brace for some reason. The only time you want to auto-format existing code is is if a project is already using a tool and there is a config file in the repo (like Sunshine which has a .clang-format). And the one exception of trailing spaces which should be fixed (that comes from the olden days when you might send a patch via email where trailing spaces would get lost and cause annoying conflicts). 
|
|
@andygrundman , |
|
Is there any test build with this feature available for Windows platform? Where can I download it? |
Here |
Thanks a lot. I'll check it :) |
|
After downloading the Windows build I've done some rapid tests and it doesn't seems to be working at all in my case. My setup is as follows: Host machine: Running at 1440p HDR (AMD Radeon Adrenaline driver) First I've enabled the new check box to activate the Upscaler enhancer. Connected to host with no issues but however the overlay info doesn't reflect any upscaler feature (pressing CTRL + ALT + SHIFT + S) Then to compare, i've deactivated the checkbox so the stream should be pure 1080p without any enhancement. To my eyes the image quality is identical, there is no difference. So I believe that I must be doing something wrong or the feature is not working. It could be helpful if, at least, the streaming overlay info could display something to know if the enhancer is being used or not. I've also tried lowering the host machine to 1080p while keeping the client working on 1440p and always selecting 1080p in Moonlight settings, but no difference. May be I'm missing something or may be the Intel GPU is not compatible with this experimental feature. |
|
Can't get this working. Is this feature using this library: https://github.com/OpenVisualCloud/Video-Super-Resolution-Library ? or it's only enabling some features already present into the GPU drivers? If this depends of the installed drivers then it should detect it before displaying the checkbox or at least display any warning, because right now I can just enable or disable it and the image quality is the same, it's just identical, no changes. |
Hi @fidoboy , If you are able to check the box, it means that your iGPU has this feature capability, otherwise it will be greyed out. When you stream at a lower resolution than your screen, make sure you use borderless fullscreen or fullscreen, select hardware acceleration and that the overlay stats is displaying "AI-enhanced" like below while streaming. |
The CPU is Intel Celeron N4500 and I can check or uncheck the checkbox, it's not grayed out. But It doesn't display that overlayed info like yours. Mine shows HEVC 10bit only but there is not "AI Enhanced" text when the feature is enabled. What else can I try? |
|
I'm using this build: https://ci.appveyor.com/api/buildjobs/fntsewbjxekye2v2/artifacts/MoonlightPortable-x64-r2725.zip The new checkbox is being displayed, it's not grayed out but I does nothing. The image quality is identical when the feature is activated or when it's not and the overlay stats does display nothing about "AI-Enhanced" I've done my tests with 1440 -> 1080 -> 1440 and also 1080 -> 1080 -> 1440. There is no changes when the feature is activated, image is exactly the same I'm sorry but I doesn't have Chrome to test. I never use that browser |
Yes, that the correct version. |
I forgot to say that I've also tried without HDR enabled in moonlight settings. The result is the same. There is no "AI-Enhanced" text in the overlay info. |
|
This is with HDR enabled:
and this is with HDR disabled:
in both cases the AI Enhacer is checked but it seems that it's not being used |
|
Works on macOS 15.4 with m3 cpu. Tested with 1080p host → 1080p stream → 1080p display, at 150mbps. It would be great to have a slider to select how much the image is sharpened, it looks over sharpened right now imo. Default: AI-enhanced: The added latency might be an issue on fast-paced games. |
Hi @jasperaelvoet , MTLFXSpatialScalerDescriptor documentation |
|
This feature is awesome! |
Great! I'm impatient to test it. I'll report back my findings. |
|
Well, I spent some time doing new tests with this feature and, in short terms, this is a placebo. There is no difference between the image quality when it's enabled or disabled. I've tried the following setups:
In both cases HDR was always enabled and in both cases I've compared between the video upscaler enabled and disabled. I have verified it because the text "AI Enhanced" was being displayed into the overlay stats. There was no any difference between them in the terms of image quality, the clarity, sharpness, etc. are exactly the same. And then I've tried the setup 1440 -> 1440 -> 1440 and the difference is very noticeable. So, in short, now the text is being displayed into the overlayed info but THERE IS NO ANY CHANGE IN IMAGE QUALITY. The image is the same if the feature is activated or not. There is no difference. |
I'm already laughing because your post is a real nonsense. What's the point to use this feature if you are streaming with the same resolution than your display? If the device where moonlight is running already uses 1080p and the host and also the stream is 1080p, where is the purppose for a video upscaler? This only have some sense if the stream is a lower resolution than the display, but it doesn't do anything if the resolution is the same. |
I think it would be beneficial if you substantiated your observations with screenshots on vs off so contributors can see what exactly you see. In the other comparison screenshots there are discernible differences. But whether you notice can be quite subjective. |
4b98aa1 to
c3c2ccf
Compare
0c21c71 to
faf324d
Compare
- plvk.cpp: use SDL_Vulkan_GetDrawableSize (not SDL_Metal_GetDrawableSize) when computing the VSR ratio on the libplacebo Vulkan path. Matches the existing call at plvk.cpp:862 and is correct on Windows/Linux (upstream PR moonlight-stream#1557 used the Metal API on a Vulkan window, yielding a wrong/zero drawable size). - streamingpreferences: default bitrateMax to 500000 (500 Mbps), the dropdown's lowest option and the pre-moonlight-stream#1859 unlock cap, so toggling "Unlock bitrate" no longer leaves the slider stuck at 150 Mbps. - session.cpp: do not enable VSR on secondary multi-stream decoders. VideoEnhancement is a process-wide singleton; concurrent streams would clobber its shared device/ratio/algo state. VSR stays on the primary stream only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
da6319e to
aff323f
Compare
|
Here are the downloadable artifacts (executables): Nothing changed, just did a rebase to keep the code up-to-date. |
|
Hi there, do you plan to add some form of sharpness settings for Metal clients? Yesterday built a merged version of your VSR branch and the master, and it is fascinating how much it helps on the host side latency (20ms --> 4ms) with no artifacts or increased host, but it is definitely oversharpened a lot. Thanks! |
@fazek08 , MetalFX does not offer any parameter of sharpening, it is On or Off. As for the rendering speed, when On, I am using a slightly different rendering path, what made a significant improvement is by disabling « waitUntilCompletion » method. |
|
Maybe some form of pre-render sharpness modifier would do the same effect, or am I on the wrong track? Other option is implementing client-side anti-aliasing which seems to me to be much more work. |
The tech used is already client-side only, and usual 3D anti-aliasing is not applicable as we receive 2d images, no 3d information included. Therefore all technologies are algorithms based on texture improvements, they will never be as good as pure DLSS for instance, but it is already pretty impressive, especially the Nvidia one on RTX which relies on AI compared to to others. |
|
Maybe I'm stupid but I have an nvidia 3060ti and VSR isnt turning on no matter what I do. Also the new renderer has some sort of incompatibility with reshade as I'm trying to inject reshade into moonlight to apply some effects to get the stream to look better but the app just crashes with VSR and refuses to work with post-stream injection whereas disabling vsr makes it work fine like on normal moonlight. If you got any other suggestions for me to fix this I'd be grateful. I've also tried changing the source code to inject directly into the app the dll but to no avail, neither enabling debugging options and messing with those settings, it always remains on default FSR1. |
This commit introduces video super-resolution and upscaling support across Windows, Linux and macOS platforms. DirectX 12 Upscaling Pipeline - Added a full D3D12 renderer to enable advanced GPU upscaling features. - Works on Windows for AMD, NVIDIA, and Intel GPUs, and on macOS (ARM) using Metal. - Supports SDR, HDR, and YUV 4:4:4 pipelines. - Uses vendor-specific driver upscalers by default (e.g., AMD AMF), with automatic fallback to a shader-based solution (typically FSR1) when unsupported or unreliable on older hardware (e.g., NVIDIA GTX). - Automatic algorithm selection based on vendor, GPU model, and available features. - Optimized for low-end GPUs (such as Intel UHD) by combining GPU upscaling with RCAS shader operations. - Uses FFmpeg DX12 hardware decoding by default; includes fallback to DX11 decoding with DX11–DX12 interop in cases where DX12 decoding causes stutters on some AMD GPUs. Rendering remains fully D3D12. - Added detailed upscaler information to the on-screen statistics overlay. - Add Shader FSR1 for Linux macOS Upscaling - Integrated MetalFX upscaler for macOS platforms. Tested Hardware - NVIDIA RTX 4070 Ti (Windows & Linux) - NVIDIA GTX 1050 (Windows & Linux) - AMD Radeon RX 7600 (Windows) - AMD Ryzen 7 780M iGPU (Windows) - Intel Arc A380 (Windows) - Intel UHD Graphics (16 EU, N95 CPU) (Windows) - Snapdragon 7c (Windows ARM) - Apple M1 Pro (macOS) - Apple M3 (macOS)
…aces A user on Windows 10 with an RTX 3060 Ti could not stream at all, while RTX 4070 Ti and 5060 Ti machines worked. The GPU was not the cause: the 3060 Ti host was the only one still on Windows 10. D3D12CreateDevice() requested ID3D12Device9, an Agility SDK / Windows 11 interface, and failed with E_NOINTERFACE (0x80004002) since we ship no Agility runtime. ID3D12GraphicsCommandList7 and ID3D12VideoProcessCommandList3 had the same problem and would have failed right after. None of the methods we actually call require those versions, so lower each member to the minimum interface the code uses. ID3D12Device and ID3D12GraphicsCommandList cover every call site; ID3D12VideoProcessCommandList1 provides ProcessFrames1(). m_VideoDevice stays ID3D12VideoDevice2 because CreateVideoProcessor1() lives there. With video enhancement enabled, AV_HWDEVICE_TYPE_D3D11VA fell through to D3D12VARenderer in both passes, so D3D11VA was never instantiated. Once D3D12 failed there was nothing left but libplacebo/Vulkan, turning "upscaler unavailable" into "no video at all". Latch the failure in VideoEnhancement so D3D11VA can take over, and disable enhancement for that session so the stats overlay does not advertise a missing upscaler. Move the D3D12VA_ENABLED and IsWindows10OrGreater() checks ahead of device creation and latch them too, which makes D3D12VA_ENABLED=0 reproduce a system that lacks the required interfaces. The FSR1 hooks were destroyed after pl_vulkan_destroy() and pl_log_destroy(), a use-after-free of the pl_gpu they were parsed on that also leaked their GPU resources on every renderer recreation. Release them while the GPU is still alive. Parse the HDR shader variant only for 10-bit streams, since the SDR hook was previously duplicated for no benefit, and log a parse failure instead of silently rendering without the hook. Video enhancement availability only reflected that FFmpeg advertises a D3D12VA hwaccel, which stays true where the device cannot be created, so the setting looked usable on affected systems. Grey out the checkbox, the mode label and the mode dropdown when D3D12 is out of reach. Finally, log the OS version at startup. Renderer availability depends on it and the logs carried no way to recover it.
A user streaming 1440p180 on a RTX 3060 Ti rendered at 53 FPS while decoding kept up at 184 FPS. Measured over eight enhancer modes on that machine, the two that run a DNN through the driver cost about 17 ms per frame while every shader path cost about 2 ms. At 180 Hz the budget is 5.55 ms, so the upscaler alone capped the stream at 58 FPS. The VSR feature was created without any quality level in its parameter block and then evaluated with NVSDK_NGX_VSR_Quality_High. NGX sizes the internal resources of a feature from the level given at creation time, so the two never matched. Set the level before NVSDK_NGX_D3D12_CreateFeature() and evaluate with the same one. Derive a ceiling from the amount of VRAM, since VSR High is out of reach for a mid-range card at a high refresh rate: 8 GB or less starts at Medium, 4 GB or less and battery power start at Low. Measure the GPU time of the graphics queue with timestamp queries resolved into a readback buffer, and step the quality down when it exceeds 80% of the frame budget. The level cannot be swapped on a live feature, evaluating with another one hangs the device, so the change goes through a renderer reload which rebuilds the feature. The level and a session cap that only ratchets down are static so they survive that reload, which is what stops the renderer from oscillating between two levels for a whole session. Stop the measurement once there is nothing left to decide, and for every enhancer other than VSR since none of them has a quality knob.
The stats overlay blinked out roughly once a second, which is its update period. notifyOverlayUpdated() moved the live texture out of m_OverlayTextures[type] before creating the new one, and raised m_OverlaySkip for the whole operation, which is what renderFrame() tests before drawing any overlay. So from the moment the update started until the new texture was uploaded and waited on, there was nothing to draw. It also held m_OverlayLock across the resource creation and the GPU upload, stalling the render thread that takes the same lock for every frame. Build the texture, the upload heap and the vertex buffer into locals, then take the lock only to swap the pointers in. m_OverlaySkip goes back to being what its name suggests, a re-entrancy guard for the update itself, and renderFrame() draws whatever complete overlay is currently published. The shader resource view was written to the start of the descriptor heap for every overlay type, while renderOverlay() binds the table at an offset of type. OverlayDebug is 0 so the stats happened to work, but OverlayStatusUpdate read a descriptor that was never written. Write each type into its own slot.
Resetting a command allocator and its command list was open coded in thirteen places, always as the same two lines. One of them reset the graphics command list with the VideoProcess allocator, which would have failed with E_INVALIDARG had that error path ever been taken. Give each frame slot its own allocator and funnel every site through resetVideoProcessCommandList() and resetGraphicsCommandList(), which wait on the fence that protects the slot before recycling it. Since a slot is only reused every m_FrameCount frames, that wait is not normally reached. No behaviour change: renderFrame() still waits for the GPU before returning, so only one frame is ever in flight.
The GPU side wait that makes the graphics queue observe what the VideoProcessor just wrote was only inserted when V-sync was on. With V-sync off the two queues ran unsynchronized and the graphics queue could sample a texture the VideoProcessor had not finished writing, which is what the "otherwise the tearing is too agressive" note on the CPU stall describes. Insert it unconditionally. The reverse wait, which keeps the next frame's VideoProcessor from overwriting a texture the current frame is still reading, only makes sense when that queue has work. On the NVIDIA VSR path both steps run on the graphics queue, so every frame was queuing a Wait on an idle VideoProcess queue that nothing would ever drain. Track whether the pipeline feeds that queue and skip it when it does not. Also record why the CPU stall at the end of renderFrame has to stay: the shader helpers rewrite their descriptors every frame, the NGX feature is a single instance and the intermediate textures are single buffered, so overlapping frames races all three and the driver reports DXGI_ERROR_DEVICE_HUNG.
Every failure path in renderFrame jumps to the Present label, which presents the current back buffer without having written anything into it. With a three buffer flip model swap chain that buffer still holds the frame from three presents ago, so a dropped frame shows up as a flash of stale or black content. Skipping the Present is not an option: renderFrame has already consumed a token from the frame latency waitable object and must give one back, otherwise the next frame blocks for the full timeout. Copy the previously presented back buffer into the current one instead, which repeats the last frame the viewer actually saw.
The issues was related to the use of Windows 10 which has few limitation in the support of D2D12, everything is now fixed. |
|
wow what an interesting feature! with this we could potentially reduce the streaming bitrate/bandwidth while maintaining same image quality. It could also be useful for improving performance on devices that struggle with GPU load or limited network bandwidth basically same as in games with upscalers any updates on when this feature might be implemented/merged? also, would it be possible to integrate this into Artemis as well, since Artemis is based on Moonlight? |
Thanks. I don't know about Artemis, but I also made this feature for moonlight-android which is the same code base, the PR should be easily compatible: |
20 participants
Video Super Resolution (VSR) for Moonlight
Video Super Resolution (VSR) is to video what DLSS is to 3D rendering.
So why not make Moonlight one of the first game streaming solutions to leverage this technology?
AI upscaling significantly reduces bandwidth usage without compromising video quality!
Tested GPUs:
Context
Video Super Resolution (VSR) is to video what DLSS is to 3D rendering.
So why not make Moonlight one of the first game streaming solutions to integrate this technology?
AI-based upscaling drastically reduces bandwidth usage while maintaining excellent video quality.
NVIDIA, Intel (link in French), and more recently AMD have all started promoting their respective AI-powered video enhancement technologies.
Implementation
Implementing VSR was not straightforward. I had to add a Video Processor component to D3D12VA to offload frame processing from the CPU to the GPU, allowing it to take advantage of additional GPU capabilities.
SettingsView.qml.d3d12va.cppandd3d12va_shaders.cpp.Platform-Specific Details:
✅ NVIDIA: Provides VSR, the best overall rendering.
✅ Intel: Is currently using FSR1 Shader, IntelVPL shall be workable but could not make it work yet.
✅ AMD: Released AMF Video Upscaling, a ML model based on FSR1 delivering impressive results.
Other VSR Implementations
VSR is also available in other Moonlight ports:
Android: Add Video Super Resolution moonlight-android#1567
iOS and tvOS: Video Super Resolution for Moonlight iOS/tvOS moonlight-ios#704
Xbox: Add Video Super Resolution for Xbox TheElixZammuto/moonlight-xbox#267
Results & Comparisons
Resolution Test:
Banding Test: