window: better evaluate when to block Solitary

[LionHeartP]

Describe your PR, what does it fix/add?

Another attempt at #16131
After #15958 and the release of GE-Proton 11-6 which fixes Steam overlay in wine-wayland, Solitary is denied for Proton because of an empty surface that is created.

This PR adds a helper function for the subsurface buffer being empty before denying Solitary and allows Direct Scanout to engage. When the, now working, Steam Overlay is brought up, Hyprland correctly blocks Solitary and Direct Scanout until the Overlay is closed.

Tested with the following:
direct_scanout = 1

• Plezy, when fullscreened, is granted Solitary and Direct Scanout until a video starts playing. When the latter happens, solitary is blocked with "not opaque, subsurfaces" as intended. That means the mpv overlay is rendered and the video is not just a black frame.
• Proton-GE 11-6 games get granted Solitary now (previously blocked by "not opaque, subsurfaces" AT ALL TIMES), until you invoke the Steam overlay. At that moment, the helper function returns nullptr for because if (surf && (surf->m_current.buffer || surf->m_mapped)), blocking Solitary correctly, and rendering the overlay and the mouse while everything can be interacted with properly.
• Games running directly on wine show no change with this PR vs. without. Solitary and DS both engage and work fine.
• Every other app when fullscreened, if no overlays or anything is involved, have Direct Scanout engaged properly (i.e. Chromium/Kitty/etc.

direct_scanout = 0
This is where my previous attempt fell short. On this PR, with the changes to Renderer.cpp we fall back to the correct subsurface to render instead of the empty dummy one.
Everything else already worked as expected anyway.

Is there anything you want to mention? (unchecked code, possible bugs, found problems, breaking compatibility, etc.)

Again, needs testing in case I missed something obvious or there is an edge case I haven't thought of.

Is it ready for merging, or does it need work?

Possibly ready. Maybe the code can be trimmed down a bit. I'll take a look at it again hopefully in a couple of days when I'm going to be back from a trip.

[vaxerski]

what's the difference between this and solitary resource? They are to achieve the same thing

[LionHeartP]

I was using getSolitaryResource as a fallback during testing.

Let me try and implement this into that instead.

[LionHeartP]

Made it so it checks in 2 steps before denying solitary. The helper function I added in the first commit made it easier because it would only run if getSolitaryResource returned nullptr.

So now, getSolitaryResource evaluates whether to block or not in multiple levels. Scans for zIndex < 0 while also checking the child subsurfaces so that when, for example, you pull up the Steam Overlay (which is now possible on latest GE Proton in wine-wayland), you block Solitary for the mouse cursor to render, but not when the overlay is not drawn thus making Solitary+DirectScanout possible.

Via isSolitaryRoot we check if the target will be blocked of solitary due to "not opaque" ONLY when the function is true, so Proton can get Solitary even though it has a non opaque parent subsurface while also blocking it correctly for, i.e., Plezy.

Tested browsers/terminal emus/random apps, they all got Solitary properly when appropriate and couldn't find any glitches myself.

Being on an AMD GPU, obviously this needs testing with nvidia and/or various DS/VRR setting combos.

For me it works as expected, or at least, I couldn't find a weird scenario where it causes issues.

[Dregu]

Still getting a black screen on GE-Proton11-6 when DS is OFF but game is solitary.

Also unrelated really, but solitary is also getting randomly blocked by "other workspaces" when a special workspace simply exists, even if it's not active. That has been happening forever though.