Thunks: stack-protector prologue clobbers r11 (the host callee address) in CallHostFunction for >=7-argument functions

[dragosol]

Summary

CallHostFunction in ThunkLibs/include/common/Guest.h passes the host callee address to the host side in guest r11 via a custom ABI. For thunked functions with 7 or more arguments, every other register is already live, so the compiler's stack-protector prologue picks r11 as its scratch register and emits

mov r11, fs:[0x28]      ; load stack canary  -> DESTROYS the host callee address

before the asm volatile("" : "=r"(host_addr)) barrier that is supposed to pin it. The canary is then stored into the host_addr slot, the host side does blr x8 on it, and the process dies with an access violation at a garbage address.

FEX's own source comment (around Guest.h:117) already predicts this hazard for functions with many arguments; the barrier as written is not sufficient.

Why it is easy to miss

The faulting address is the glibc stack canary, so it:

• is high-entropy and different on every run (looks like random corruption, not a deterministic bug),
• always ends in a zero byte (glibc canaries are 0x??????????????00),
• surfaces to the emulated app as a crash "in the graphics driver", pointing at the wrong component.

In our case it presented as a crash inside Mesa's venus Vulkan driver at vn_GetQueryPoolResults. A gdb breakpoint on that function, armed 2.5 s before the fault, never fired - the driver was never entered. The reported ip was the intended callee, not the actual PC.

Evidence

libGL-guest.so, affected instantiation, before the fix:

movq %fs:0x28, %r11        ; canary load destroys the incoming callee address
movq %r11, 0x38(%rsp)      ; canary slot
movq %r11, 0x30(%rsp)      ; host_addr field   <== host side does blr on this

Host side (Host.h) then does ldur x8, [x0, #<off>] / blr x8 and jumps into the canary.

Caught with catch signal SIGSEGV conditioned on si_addr == __stack_chk_guard:

__stack_chk_guard = 0xaf84028d590ad200
pc  0xaf84028d590ad200      <- PC == si_addr  => instruction fetch at the canary
x8  0xaf84028d590ad200      <- branch-target register
sp  0x80089e6cfe10          <- sane; all eight arguments correct

Backtrace frame #1 is GuestWrapperForHostFunction<...>::Call<...> in the thunk library - the thunk itself, not the driver.

Scope in our build

Instantiations of CallHostFunction whose prologue loads the canary into r11 before the callee use:

guest thunk lib	instantiations	affected
libGL-guest.so	736	66
libvulkan-guest.so	476	9 (incl. vkGetQueryPoolResults, vkCmdPipelineBarrier, vkCmdBlitImage, vkCmdWaitEvents, vkCmdCopyQueryPoolResults)
libcuda-guest.so	363	5
EGL / drm / wayland-client / asound / VDSO	-	0

All affected take >= 7 arguments, consistent with the register-pressure explanation.

Fix

__attribute__((no_stack_protector)) on CallHostFunction resolves it. After rebuilding, 0 instantiations load the canary into r11, the incoming r11 flows straight to the host_addr slot, and exported symbol sets are byte-identical (verified by diffing symbol names, not just counts).

A more targeted alternative, if stripping the mitigation is undesirable, is to force the value into a register the prologue cannot reuse:

register uintptr_t p asm("r11");
asm volatile("" : "=r"(p));

Happy to open a PR with whichever approach maintainers prefer.

Host side

We audited the mirror-image ABI on the aarch64 host side (Host.h, LOAD_INTERNAL_GUESTPTR_VIA_CUSTOM_ABI reading x11). All 42 CallGuestPtr instantiations read x11 before any write, so it does not reproduce there - but for incidental reasons: the aarch64 canary sequence allocates x8/x9, and AAPCS64 passes 8 args in registers so the starvation that forces r11 on x86-64 never arises. Safe by circumstance, not by construction.

Environment

• FEX built from main (FEX-2608 + ~130 commits)
• Host: aarch64 Linux (Arch ARM) guest under QEMU on Apple Silicon (M3 Max)
• Guest thunks built for x86-64
• Reproduced with Overwatch 2 under Proton/Wine; DXVK reaches vkGetQueryPoolResults during D3D11 device creation and hits an affected instantiation first

Verification gotcha for reproducers

Do not verify with a native objdump on an aarch64 host - it cannot disassemble x86-64 guest libraries and exits 0 after a few lines, so objdump -d <lib> | grep fs:0x28 silently reports 0 matches even for known-broken libs. Use llvm-objdump. Also, raw fs:0x28 counts over-report: most hits are ordinary exported pack functions using r11 as scratch with no host_addr. The meaningful metric is a canary load into r11 inside CallHostFunction.