host UDS: recvmsg fails on an SCM_RIGHTS-imported socket with SO_PASSCRED enabled

[hansent]

Background. We are evaluating gVisor with nvproxy to run GPU video-inference workloads for several tenants on one node, sharing the GPU through NVIDIA MPS so each tenant gets a device-memory limit. MPS clients register with a host-side control daemon over a Unix socket, and the daemon hands each client its server connection via SCM_RIGHTS. Tracing that handshake from inside a sandbox (runsc's own strace) showed the CUDA client dying on the first recvmsg of that passed socket; the socket had SO_PASSCRED enabled. Everything below is the minimal, GPU-free version of that failure, arrived at by eliminating socket type, MSG_CMSG_CLOEXEC, nested SCM_RIGHTS and the receiving flags one at a time (all of those work).

Summary

A sandbox receives a connected AF_UNIX socket from a host process via SCM_RIGHTS over a mounted host Unix socket (--host-uds=open). If the host side enabled SO_PASSCRED on that socket before passing it, recvmsg with a control buffer on it fails with EINVAL inside the sandbox, whether the host attached only SCM_CREDENTIALS or SCM_CREDENTIALS plus SCM_RIGHTS; a plain recv without a control buffer works, and with SO_PASSCRED off everything works. Under runc all cases work. The kernel attaches SCM_CREDENTIALS to every message delivered on a SO_PASSCRED socket, and the sentry's receive path for host-imported sockets hands all host control messages to a parser that only accepts SCM_RIGHTS.

Reproducer (no GPU, no Kubernetes; Linux)

uds_passcred_repro.py: https://gist.github.com/hansent/e0645f1e2cdbac07f0ac9728bdd51bba (pinned: https://gist.github.com/hansent/e0645f1e2cdbac07f0ac9728bdd51bba/9bf3a03488107ea3e92b787aa3ed1e4c76b99cb1); the same file is embedded below. The server accepts a SOCK_SEQPACKET control connection, creates one SOCK_STREAM socketpair per client, enables SO_PASSCRED on both ends (unless --no-passcred), passes one end via SCM_RIGHTS, waits for "hi" on it, and then sends exactly one message on it. Each client invocation runs one receive case on that passed socket, so stream framing cannot mix cases.

Dedicated runsc runtime in /etc/docker/daemon.json (Docker has no per-run runtime-flag option), then restart Docker:

{ "runtimes": { "runsc-uds-repro": { "path": "/usr/local/bin/runsc", "runtimeArgs": ["--host-uds=open", "--directfs=false"] } } }

mkdir -p /tmp/uds-repro && python3 uds_passcred_repro.py server /tmp/uds-repro        # host, leave running
for c in recv recvmsg-creds recvmsg-creds-rights; do
  docker run --rm --runtime=runc -v /tmp/uds-repro:/uds -v "$PWD":/r:ro python:3.11-slim python3 /r/uds_passcred_repro.py client /uds $c
done
for c in recv recvmsg-creds recvmsg-creds-rights; do
  docker run --rm --runtime=runsc-uds-repro -v /tmp/uds-repro:/uds -v "$PWD":/r:ro python:3.11-slim python3 /r/uds_passcred_repro.py client /uds $c
done
# control: stop the server, restart it with --no-passcred, repeat the runsc loop

(--directfs=false is only so that the sandbox's own SCM_CREDENTIALS hello is accepted by the server, see the companion credentials issue; it does not affect the receive behaviour reported here.)

Expected (runc) vs actual (runsc release-20260817.0, kvm platform); the same script was run under containerd with the same two runsc flags:

case (one client invocation each)	runc	runsc
recv: recv() without a control buffer	OK payload	OK payload
recvmsg-creds: recvmsg() with a control buffer, only host-attached credentials	OK, cmsg SCM_CREDENTIALS	EINVAL
recvmsg-creds-rights: recvmsg() with a control buffer, credentials + SCM_RIGHTS pipe	OK, both cmsgs, pipe usable	EINVAL
all three with the server started --no-passcred	OK	OK

Earlier runs with SOCK_SEQPACKET and SOCK_DGRAM socketpairs gave the same EINVAL for the two recvmsg cases. Inside the sandbox getsockopt(SO_PASSCRED) on the imported socket returns 0 (runc: 1).

uds_passcred_repro.py:

#!/usr/bin/env python3
"""Reproducer: recvmsg fails on an SCM_RIGHTS-imported Unix socket that has SO_PASSCRED enabled (gVisor).

Linux only (SO_PASSCRED / SCM_CREDENTIALS). One receive case per client invocation, so SOCK_STREAM framing cannot
mix the cases:

  server:  python3 uds_passcred_repro.py server DIR [--no-passcred]
  client:  python3 uds_passcred_repro.py client DIR {recv,recvmsg-creds,recvmsg-creds-rights}

Per client connection the server: accepts on DIR/sock (SOCK_SEQPACKET, SO_PASSCRED on the listener), reads the
client's "hello:<case>" (with SCM_CREDENTIALS), creates ONE SOCK_STREAM socketpair, enables SO_PASSCRED on both ends
(unless --no-passcred), passes one end via SCM_RIGHTS, waits for "hi" on it, then sends exactly ONE message on it:
  recv / recvmsg-creds        -> b"payload" (no ancillary data attached by the server; the kernel adds SCM_CREDENTIALS
                                 to the receiver if the socket has SO_PASSCRED)
  recvmsg-creds-rights        -> b"payload" with one pipe write-end attached via SCM_RIGHTS; the server then reports
                                 whether the client wrote through that pipe.
The client receives that one message with the selected shape and prints OK or the error.
"""
import argparse, array, os, select, socket, struct, sys

CASES = ("recv", "recvmsg-creds", "recvmsg-creds-rights")

def serve(dirpath, passcred):
    path = os.path.join(dirpath, "sock")
    try: os.unlink(path)
    except FileNotFoundError: pass
    srv = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); srv.bind(path); os.chmod(path, 0o777); srv.listen(4)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
    print(f"server: listening on {path}, SO_PASSCRED on passed sockets = {passcred}", flush=True)
    while True:
        c, _ = srv.accept(); c.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
        try:
            msg, anc, _, _ = c.recvmsg(256, socket.CMSG_SPACE(64))
            creds = [struct.unpack("3i", d[:12]) for lvl, typ, d in anc if typ == socket.SCM_CREDENTIALS]
            case = msg.decode(errors="replace").split(":", 1)[-1]
            print(f"server: client case={case!r} kernel creds (pid, uid, gid) = {creds}", flush=True)
            a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
            if passcred:
                a.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1); b.setsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED, 1)
            c.sendmsg([b"FD"], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", [b.fileno()]))]); b.close()
            a.settimeout(10); print(f"server: passed a SOCK_STREAM socket, got {a.recv(16)!r} on it", flush=True)
            if case == "recvmsg-creds-rights":
                r, w = os.pipe()
                a.sendmsg([b"payload"], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, array.array("i", [w]))]); os.close(w)
                rl, _, _ = select.select([r], [], [], 5)
                print("server: sent payload + SCM_RIGHTS pipe; pipe read:", os.read(r, 64) if rl else b"<timeout: client did not write>", flush=True); os.close(r)
            else:
                a.send(b"payload"); print("server: sent payload (no ancillary data from the server)", flush=True)
            a.close()
        except Exception as e: print("server: error", type(e).__name__, e, flush=True)
        finally: c.close(); print("server: --- done ---", flush=True)

def run_client(dirpath, case):
    s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.connect(os.path.join(dirpath, "sock"))
    s.sendmsg([f"hello:{case}".encode()], [(socket.SOL_SOCKET, socket.SCM_CREDENTIALS, struct.pack("3i", os.getpid(), os.getuid(), os.getgid()))])
    print(f"client[{case}]: uid={os.getuid()} sent SCM_CREDENTIALS ok", flush=True)
    s.settimeout(10); _, anc, _, _ = s.recvmsg(16, socket.CMSG_SPACE(64))
    fd = [x for lvl, typ, d in anc if typ == socket.SCM_RIGHTS for x in array.array("i", d).tolist()][0]
    ps = socket.socket(fileno=fd); ps.settimeout(6)
    print(f"client[{case}]: received passed socket fd={fd} type={ps.type} SO_PASSCRED as seen here={ps.getsockopt(socket.SOL_SOCKET, socket.SO_PASSCRED)}", flush=True)
    ps.send(b"hi")
    try:
        if case == "recv":
            m, anc2 = ps.recv(64), []
        elif case == "recvmsg-creds":
            m, anc2, _, _ = ps.recvmsg(64, 4096, 0)
        else:
            m, anc2, _, _ = ps.recvmsg(64, 4096, socket.MSG_CMSG_CLOEXEC)
        fds = [x for lvl, typ, d in anc2 if typ == socket.SCM_RIGHTS for x in array.array("i", d).tolist()]
        print(f"client[{case}]: OK msg={m!r} cmsg_types={[typ for _, typ, _ in anc2]} passed_fds={fds}", flush=True)
        for x in fds: os.write(x, b"written-through-passed-pipe"); os.close(x)
    except OSError as e:
        print(f"client[{case}]: FAILED {type(e).__name__}: [Errno {e.errno}] {e.strerror}", flush=True); sys.exit(1)
    finally: ps.close(); s.close()

if __name__ == "__main__":
    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = p.add_subparsers(dest="mode", required=True)
    ps_ = sub.add_parser("server"); ps_.add_argument("dir"); ps_.add_argument("--no-passcred", action="store_true", help="do not enable SO_PASSCRED on the passed socket (control)")
    pc = sub.add_parser("client"); pc.add_argument("dir"); pc.add_argument("case", choices=CASES)
    args = p.parse_args()
    serve(args.dir, not args.no_passcred) if args.mode == "server" else run_client(args.dir, args.case)

Likely source path

• pkg/sentry/socket/unix/transport/host.go: an SCM_RIGHTS-imported socket becomes a host-backed connected endpoint; SO_PASSCRED is read at init, and the receive path assumes the imported fd will not have it enabled.
• A receive with a requested control buffer passes the host control messages to pkg/unet/unet.go ExtractFDs, which calls ParseUnixRights on every message and errors on any other type; the host-attached SCM_CREDENTIALS therefore surfaces as EINVAL.
• Same on master at 80bb741 (2026-09-01) after the socket refactor; the receive and control-message behaviour is unchanged.

Impact

NVIDIA MPS: the control daemon hands each CUDA client its server connection this way, with SO_PASSCRED enabled (a runc client trace shows SCM_CREDENTIALS on every message received on it). A sandboxed CUDA client that has already authenticated (NEW CLIENT … from user 65534, NEW SERVER … Ready in the daemon log) fails on the first recvmsg of that passed socket and reports CUDA error 805 "MPS client failed to connect"; runsc strace: recvmsg(0x9 host:[7], {…, control_len=4096}, MSG_CMSG_CLOEXEC) = -1 errno=22. The MPS sockets visible in the daemon's network namespace are SOCK_SEQPACKET; the minimal SOCK_SEQPACKET case produces the same EINVAL.

Environment

$ runsc --version
runsc version release-20260817.0
spec: 1.2.1

runsc release-20260817.0 (spec 1.2.1), platform kvm, --host-uds=open --directfs=false; validated under containerd 2.1.4 (io.containerd.runsc.v1 shim) on Kubernetes 1.33.4, Ubuntu 22.04, no user namespaces; the Docker commands above follow Docker's alternative-runtime configuration. Client uid does not matter for this bug (root and uid 1000 behave the same).

Ask

When the imported socket is presented to the sandbox with SO_PASSCRED=0, ignore host-attached SCM_CREDENTIALS while continuing to extract and deliver SCM_RIGHTS. Alternatively, preserve the inherited SO_PASSCRED state and translate the credentials into sandbox-visible credentials. Either would be preferable to returning EINVAL for otherwise valid data. Happy to test a patch.