[Feature Request] Asynchronous On-demand FD-passing for virtio-pmem via userfaultfd

[joy-allen]

Feature Request

Standard virtio-pmem backends in Firecracker are typically backed by local files. However, in high-density container environments (e.g., using Nydus or EROFS-based lazy loading), image data is often stored in remote registries and fetched on-demand.

Currently, achieving "lazy-loading" in Firecracker requires either virtio-block (which lacks DAX and causes memory-heavy double-buffering) or complex vhost-user-fs setups. This feature request proposes an asynchronous, on-demand FD-passing backend for virtio-pmem using userfaultfd. This allows MicroVMs to start instantly while sharing the Host's page cache with zero-copy overhead, perfectly aligning with Firecracker's performance and security philosophy.

Describe the desired solution

We propose a new UffdBackend for the virtio-pmem device that leverages userfaultfd (UFFD) and Unix Domain Socket (UDS) file descriptor passing.

Architecture

1. UFFD Registration: The VMM creates an anonymous RO mapping for the virtio-pmem address space and registers it with userfaultfd.
2. NBD-style Control Plane: The VMM communicates with an external image service (e.g., nydusd) via UDS using a lightweight, asynchronous protocol inspired by NBD.
   • PROBE: Queries the service for existing local chunks at boot to perform initial mappings.
   • FETCH: Triggered by a UFFD page fault; the VMM sends a request with (pos, len) and returns to the event loop.
3. FD-Passing & Remapping: The service replies using SCM_RIGHTS to pass File Descriptors. The VMM receiver thread then performs an mmap(MAP_FIXED) to map the blob-backed FD into the Guest's address space.
4. Zero-Copy Execution: Once mapped, the Guest accesses the data via DAX (e.g., EROFS + DAX). This bypasses the VMM entirely for subsequent reads, sharing the Host Page Cache directly.

Protocol Definition

• Request: magic (u32), type (u32), handle (u64), pos (u64), len (u32)
• Reply: magic (u32), code (u32), handle (u64), dev_sz (u64), ranges_count (u32)
• Control Message: [fd, off, len, dev_off] (passed via SCM_RIGHTS)

Describe possible alternatives

1. Virtio-fs with DAX

While virtio-fs is a common solution, it is significantly more complex to implement and audit for Firecracker. It requires a stateful FUSE control plane and a dynamic DAX window management protocol (SetupMapping/RemoveMapping). Our virtio-pmem approach treats the image as a static, flat address space, keeping the VMM implementation stateless and minimizing the attack surface.

2. Standard virtio-block

virtio-block lacks DAX support. Every block access requires a VM-exit and results in "double-buffering" (data exists in both the Host and Guest page caches), which significantly increases memory pressure in high-density deployments.

3. UFFDIO_COPY

The VMM could fetch data into a userspace buffer and use UFFDIO_COPY. However, this introduces an unnecessary memcpy and higher CPU overhead compared to the proposed mmap + FD-passing approach, which natively shares the page cache.

How do you work around not having this feature?

Currently, we have to pre-fetch the entire container image before VM startup to use virtio-pmem, which leads to high "Time To First Byte" (TTFB) and wastes local storage/memory if only a fraction of the image is actually accessed by the Guest.

Additional context

1. Prototype Status

We have already implemented a functional prototype of this mechanism. The VMM is capable of:

• Handling userfaultfd events in a dedicated thread.
• Communicating with a modified nydusd via the NBD-style UDS protocol.
• Performing mmap(MAP_FIXED) with FD passing and successfully waking up the Guest vCPU.

2. Failure Handling & Resilience

A critical concern for this feature is the stability of the UDS connection. Our implementation includes:

• Auto-reconnect: The VMM can transparently re-establish the connection to the image service (e.g., if nydusd restarts).

3. Use Case: Large-scale Serverless & AI Inference

In serverless environments, MicroVMs are short-lived but require instant access to large base images (often several GBs for AI models or thick containers). This feature enables:

• Shared Host Page Cache: Multiple MicroVMs using the same base image will map the same physical pages on the Host, significantly reducing the memory footprint (RSS) compared to virtio-block.
• Reduced Cold Start Latency: By only fetching the metadata and the entry-point code chunks, the effective "Ready" time is decoupled from the total image size.

4. Comparison with Standard UFFD usage

Unlike the traditional UFFDIO_COPY approach used in post-copy migration, this "FD-passing" approach is uniquely suited for read-only DAX devices. It treats the VMM as a coordinator rather than a data buffer, which is consistent with Firecracker's goal of being a "thin" VMM.

Checks

• Have you searched the Firecracker Issues database for similar requests?
• Have you read all the existing relevant Firecracker documentation?
• Have you read and understood Firecracker's core tenets?

[hsiangkao]

btw, @ChengyuZhu6 is now working on another approch to support vmdk format instead to pass in static mapping for virtio-blk and virtio-pmem too, personally I think both two ways are unconflicted, and hopefully @ChengyuZhu6 could raise a PR soon for virtio-blk vmdk support first.

[ShadowCurse]

Hi @joy-allen, @hsiangkao, thank you for opening this feature request.
The approach you propose is quite interesting, but I see a couple of issues:

• Having a pmem specific machinery for UFFD handling is not ideal
• Implementing a custom protocol for a specific external service is also not something we would like to have
• Spinning a separate thread also seems redundant. UFFD can be added to VMM thread.

After spending some time thinking about the general problem you are trying to solve, it seems it comes down to: when guest page faults, you want to have a no-copy UFFD-like behavior that just mmaps the fd instead of doing a copy.
My current thoughts on possible implementation are:
If handling of UFFD be made external to FC, then the external service can receive UFFD fd and track page faults by itself. And when page fault occurs, it can send a Firecracker a message like the one you propose with fd, and FC will mmap it in the guest address space. This can way this mechanism can act not only on pmem, but on the whole guest physical memory range. Firecracker already supports UFFD for normal guest RAM, so maybe you can extend it to also support pmem and allow for the back to FC communication path for this mmap operation.

What do you think about this?

[joy-allen]

Hi @ShadowCurse , thanks a lot for the detailed feedback — it’s very helpful and I largely agree with your direction (making this reusable across the guest memory range rather than pmem-specific).

Next, we’ll take a closer look at how this could be built by extending Firecracker’s existing UFFD support for normal guest RAM, and see what the minimal changes in FC would look like.

Regarding the external → Firecracker part you mentioned (“when a page fault occurs, the external service can send Firecracker a message … and FC will mmap it in the guest address space”): do you have any thoughts on what an acceptable protocol for that message should look like?

[ShadowCurse]

For the protocol, Firecracker does not support any specific ones. The UFFD handshake is done here: https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/persist.rs#L588 and it is just Firecracker sending a json payload with mapping information and the UFFD fd to the unix socket. I guess it would be better for consistency to keep the json payload format, keep the socket open and add it to vmm event_manager. Then wait for the json payload in a form { "guest_addr": ..., "len": ... } with fd attached to the message.

[joy-allen]

@ShadowCurse Thanks for the guidance. I'm going to prototype the “external service drives mapping, but VMM performs the actual mmap into guest memory” approach first, to validate feasibility and performance.

Prototype (Kata Runtime + External store + Firecracker VMM)

                +---------------------------+
                |        Firecracker VMM    |
                |  event loop:              |
                |    recv [fd,off,len,gpa]  |
                |    mmap(fd range)         |
                |    UFFDIO_WAKE(gpa range) |
                +-------------^-------------+
                              | unix sock (json + SCM_RIGHTS)
                              |
+-----------------------------+------------------------------+
|                     Kata Runtime (UFFD)                    |
|  register UFFD (MISSING) on guest memory range             |
|  on PAGEFAULT(gpa):                                        |
|    request chunk from External Service                     |
|    recv [fd,off,len,gpa] and forward to VMM                |
+-----------------------------^------------------------------+
                              | unix sock / nbd
                              |
                +-------------+--------------+
                |      External Store        |
                |  fetch chunk (remote/local)|
                |  return [fd,off,len,gpa]   |
                +----------------------------+