Firecracker networking improvements

[bchalios]

This is a tracker for investigating improvements in the Firecracker networking stack. We are mainly looking to improve
performance (throughput, latency and CPU utilization) of the emulation logic. However, we want to keep an eye on simplifying the control plane.

Currently, we want to focus our efforts towards a vhost-user-net data-plane, however, we will evaluate other alternatives along the way.

Additional context

Related issues:

Macvtap: #1933
vhost-net: #3707, #4312

Checks

• Have you searched the Firecracker Issues database for similar requests?
• Have you read all the existing relevant Firecracker documentation?
• Have you read and understood Firecracker's core tenets?

[sagoresarker]

I think it can extend its functionality by not relying solely on TUN/TAP.

[DemiMarie]

I think the most important change here is to move away from the Linux tap interface to something with much better performance. Tap devices cannot support a large number of packets per second, whereas vhost-net, XDP, and other solutions can.

[bchalios]

Hello all.

I wanted to give an update on our work on this front and present some results regarding the improvements we shipped with release v1.10.

We implemented three optimizations:

• Generic VirtIO queue optimizations: We removed a number of redundant safety checks in our emulation code. These checks are present in the upstream vm-memory crate, but they’re redundant in our logic, since we already make sure that all the safety properties are upheld in downstream Firecracker code.
• Support VirtIO net mergeable buffers: This VirtIO feature allows the guest driver to perform smaller memory allocations, wasting less memory in the VirtIO queue, and increasing efficiency of memory allocations on the guest side.
• Avoid a memory copy in the RX path: Before this optimization, we were performing two memory copies for every ethernet frame we received; one copy from the TAP device to a Firecracker buffer and a second one from the Firecracker buffer to guest memory. Now, we use the readv system call to read directly from the TAP device in guest memory.

Results

Applying these optimizations TCP throughput test reports an increase of 20% for RX and 10% for TX throughput average across all the combinations of instances, guest and host kernels Firecracker currently supports.

In more detail, the per-instance type improvements are:

Instance type	RX throughput improvement (%)	TX throughput improvement (%)
m5n	12.0	9.6
c5n	8.2	7.6
m6i	15.2	18.0
m6g	38.1	2.1
m7g	24.7	11.3

Here, you can see a comparison of the absolute numbers for TCP throughput between v1.9.1 and v1.10.0 Firecracker releases, for RX:

and TX:

These are the results from our throughput test. Briefly, this test uses iperf3 to measure TCP thoughput between a guest and the host it is running on both for TX and RX. It tests this for all our supported instance types and kernel configurations. The test uses microVM configurations with 1 and 2 vCPUs.

Apart from these optimizations, we have considered in the past vhost-net (#3707) and vhost-user-net. We discarded the former due to security concerns regarding a direct guest to host kernel attack surface which didn't match our security bar. For the latter, we realized that the bulk of the work there lies in the implementation of the back-end which is a responsibility that lives outside Firecracker, limiting the impacts which we can make for our users. For this reason, we decided to de-prioritize it (at least) for the time being.

As a result, we focused our efforts on improving the current back-end, which seemed more feasible according to our time frame.

I will close this issue as we do not plan to work on a specific optimization on this front in the near future. As always, we welcome the community to open issues or PRs with specific requests and ideas.