mana: save and restore mana devices when keepalive is enabled

[justus-camp-microsoft]

This adds the code that utilizes GdmaDevice::restore and VfioDevice::restore in cases where MANA keepalive is enabled. This requires a GuestServicingFlag to be set to enable MANA keepalive, a command line parameter of OPENHCL_MANA_KEEP_ALIVE=1, and for OPENHCL_ENABLE_VTL2_GPA_POOL to be set with enough memory for keepalive to function.

I've also modified the interactive console's service-vtl2 to take arguments for --mana-keepalive and --nvme-keepalive so that keepalive can be manually tested with the console.

[Copilot]

Pull Request Overview

This PR implements MANA keepalive functionality for OpenHCL, allowing MANA device state and DMA memory to be preserved across servicing operations. This builds on the existing keepalive infrastructure by adding MANA-specific save/restore capabilities alongside the previously implemented NVMe keepalive feature.

Key changes:

• Added comprehensive save/restore functionality for MANA devices including driver state and memory preservation
• Extended command-line and flag support for MANA keepalive configuration
• Added test coverage for MANA keepalive functionality

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vmm_tests/vmm_tests/tests/tests/x86_64/openhcl_linux_direct.rs	Removed duplicate MANA servicing test to avoid conflicts
vmm_tests/vmm_tests/tests/tests/multiarch/openhcl_servicing.rs	Added MANA NIC validation function and new test for keepalive functionality
vm/devices/net/net_mana/src/lib.rs	Updated test calls to include new mana_state parameter
vm/devices/net/mana_driver/src/save_restore.rs	Added ManaSavedState and ManaDeviceSavedState protobuf structures
vm/devices/net/mana_driver/src/mana.rs	Added save method and state restoration logic to ManaDevice
vm/devices/net/mana_driver/src/gdma_driver.rs	Removed dead_code attributes from save/restore methods
vm/devices/get/guest_emulation_device/src/lib.rs	Added mana_keepalive flag support to GED capabilities
vm/devices/get/get_resources/src/lib.rs	Added mana_keepalive field to GuestServicingFlags
vm/devices/get/get_protocol/src/lib.rs	Added enable_mana_keepalive bit to SaveGuestVtl2StateFlags
petri/src/worker.rs	Added mana_keepalive field mapping for servicing flags
petri/src/vm/mod.rs	Added enable_mana_keepalive field to OpenHclServicingFlags
openvmm/openvmm_entry/src/lib.rs	Added CLI support for mana-keepalive parameter
openhcl/underhill_core/src/worker.rs	Integrated MANA state handling into VM lifecycle management
openhcl/underhill_core/src/servicing.rs	Added mana_state field to servicing state structures
openhcl/underhill_core/src/options.rs	Added OPENHCL_MANA_KEEP_ALIVE environment variable support
openhcl/underhill_core/src/lib.rs	Wired mana_keep_alive option through worker configuration
openhcl/underhill_core/src/emuplat/netvsp.rs	Implemented comprehensive MANA device save/restore in VF manager
openhcl/underhill_core/src/dispatch/mod.rs	Added network settings save method and MANA state coordination

Comments suppressed due to low confidence (1)

openhcl/underhill_core/src/emuplat/netvsp.rs:1

• Using expect() can cause panic if the memory with matching PFN is not found. Consider returning a proper error instead of panicking in production code.

// Copyright (c) Microsoft Corporation.

[Copilot]

Using unwrap() on Arc::into_inner() can panic if there are multiple references to the Arc. Consider using a safer approach with proper error handling or ensuring single ownership before this call.

Suggested change

	let inner = Arc::into_inner(self.inner).unwrap();
	let inner = match Arc::into_inner(self.inner) {
	Some(inner) => inner,
	None => {
	tracing::error!("Failed to save MANA device state: multiple references to device exist");
	return (
	Err(anyhow::anyhow!("Failed to save MANA device state: multiple references to device exist")),
	// We cannot recover the device, so return a default error value.
	// This assumes T: Default, otherwise consider another approach.
	// For now, use std::mem::zeroed() as a placeholder (unsafe).
	unsafe { std::mem::zeroed() },
	);
	}
	};

[Copilot]

Using std::mem::forget() prevents the device destructor from running, which could lead to resource leaks. Consider documenting why this is necessary or finding an alternative approach that properly manages the device lifetime.

Suggested change

std::mem::forget(device);

[mattkur]

Have you given thought to how you will validate that servicing used keepalive, rather than falling back to the non-KA path?

In addition, please consider adding cases for going to/from downlevel.

[justus-camp-microsoft]

That's a fair question, and I actually ran into a race condition during manual testing where it looked like we successfully went through the save/restore path but actually hadn't.

For a short answer, I'm not sure how to validate that we went through the codepath we think we did without parsing some logs (do we see a gdma restoration message etc.). How is NVMe validating this?

[mattkur]

@gurasinghMS can give you some pointers. But in short, we inject faults into the nvme emulator, such that a restore would never succeed if keepalive isn't used. In addition, I'm adding checks that look at the driver state in VTL2, so that we can ascertain if it thinks it will go down the right path (this is more akin to look at logs, as you say)

[gurasinghMS]

For the nvme-keepalive side, we ended up just forking out the nvme-emulator in to a test-only emulator where we added hooks to fault certain functions (This was much easier than adding test hooks alongside the existing implementation).
To test that keepalive was being used I added tests that would check for command activity during restore. In our case, the driver should never be issuing CREATE_IO_QUEUE commands during restore. Happy to discuss more offline too if you would like

[mattkur]

Presumably you need some check here (or at a higher layer?) to gracefully fail the save if this memory is not from a persistent pool. (You don't have that primitive from the DMA APIs yet, ofc).

[justus-camp-microsoft]

The DMA client usage is definitely very optimistic/fail-fast in this PR. I need to take another pass and think more critically about how to handle failures here or if we're going to pass that responsibility onto the DMA APIs somehow. Do you have thoughts here?

I know the DmaClient will error if all marked persisted memory isn't restored, but saving the memory this way I don't think there's any way to to check that we're not saving some non-persisted memory. Maybe DmaClient should have some method that spits out a common MemorySavedState that can be passed into persisted state and it can double check that it's persisted memory? Not sure

[mattkur]

You'd check at allocation time. See my suggested change in #2087 .

[mattkur]

Nice. Few questions to get started.

[mattkur]

This code will need to be more discerning here:

• Use the DMA client with persistent allocations for the memory regions you will save/restore
• Use a new DMA client (without persistent allocations) for the memory regions you will not save/restore

The private pool is a limited resource, and should only be used for memory that needs to come from it. @chris-oo FYI.