[Bug] #UD exception on VMClock IRQ introduced by v1.15.0

[abarisani]

Hi.

I am using our bare metal Go microvm library to launch our example unikernel and I noticed that it was broken by b154577

What I am experiencing since that commit is CPU landing in our invalid opcode handler.

So it seems that the new VMClock GSI (which I presume is 6) is being treated as exception #6 rather than being handled by the IOAPIC correctly.

This might be of course a fault in our code, however note that:

• with v1.14.3 this doesn't happen, as VMClock is not raising IRQs, but also note that VirtIO GSIs are correctly handled.
• with qemu, cloud_hypervisor the same IOAPIC and IRQ handling code does not have issues

Am I missing some configuration aspect to map GSI ids correctly?

I am using a very simple configuration:

{
  "boot-source": {
    "kernel_image_path": "example"
  },
  "network-interfaces": [
    {
      "iface_id": "net1",
      "guest_mac": "06:00:AC:10:00:02",
      "host_dev_name": "tap0"
    }
  ],
  "machine-config": {
    "vcpu_count": 1,
    "mem_size_mib": 4096,
    "smt": false,
    "track_dirty_pages": false,
    "huge_pages": "None",
    "gdb_socket_path": "/tmp/gdb.socket"
  },
"drives": [
    {
        "drive_id": "dummy",
        "path_on_host": "/dev/null",
        "is_root_device": false,
        "is_read_only": false
    }
  ]
}

Thanks

[zulinx86]

Hi @abarisani

Thank you for the report and sorry for the delay.

As noted in the CHANGELOG, the addition of VMClock introduced one extra GSI.

https://github.com/firecracker-microvm/firecracker/blob/main/CHANGELOG.md#1150

#5564: which added support for VMClock, uses one extra GSI for the VMClock device itself which reduces the available GSIs for VirtIO devices. New maximum values is 92 devices on Aarch64 and 17 devices on x86.

Also, the number of GSIs to be allocated depends on how many MMIO devices you attach. For example, if you attach only one disk and launch a microVM without PCI enabled, the allocation is as follows:

• GSI 1 => i8042
• GSI 4 => serial port
• GSI 5 => disk
• GSI 6 => VMGenID device
• GSI 7 => VMClock device

# cat /proc/interrupts 
           CPU0       CPU1       
 24:          0          0   IO-APIC   6-edge      ACPI:Ged
 25:          0          0   IO-APIC   7-edge      ACPI:Ged
 26:        763          0   IO-APIC   5-edge      virtio0
 27:        309          0   IO-APIC   4-edge      ttyS0
 28:          0          8   IO-APIC   1-edge      i8042

We haven't seen such an issue with our guest Linux kernels so far. It appears you are using your own custom unikernel, so I think you'll need to make it possible to handle the extra GSI on its end.

Thanks,

[abarisani]

What I don't understand is why the IOAPIC is not able to redirect correctly the GSI, clearly I shouldn't get it as exception 6 (undefined instruction) right? Thanks!

[abarisani]

Disabling the Master PIC with reg.Out8(0x21, 0xff) solves the UD exception.