Randomly checkpoint inconsistent: missing or broken bitmap

[rdziwinski]

Version used
2.26

Describe the bug
I use virtnbdbackup at several nodes, and many VM are backed up correctly, but sometimes I get the error:

Redefining checkpoint failed: [virtnbdbackup.0]: checkpoint inconsistent: missing or broken bitmap 'virtnbdbackup.0' for disk 'vda'

That happens randomly in about 5-10% of VMs. I can't find any common features between the problematic VMs. I checked if the VMs were migrated or unexpected shutdown, but not.

2025-03-31 06:23:41,803 INFO: Sukces: one-2080
2025-04-01 04:25:32,964 INFO: Sukces: one-2080
2025-04-02 04:33:52,778 ERROR: Error: one-2080
2025-04-03 04:45:41,164 ERROR: Error: one-2080
2025-04-04 04:57:24,819 ERROR: Error: one-2080
2025-04-05 04:22:37,996 ERROR: Error: one-2080
2025-04-06 04:45:41,329 ERROR: Error: one-2080
2025-04-07 11:02:51,625 ERROR: Error: one-2080
2025-04-08 04:25:08,236 ERROR: Error: one-2080
2025-04-09 04:20:16,532 INFO: Sukces: one-2080
2025-04-10 04:53:34,550 INFO: Sukces: one-2080
2025-04-11 04:43:03,115 ERROR: Error: one-2080

Command what I run:

/usr/local/bin/virtnbdbackup -d one-2080 -o /srv/backup/2025-W15/one-2080 -l auto --noprogress --checkpointdir /srv/backup/checkpoints/one-2080 --verbose

Content of backups and checkpoints directory:

$ ls -lah /srv/backup/2025-W15/one-2080
total 8.0G
drwxr-xr-x   2 root root 4.0K Apr 11 14:04 .
drwxr-xr-x 508 root root  16K Apr 11 05:21 ..
-rw-r--r--   1 root root 3.9K Apr  8 22:05 backup.auto.04082025220422.log
-rw-r--r--   1 root root 4.1K Apr  9 04:20 backup.auto.04092025042003.log
-rw-r--r--   1 root root 4.1K Apr 10 04:53 backup.auto.04102025045325.log
-rw-r--r--   1 root root 1.7K Apr 11 04:43 backup.auto.04112025044302.log
-rw-r--r--   1 root root 2.9K Apr 11 14:04 backup.auto.04112025140442.log
-rw-r--r--   1 root root   57 Apr 10 04:53 one-2080.cpt
-rw-r--r--   1 root root 6.6G Apr  8 22:05 vda.full.data
-rw-r--r--   1 root root   10 Apr  8 22:05 vda.full.data.chksum
-rw-r--r--   1 root root 247M Apr  9 04:20 vda.inc.virtnbdbackup.1.data
-rw-r--r--   1 root root   10 Apr  9 04:20 vda.inc.virtnbdbackup.1.data.chksum
-rw-r--r--   1 root root 1.2G Apr 10 04:53 vda.inc.virtnbdbackup.2.data
-rw-r--r--   1 root root   10 Apr 10 04:53 vda.inc.virtnbdbackup.2.data.chksum
-rw-r--r--   1 root root  488 Apr  8 22:05 vda.virtnbdbackup.0.qcow.json
-rw-r--r--   1 root root  488 Apr  9 04:20 vda.virtnbdbackup.1.qcow.json
-rw-r--r--   1 root root  488 Apr 10 04:53 vda.virtnbdbackup.2.qcow.json
-rw-r--r--   1 root root 6.5K Apr  8 22:05 vmconfig.virtnbdbackup.0.xml
-rw-r--r--   1 root root 6.5K Apr  9 04:20 vmconfig.virtnbdbackup.1.xml
-rw-r--r--   1 root root 6.5K Apr 10 04:53 vmconfig.virtnbdbackup.2.xml
$  ls -lah /srv/backup/checkpoints/one-2080
total 52K
drwxr-xr-x   2 root root 4.0K Apr 10 04:53 .
drwxr-xr-x 661 root root  20K Apr 11 05:21 ..
-rw-r--r--   1 root root 6.2K Apr  8 22:04 virtnbdbackup.0.xml
-rw-r--r--   1 root root 6.2K Apr  9 04:20 virtnbdbackup.1.xml
-rw-r--r--   1 root root 6.2K Apr 10 04:53 virtnbdbackup.2.xml

But there are no checkpoints:

$ virsh checkpoint-list one-2080
 Name   Creation Time
-----------------------

Expected behavior
Could you provide some troubleshooting steps to help resolve the problem?

Hypervisor and/or virtual machine information:

• OS: Debian 12.9
• HV type: OpenNebula
• Virtual machine configuration [virsh dumpxml ..]

<domain type='kvm' id='356'>
  <name>XXXX/name>
  <uuid>XXXX</uuid>
  <title>XXXX</title>
  <maxMemory slots='8' unit='KiB'>33554432</maxMemory>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static' current='4'>16</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <shares>100</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <numa>
      <cell id='0' cpus='0-15' memory='8388608' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm-one</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap' iothread='1'/>
      <source file='/var/lib/one//datastores/167/2080/disk.0' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/one//datastores/167/2080/disk.1' index='1'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='4' iothread='2'/>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='XXXX'/>
      <source bridge='ovs-br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='XXXX/>
      </virtualport>
      <bandwidth>
        <inbound average='125000' peak='125000' burst='125000'/>
        <outbound average='125000' peak='125000' burst='125000'/>
      </bandwidth>
      <target dev='one-2080-0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='XXXX'/>
      <source bridge='ovs-br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='XXXX'/>
      </virtualport>
      <bandwidth>
        <inbound average='125000' peak='125000' burst='125000'/>
        <outbound average='125000' peak='125000' burst='125000'/>
      </bandwidth>
      <target dev='one-2080-1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-356-one-2080/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='7980' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>XXXX</label>
    <imagelabel>XXXX</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+9869:+9869</label>
    <imagelabel>+9869:+9869</imagelabel>
  </seclabel>
</domain>

Logfiles:

2025-04-11 04:43:03,115 ERROR: stderr: [2025-04-11 04:43:02] INFO lib common - printVersion [main]:  Version: 2.21 Arguments: /usr/local/bin/virtnbdbackup -d one-2080 -o /srv/backup/2025-W15/one-2080 -l auto --noprogress --checkpointdir /srv/backup/checkpoints/one-2080
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  Backup level: [auto]
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  Store checkpoints in: [/srv/backup/checkpoints/one-2080]
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  Libvirt library version: [9000000]
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  NBD library version: [1.14.2]
[2025-04-11 04:43:02] INFO root check - targetDir [main]:  Backup mode auto: executing incremental backup.
[2025-04-11 04:43:02] INFO root disktype - Optical [main]:  Skipping attached [cdrom] device: [hda].
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  Backup will save [1] attached disks.
[2025-04-11 04:43:02] INFO root virtnbdbackup - main [main]:  Concurrent backup processes: [1]
[2025-04-11 04:43:02] INFO root checkpoint - create [main]:  Loading checkpoints from: [/srv/backup/2025-W15/one-2080/one-2080.cpt]
[2025-04-11 04:43:03] INFO root checkpoint - redefine [main]:  Loading checkpoint list from: [/srv/backup/checkpoints/one-2080]
[2025-04-11 04:43:03] INFO root checkpoint - redefine [main]:  Redefine missing checkpoint: [virtnbdbackup.0]
[2025-04-11 04:43:03] ERROR root checkpoint - redefine [main]:  Redefining checkpoint failed: [virtnbdbackup.0]: checkpoint inconsistent: missing or broken bitmap 'virtnbdbackup.0' for disk 'vda'
[2025-04-11 04:43:03] ERROR root virtnbdbackup - main [main]:  Failed to redefine checkpoints.

Workaround:
After removing backups and checkpoints dirs backups become possible.

[abbbi]

This can happen if vm is powered off without qemu beeing able to write the bitmaps or if vm is migratet to another host, check readme in this case, or your checkpoint directory isnt in sync with the checkpoints

[abbbi]

Or some other component Alters the vm disks images (resize)? In between backups

[rdziwinski]

Hi, I read the readme. And none of the things mentioned therein happened. Disk didn't resize either.
Directories with checkpoints and backups are nfs mounts.

[abbbi]

Use qemu-img to check bitmap state.. no complains by other users so I guess it’s a issue caused by other components

[abbbi]

maybe opennebula re-deploys (undefine/define) virtual machine each time it is powered off and as such checkpoint/bitmap information is lost.
if libvirt virsh checkpoint-list shows no checkpoints, there must have been something happening on libvirt side.

[gitthangbaby]

Happens to me and it correlates to sudden turn off of the VM (sudden freeze of the server). So I understand the explanation.
I wish turning to full mode deleted the files and checkpoints. No point adding such a huge snapshot. Not only in this situation, but generally. I have to implement a wrapper doing this:

• run auto mode every day
• run full mode (after a cleanup) every week

This way I have a weekly rotation. Which is great to keep NVME wear to minimum, but not great to have as many backups as possible, it's only up to 7 (from 1 to 7). I have no space to do what "auto" mode intents to do in the documentation - make a new folder and continue snapshots.