Docker support in the VM

[ddelbondio]

I would like to use microsandbox to run local AI coding agents in an fully isolated way. Generally my tests are very promising.

Getting docker to work in the VMs itself is quite challenging though. Is this something you see as goal for this project to allow this?

Problems I have found so far:

• The kernel from libkrunfw is missing support for the entire network setup required by docker (no netfilter/nft, no briding, etc). Looking at the way upstream handles kernel flags, they seem to be super conscious about kernel size, so we might have to patch their kernel config downstream
• Since docker uses overlayfs, the current mount setup breaks docker, since we can't layer the docker overlayfs on the root overlayfs. This is easily fixable by using fuse-overlayfs. It would feel a bit cleaner though if we could create volumes as virtual block devices though.

Is this something worth following up on? Would it be possible to handle a different kernel and the block device with a plugin later?

[appcypher]

The kernel from libkrunfw is missing support for the entire network setup required by docker (no netfilter/nft, no briding, etc). Looking at the way upstream handles kernel flags, they seem to be super conscious about kernel size, so we might have to patch their kernel config downstream

This is the second ask for netfilter so we will probably enable that in the kernel config.

Since docker uses overlayfs, the current mount setup breaks docker, since we can't layer the docker overlayfs on the root overlayfs. This is easily fixable by using fuse-overlayfs. It would feel a bit cleaner though if we could create volumes as virtual block devices though.

The current v0.3 versions use host-side overlayfs implementation which has a lot of issues. We have removed that in main, in favor of image-backed OCI layers and we now use the guest-side kernel overlayfs for performance and compatibility. This should solve the filesystem issue.

[ddelbondio]

This is the second ask for netfilter so we will probably enable that in the kernel config.

I had to enable 50 different network related flags to make docker happy (I can provide them if you want). I like the idea from #600 of just being table to provide our own kernel.

We have removed that in main, in favor of image-backed OCI layers and we now use the guest-side kernel overlayfs for performance and compatibility.

I am actually testing with main, and the problem persists in the current version. I am not exactly sure what docker is doing there exactly. I have never seen docker creating a separate mount directly in /var/lib/docker.

The mounts looks like this:

/                  overlay                  overlay  rw,relatime,lowerdir=/.msb/rootfs/lower,upperdir=/.msb/rootfs/upperfs/upper,workdir=/.msb/rootfs/upperfs/work,uuid=on
├─/.msb            msb_runtime              virtiofs rw,relatime
...
└─/var/lib/docker  overlay[/var/lib/docker] overlay  rw,relatime,lowerdir=/.msb/rootfs/lower,upperdir=/.msb/rootfs/upperfs/upper,workdir=/.msb/rootfs/upperfs/work,uuid=on

Container start fails because docker wants to create an overlay where upperdir points to another overlayfs (which as far as I understand is not supported at all)

overlay: filesystem on /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/3/work not supported as upperdir

As I said, this is can be easily worked around with fuse-overlayfs. But being able to have a block device as volume backed by a file in the host system might make sense. Then I could just mount that directly to /var/lib/docker.

[appcypher]

I had to enable 50 different network related flags to make docker happy (I can provide them if you want). I like the idea from #600 of just being table to provide our own kernel.

Please do. I raised a PR for it already btw. Anything I'm missing?

We are also making it easier to specify your own libkrunfw library. We already support setting it in the global config btw.

Container start fails because docker wants to create an overlay where upperdir points to another overlayfs (which as far as I understand is not supported at all)

I see.

But being able to have a block device as volume backed by a file in the host system might make sense. Then I could just mount that directly to /var/lib/docker.

You are right! That's a gap and I'm fixing it right now.

[appcypher]

I forgot to mention. Docker works on the latest version, 0.4.3.

msb run \
  --memory 2G \
  --entrypoint sh \
  docker:dind \
  -- -lc '
    dockerd --data-root=/tmp/docker >/tmp/dockerd.log 2>&1 &
    for i in $(seq 1 60); do
      docker info >/dev/null 2>&1 && exec docker run --rm hello-world
      sleep 1
    done
    cat /tmp/dockerd.log
    exit 1
  '

Right now, I still have to set the docker data root to /tmp because /tmp is auto mounted as tmpfs in microsandbox.

In a future release (0.4.4 probably) with proper mounted disk image, the data root will be persisted across restarts.