grub-install doesn't work with ZFS pools installed on partitions

[cyphar]

I opened an issue for this in the zfsonlinux/grub project, but given that all of the OpenZFS documentation tells you to configure ZFS on separate disk partitions (bpool, rpool, and the ESP partition), it is kind of important to note that grub-install will fail to install on any such /boot (based on my testing this happens even if /boot is not ZFS -- grub2-install will effectively grub2-probe all of the filesystems on the disk).

It appears that someone has made this work recently, but they used LVM-managed RAID to configure the device for /boot (which then used XFS, not ZFS) and that appeared to work for grub2-install.

Unless I'm missing something, it seems that the only really practical solution is to use systemd-boot (though I'm still working on testing this), which means storing the kernels and initrd in the ESP partition (/boot/efi) and skipping all of standard /boot grub shenanigans entirely. Maybe there should be some kind of caution about this issue, as debugging this took me quite a long time and there really isn't any documentation about how painful this all actually is. Another option is ZFSBootMenu but I'm not sure how I feel about using something completely unsupported by distros to boot them.

[gmelikov]

I think I'm missing something. But for several years for Debian we use partitions as main way to install root-on-zfs https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Bookworm%20Root%20on%20ZFS.html , and grub-install works well. Yes, grub-probe won't see root ZFS pool with additional not-supported features, but looks like on Debian grub-install won't try to do so.

Do I assume correctly that you use openSUSE? I've found some grub-install differences with it some time ago #258 , but I'm not a user of openSUSE and I didn't have enough fuel to find what's the difference, ubuntu/debian (which I use extensively) doesn't have such problems. Is it correct to speak that this issue applies only on openSUSE?

[cyphar]

I built grub from the upstream sources and the issue is the same (I outlined it in the bug report I linked -- zfsonlinux/grub#29).

Basically grub2-probe's internal ZFS parser thinks there is a checksum error if you run it on any ZFS filesystem that is from a zpool on a disk partition. No such error happens if you are dealing with a zpool using the whole disk. This happens even if the partition is partition 1 and it allocates the entire disk (well, minus the standard padding that gdisk adds). grub2-install has the same error and grub2-install -vv confirms that it is the same cause. This isn't a problem with featuresets either, as the only variable that changes between my tests is whether you use zpool create foo /dev/sda or zpool create foo /dev/sda1 (plus all of the -o flags, including -o compatibility=grub2).

Maybe Deban and/or Ubuntu have some patches to fix this behaviour, but upstream grub (and openSUSE) absolutely does not work with the setup described in the guide. My suspicion is there is some bug in the offset calculation code, but the code is quite horrifically complicated and I think a more seasoned ZFS developer would need to look at it to find where the bug is.

This also happens even if /boot is not on ZFS! This blog post mentions (and my surface-level testing concurs) that GRUB appears to scan other filesystems even if they aren't the ones used for /boot (the documentation here also appears to corroborate this and tells you to use -o compatibility=grub2 even for rpool, though it doesn't explicitly spell it out and is a bit self-contradictory as the justification for creating a separate bpool was that you don't need to restrict features for rpool).

The ZFS parser code in GRUB does not appear to have been updated recently, so this behaviour is fairly longstanding. I haven't tried to do this on Debian or Ubuntu (where it presumably works) but it can't be an openSUSE-specific issue because it also happens with upstream (unpatched) grub.