akmods module fails to install zfs

[ruzko]

background

the zfs kmod is not in the ublue-os akmods copr repo.
The akmods module script doesn't handle zfs separately, so zfs kmod sources don't get downloaded even if zfs is listed in the akmod module in the recipe. This causes bluebuild to fail.

The documentation explains briefly how to handle akmods that are in a different repo than the ublue-os akmods copr repo, but it remains unclear to me how this will work for the zfs kmod.

Do I add a rpm-ostree module to install http://download.zfsonlinux.org/fedora/44/zfs-release-3-1.fc44.noarch.rpm, which puts zfs.repo into /etc/yum.repos.d/?
Do I list http://download.zfsonlinux.org/fedora/44/x86_64/ in the repo section of rpm-ostree?

alternatively, if I avoid the akmods module and try installing zfs using dnf modulue, it installs zfs-dkms and tries to build a zfs kmod for the running kernel of my host machine, which fails.

steps to reproduce

make a barebones recipe using the template, I used bazzite-nvidia-stable-44 as base.

name: bazzite-nvidia-zfs
description: This is my personal OS image.

base-image: ghcr.io/ublue-os/bazzite-nvidia
image-version: stable-44
platforms:
  - linux/amd64

modules:
- type: akmods
  base: coreos-stable (or main)
  install:
    - zfs

run bluebuild build recipes/recipe.yml

build log

l/bazzite-nvidia-zfs:latest_linux_amd64 => [7/7] STEP 18/29: RUN --mount=type=bind,from=stage-files,src=/files,dst=/tmp/files,rw --mount=type=bind,from=ghcr.io/blue-build/modules/akmods:latest,src=/modules,dst=/tmp/modules,rw --mount=type=bind,from=stage-akmods-main,src=/rpms,dst=/tmp/rpms,rw --mount=type=bind,src=.bluebuild-scripts_,dst=/tmp/scripts/,Z --mount=type=cache,sharing=locked,dst=/var/cache/rpm-ostree,id=rpm-ostree-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/libdnf5,id=dnf-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/zypp,id=zypper-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/apk,id=apk-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/apt,id=apt-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/pacman,id=pacman-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/usr/lib/sysimage/cache/pacman,id=pacman-sysimage-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs /tmp/scripts/run_module.sh 'akmods' '{"type":"akmods","base":"main","install":["zfs"]}'
l/bazzite-nvidia-zfs:latest_linux_amd64 => Trying to pull ghcr.io/blue-build/modules/akmods:latest...
l/bazzite-nvidia-zfs:latest_linux_amd64 => Getting image source signatures
l/bazzite-nvidia-zfs:latest_linux_amd64 => Copying blob sha256:f876ee7a4cf039ff5d4de3dd22beb641fc33990268054445c4b87840b98e06e4
l/bazzite-nvidia-zfs:latest_linux_amd64 => Copying config sha256:d196e1a108e763d5d6de97ce9282065fd64b029dbf5965c39ed5bdcb3a61ebe4
l/bazzite-nvidia-zfs:latest_linux_amd64 => Writing manifest to image destination
l/bazzite-nvidia-zfs:latest_linux_amd64 => ============================ Start 'akmods' Module ============================
l/bazzite-nvidia-zfs:latest_linux_amd64 => Installing akmods
l/bazzite-nvidia-zfs:latest_linux_amd64 => Total length of modules to install: 1
l/bazzite-nvidia-zfs:latest_linux_amd64 => Installing: zfs
l/bazzite-nvidia-zfs:latest_linux_amd64 => Updating and loading repositories:
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  Fedora 44 - x86_64 - Updates Archive   100% |   6.8 KiB/s |   3.4 KiB |  00m00s
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  Fedora 44 - x86_64 - Updates           100% |   9.9 KiB/s |  18.9 KiB |  00m02s
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  Copr repo for akmods owned by ublue-os 100% |   3.0 KiB/s |   1.5 KiB |  00m01s
l/bazzite-nvidia-zfs:latest_linux_amd64 => Repositories loaded.
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  https://mirrors.rpmfusion.org/free/fed 100% |  11.2 KiB/s |  11.5 KiB |  00m01s
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  https://mirrors.rpmfusion.org/nonfree/ 100% |  10.7 KiB/s |  11.5 KiB |  00m01s
l/bazzite-nvidia-zfs:latest_linux_amd64 => Package                    Arch   Version Repository        Size
l/bazzite-nvidia-zfs:latest_linux_amd64 => Installing:
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  rpmfusion-free-release    noarch 0:44-3  @commandline   5.6 KiB
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  rpmfusion-nonfree-release noarch 0:44-3  @commandline   5.8 KiB
l/bazzite-nvidia-zfs:latest_linux_amd64 =>
l/bazzite-nvidia-zfs:latest_linux_amd64 => Transaction Summary:
l/bazzite-nvidia-zfs:latest_linux_amd64 =>  Installing:         2 packages
l/bazzite-nvidia-zfs:latest_linux_amd64 =>
l/bazzite-nvidia-zfs:latest_linux_amd64 => Total size of inbound packages is 23 KiB. Need to download 0 B.
l/bazzite-nvidia-zfs:latest_linux_amd64 => After this operation, 11 KiB extra will be used (install 11 KiB, remove 0 B).
l/bazzite-nvidia-zfs:latest_linux_amd64 => Running transaction
l/bazzite-nvidia-zfs:latest_linux_amd64 => [1/4] Verify package files              100% |   0.0   B/s |   2.0   B |  00m00s
l/bazzite-nvidia-zfs:latest_linux_amd64 => [2/4] Prepare transaction               100% |   8.0   B/s |   2.0   B |  00m00s
l/bazzite-nvidia-zfs:latest_linux_amd64 => [3/4] Installing rpmfusion-nonfree-rele 100% | 922.4 KiB/s |   7.4 KiB |  00m00s
l/bazzite-nvidia-zfs:latest_linux_amd64 => [4/4] Installing rpmfusion-free-release 100% |  27.2 KiB/s |   7.2 KiB |  00m00s
l/bazzite-nvidia-zfs:latest_linux_amd64 => Warning: skipped OpenPGP checks for 2 packages from repository: @commandline
l/bazzite-nvidia-zfs:latest_linux_amd64 => Complete!
l/bazzite-nvidia-zfs:latest_linux_amd64 => error: Handling argument /tmp/rpms/kmods/*zfs*.rpm: No such file or directory (os error 2)
l/bazzite-nvidia-zfs:latest_linux_amd64 => ============================ Failed 'akmods' Module ============================
l/bazzite-nvidia-zfs:latest_linux_amd64 => Error: building at STEP "RUN --mount=type=bind,from=stage-files,src=/files,dst=/tmp/files,rw --mount=type=bind,from=ghcr.io/blue-build/modules/akmods:latest,src=/modules,dst=/tmp/modules,rw --mount=type=bind,from=stage-akmods-main,src=/rpms,dst=/tmp/rpms,rw --mount=type=bind,src=.bluebuild-scripts_,dst=/tmp/scripts/,Z --mount=type=cache,sharing=locked,dst=/var/cache/rpm-ostree,id=rpm-ostree-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/libdnf5,id=dnf-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/zypp,id=zypper-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/apk,id=apk-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/apt,id=apt-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/var/cache/pacman,id=pacman-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs --mount=type=cache,sharing=locked,dst=/usr/lib/sysimage/cache/pacman,id=pacman-sysimage-cache-bazzite-nvidia-zfs-stable-44-stage-bazzite-nvidia-zfs /tmp/scripts/run_module.sh 'akmods' '{"type":"akmods","base":"main","install":["zfs"]}'": while running runtime: exit status 1
ERROR => Failed:
  × Failed to build localhost/bazzite-nvidia-zfs:latest_linux_amd64

  Try running with `-vv` for trace logging.
  You can also open an issue at https://github.com/blue-build/cli/issues

environment

OS: nixos-unstable (26.05)
bluebuild version: 0.9.35
bluebuild installed using: cargo

related issue

ublue-os/bazzite#1438

[xynydev]

This is clearly an oversight on our part, with the module just not being updated to handle zfs being distributed in the akmods-zfs image. Stuff like this happens when we have features that depend on ublue stuff, but do not dogfood them or keep up with how they evolve. They change things pretty often.

To fix this issue, we actually gotta modify the akmods.j2 template in CLI which pulls in the akmods using COPY, such that it also copies in akmods-zfs when the user wants zfs. Probably a fast fix for our resident rustheads. Moving the issue over...

[gmpinder]

This might actually be a good time to update the akmods module to use skopeo copy to get the rpms instead. Some parts of the module already does this, so we should migrate fully so we can remove complexity and inter-dependency in the CLI code.

[xynydev]

Good idea.

[ruzko]

please ignore this comment if you'd like to work undisturbed :)

I'd love to help solve this issue if I can, although I'm not great at rust. my server needs zfs and legacy proprietary nvidia driver to work.

tried just plopping a fourth if let block into akmods.j2, but that errored out because the images vector of the AkmodsInfo in recipe/src/akmods_info.rs only allows for up to 3 images. When I added another Option<String> to the images vector, another thing broke.

  {%- if let Some(zfs_image) = info.images.3 %}
COPY --from=ghcr.io/ublue-os/{{ zfs_image }} /rpms /rpms
  {%- endif %}

[gmpinder]

please ignore this comment if you'd like to work undisturbed :)

I'd love to help solve this issue if I can, although I'm not great at rust. my server needs zfs and legacy proprietary nvidia driver to work.

tried just plopping a fourth if let block into akmods.j2, but that errored out because the images vector of the AkmodsInfo in recipe/src/akmods_info.rs only allows for up to 3 images. When I added another Option<String> to the images vector, another thing broke.

  {%- if let Some(zfs_image) = info.images.3 %}
COPY --from=ghcr.io/ublue-os/{{ zfs_image }} /rpms /rpms
  {%- endif %}

So, yes, this is how we used to do it, but now we want to try to remove the need for this template change and instead copy everything from the image within the script instead of pulling the image in the container file.