Centralized kernel pinning via akmods repo

[castrojo]

Both Bluefin and Aurora need to constantly pin kernels to avoid regressions. This is done per-repo and we haven't had any cases where one needed to pin and the other didn't.

I'm adding Bluefin LTS to the mix, we want to follow the same release cadence as the other Bluefins. So, we need a centralized place to pin kernels. Since we're caching everything in the akmods repo, sherman suggested that I file this so that we can discuss.

The contributor UX should be simple enough where more people can submit PRs for a kernel pin. Ideas:

The ideal dream: There's a text file in the root of the akmods repo with the string 6.14.10 in it, and that's it. All the workflows do not need to be edited. This kicks off renovate and Aurora/Bluefin will have the fix in about 30 minutes. And both teams will know that's the place to check. If it's empty it defaults to the CoreOS kernel version. Nice.

Probable solution: Let's leave well documented comments wherever you do this and a "How to pin a kernel section" in the README. It is also unclear to me why some versions have a 200 or a 100 in them, etc.

Not ideal: Having to PR akmods and each image repo for every kernel pin.

Worst case: I end up using github copilot all the time and end up replacing half the workflows with destiny lore LLM hallucinations.

(Bazzite does it's own thing)

[inffy]

Agree, would be nice if we could just pin in akmods. Not that iys a big chore to pin in Aurora/Bluefin but still

[m2Giles]

We already do this with akmods inside of the kernel-cache portion.

Ucore never implemented kernel pinning so we did it in akmods.

[bsherman]

We already do this with akmods inside of the kernel-cache portion.

Ucore never implemented kernel pinning so we did it in akmods.

Yep, this is why I asked @castrojo to file an issue... so we can use it to standardize at least the bluefins (and aurora) with the kernel-cache version pinning.

This will then match pretty close to how uCore does it.

Bazzite can do what they want, but they aren't as impacted by this given they already effectively pin by controlling their own kernel build.

At the moment, I'm not confident we have a single way to pin a kernel in the akmods kernel-cache, but i want to use this issue to ensure we do, so we can document it as the standard way to pin which makes things less painful for users of akmods.

[castrojo]

Yeah if custom image builders have an easy way to get on gated kernels then that's something valuable we could provide to the community.

[bsherman]

For the record, "HOW" we pin a kernel in akmods is like this:

        # Kernel Pin Location
        # this gives us a consistent place to handle pinning for coreos stable dependent usages
        # including: ucore stable, bluefin/aurora stable, bluefin lts (using centos-kmodsig version matching)
        if [[ "{{ kernel_flavor }}" =~ coreos-stable ]]; then
            image_linux="6.14.11-300.fc42.$(uname -m)"
        fi

From: https://github.com/ublue-os/akmods/blob/797e35e2dcde2ec94f13a9902cfe96b05a8fe2b0/Justfile#L67-L75

The major benefit of doing it in akmods is a central pin which helps everyone consuming the cached kernels and kmods.

So, we've only ever done this for coreos-stable kernels, but that means Bluefin and Aurora "stable" AND uCore have can all benefit.

To pin for Bluefin LTS, we need to add an appropriate conditional somewhere like:

        if [[ "{{ kernel_flavor }}" =~ centos ]]; then
            image_linux="6.14.11-300.el10.$(uname -m)"
        fi

[bsherman]

Upon re-reading the Justfile I do think I'll need to update the PIN location to be more versatile... I'll report back when accomplished.

Edit: my reading last night was hurried... I had to remember the big picture on how we are using kernel gating plus pinning.

Updated clarification:

The pin location I shared above IS CORRECT for both CentOS Kmods SIG and Fedora CoreOS kernels because we are using the coreos-stable kernel version to gate centos-kmodsig kernels in akmods.

Bluefin LTS uses the stock centos kernel (from akmods) which doesn't currently have a pin location (but one could be added), while Bluefin LTS(HWE) uses the centos-kmodsig kernel (from akmods).

This note already exists in the script:

        # Kernel Pin Location
        # this gives us a consistent place to handle pinning for coreos stable dependent usages
        # including: ucore stable, bluefin/aurora stable, bluefin lts (using centos-kmodsig version matching)

The reasoning for gating centos-kmodsig on the version from coreos-stable is to delay release of the latest kernel versions to a "more stable" OS yet still enabling a newer version than in the stock kernel.