Determination of current kernel fails on RHEL 8

[jhowe-uw]

Attempting to use infra.leapp v 1.6.1 components to upgrade from RHEL 8.10 to RHEL 9.6

From infra/leapp/roles/remediate/tasks

• leapp_multiple_kernels.yml
• leapp_newest_kernel_not_in_use.yml

Query for latest kernel ( assumes array[0] entry )

$ rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | sort -Vr
4.18.0-553.el8_10.x86_64
4.18.0-553.64.1.el8_10.x86_64
4.18.0-553.58.1.el8_10.x86_64

$ grubby --default-kernel | sed 's/^\/boot\/vmlinuz\-//'
4.18.0-553.el8_10.x86_64 

Alternate method:

$ rpm -q --last kernel
kernel-4.18.0-553.64.1.el8_10.x86_64          Fri 01 Aug 2025 11:41:12 AM PDT
kernel-4.18.0-553.58.1.el8_10.x86_64          Mon 28 Jul 2025 02:36:23 PM PDT
kernel-4.18.0-553.el8_10.x86_64               Tue 13 May 2025 03:55:31 PM PDT

So perhaps an alternate incantation to get a list of older kernels to purge in prep for leapp upgrade proper: ( infra.leapp.remediation::leapp_multiple_kernels )

$ rpm -q --last kernel | sed '1d'
kernel-4.18.0-553.58.1.el8_10.x86_64          Mon 28 Jul 2025 02:36:23 PM PDT
kernel-4.18.0-553.el8_10.x86_64               Tue 13 May 2025 03:55:31 PM PDT

$ rpm -q --last kernel | sed '1d' | awk '{print $1}'
kernel-4.18.0-553.58.1.el8_10.x86_64
kernel-4.18.0-553.el8_10.x86_64

Where this manifested for me was during the infra.leapp.analysis phase, as I bring the VM to the most current 8.10 rpms prior to analysis call.

I see an incorrect grubby default kernel assignment, so maybe RHEL has a bug somewhere here where dnf update -y isn't assigning the most recent kernel properly ( as noted in grubby output above ).

So a couple of edge case issues here:

1. sorting most recent kernel logic needs updating
2. issue in RHEL 8.10 where the most recent kernel isn't being assigned the default kernel

rpm --last seems more predictable than using the sort -Vr ( maybe that was consistent in the past, or --last is a new feature in more current versions of rpm )

I did a quick search on access.redhat.com to see if this grubby stale kernel issue is known, I didn't find any. It's late, I will follow-up tomorrow.

Thanks!

[swapdisk]

Hi! Thank you for raising the issue.

The first issue you listed is that those two playbooks under the remediate role are incorrectly sorting the installed kernel package versions. You suggested using rpm -q --last kernel as an alternative, but that could also be incorrect because we would be assuming the chronological order of installing the packages is the same as the package version order. While that may normally be true, there is still the possibility that the kernel packages could have been installed out of order, so depending on the install time sort is no good.

I messed around with this on a RHEL 7 host and found the following alternative works to give the expected list of kernel packages in reverse version order...

rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | grub2-rpm-sort | tac

Please try this command on your host and let us know how it works for you?

The next issue is the question of the incorrect grubby default kernel setting. If that was true before you ran the remediate role, we can assume the problem is not because of infra.leapp. However, I can see how this could happen because of the incorrect sorting happening with the "Get list of installed kernels packages sorted by the newest version" and then the wrong kernel version being set as the default with the "Set default kernel to latest" task. In that case, fixing the sorting would resolve both of the edge case issues you listed.

Looking forward to hearing your thoughts. Thanks, again!

[jhowe-uw]

That works on my RHEL 8 system:

rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | grub2-rpm-sort | tac
4.18.0-553.64.1.el8_10.x86_64
4.18.0-553.58.1.el8_10.x86_64
4.18.0-553.el8_10.x86_64

I'm still tracking down why this particular VM's default kernel is the old one ( yeah edge cases! ).

I've seen the remediation for grub to grub2. IDK if using grub2-rpm-sort here impacts that one, or if you need a split query based on OS version. ( Are people still running RHEL 6? )

I appreciate the insight into using rpm --last and it's gotcha's.

Thanks!

[swapdisk]

Yes, there are still some folks upgrading from RHEL 6. However, the remediate role is only supported for RHEL 7 and 8, so we can get away with assuming we have grub2.