Does CentOS 7 incorrectly sort kernel menu entries in grub.cfg?

Question

I've noticed an unexpected ordering of grub menu entries on a CentOS 7 system:

It has following kernels installed:

$ ls /boot/vmlinuz* -ltr
Jun 30 14:17 /boot/vmlinuz-3.10.0-123.el7.x86_64
Nov  6 16:14 /boot/vmlinuz-3.10.0-123.9.3.el7.x86_64
Nov 23 17:12 /boot/vmlinuz-0-rescue-c61cbe0918ab45e0927fb5d31cf45f98

In my interpretation of the version scheme the version '3.10.0-123.9.3.el7' is greater than '3.10.0-123.el7'. This is also consistent with the files mtime - and also matches the uname -a outputs:

3.10.0-123.el7.x86_64     Mon Jun 30 12:09:22 UTC 2014
3.10.0-123.9.3.el7.x86_64 Thu Nov 6  15:06:03 UTC 2014

But the /boot/grub2/grub.cfg uses another order:

$ grep vmlinuz-3 /boot/grub2/grub.cfg | sed 's/root=.*//'
linux16 /vmlinuz-3.10.0-123.el7.x86_64 
linux16 /vmlinuz-3.10.0-123.9.3.el7.x86_64

Huh?

Since the system got some additional kernel parameters, grub.cfg was explicitly re-generated via following command:

# grub2-mkconfig -o /boot/grub2/grub.cfg 

Which should be the official method - as documented in the manual.

The ordering is apparently implemented via:

/etc/grub.d/10_linux
  -> /usr/share/grub/grub-mkconfig_lib
     -> version_find_latest()
        -> version_test_gt()

Is this a well known bug in grub2-mkconfig?

I couldn't find a bug report for it, though.

Surprisingly, on another CentOS 7 machine (which is also up-to-date) the grub.cfg order is correct:

$ grep vmlinuz /boot/efi/EFI/centos/grub.cfg | sed 's/root=.*//'
linuxefi /vmlinuz-3.10.0-123.9.3.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.9.2.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.8.1.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.6.3.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.el7.x86_64 
linuxefi /vmlinuz-0-rescue-48235f1ad5c943c3a7dfd1551a1fc5b8 

The difference between the two machines is: on the 2nd machine grub2-mkconfig was never executed manually.

And indeed, when executing it manually the order is also wrong:

# grub2-mkconfig -o del.cfg
# grep vmlinuz del.cfg | sed 's/root=.*//' 
linuxefi /vmlinuz-3.10.0-123.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.9.3.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.9.2.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.8.1.el7.x86_64 
linuxefi /vmlinuz-3.10.0-123.6.3.el7.x86_64 
linuxefi /vmlinuz-0-rescue-48235f1ad5c943c3a7dfd1551a1fc5b8

Thus, it seems that, when installing kernel updates via yum update, the install script does not execute a grub-2-mkconfig -o /boot/efi/EFI/centos/grub.cfg. How is the grub.cfg regenerated during a kernel package install then?

Answer 1

This is a well known bug:

• grub2-mkconfig creates boot choices in wrong order (CentOS Bug 0007651)
• grub2-mkconfig wrong sorting (RHEL Bug 1124074)
• grub2 wrong sorting (Bug 678840 - fixed)

To determine how a kernel package updates grub.cfg one can display the scripts via:

$ yum whatprovides /boot/vmlinuz-3.10.0-123.9.3.el7.x86_64
kernel-3.10.0-123.9.3.el7.x86_64 : The Linux kernel
[..]
$ rpm -q --scripts kernel-3.10.0-123.9.3.el7.x86_64

This shows that /usr/sbin/new-kernel-pkg is called - which in turn calls grubby.

Workaround

(until it is fixed in RHEL/CentOS)

--- /usr/share/grub/grub-mkconfig_lib.orig  2014-06-30 18:16:11.000000000 +0200
+++ /usr/share/grub/grub-mkconfig_lib       2014-11-26 17:38:57.814000000 +0100
@@ -255,13 +255,24 @@

 version_find_latest ()
 {
-  version_find_latest_a=""
-  for i in "$@" ; do
-    if version_test_gt "$i" "$version_find_latest_a" ; then
-      version_find_latest_a="$i"
-    fi
-  done
-  echo "$version_find_latest_a"
+  # workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1124074
+  # 'grub2-mkconfig wrong sorting'
+  {
+    for i in "$@"; do
+      echo $i
+    done | grep -v rescue | sort -V
+    for i in "$@"; do
+      echo $i
+    done | grep rescue | sort -V
+  } | head -n 1
 }

Answer 2

I posted a patch to the two bugzilla.redhat.com bugs that actually fixes this issue. maxschlepzig's patch is very close to the right answer, but not quite. My patch is based on his.

Answer 3

I think the way you determine what happens after a kernel is installed is as follows using rpm + the --scripts switch.

Example

$ rpm --scripts -q kernel-$(uname -r)
postinstall scriptlet (using /bin/sh):

if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&
   [ -f /etc/sysconfig/kernel ]; then
  /bin/sed -r -i -e 's/^DEFAULTKERNEL=kernel-smp$/DEFAULTKERNEL=kernel/' /etc/sysconfig/kernel || exit $?
fi
preuninstall scriptlet (using /bin/sh):
/bin/kernel-install remove 3.16.6-203.fc20.x86_64 /boot/vmlinuz-3.16.6-203.fc20.x86_64 || exit $?
posttrans scriptlet (using /bin/sh):
/bin/kernel-install add 3.16.6-203.fc20.x86_64 /boot/vmlinuz-3.16.6-203.fc20.x86_64 || exit $?

This is broken up into sections: postinstall, preuninstall, and postrans. The tool doing the install/removal of kernels is this script:

/bin/kernel-install add <kernel label> </path/to/boot/vmlinuz-...> || exit $

Who owns this script?

The script kernel-install is part of Systemd.

$ rpm -qf /bin/kernel-install 
systemd-208-28.fc20.x86_64