How to debug why I get no boot prompt to decrypt LUKS volume containing root filesystem? [systemd-cryptsetup]

Question

I am currently using Rocky 9 + Linux kernel 5.14.0 + security / stability patches.

I have two GPT partitions that contain multiple LVM filesystems. One partition is encrypted (LUKS), and one is not. The non-encrypted partition boots to multi-user fine. For the encrypted partition, I do not get a prompt to enter the passphrase.

Within a few minutes, it breaks out to a shell where I can poke around - I added rd.shell and rd.break to my kernel command line arguments. I can use cryptsetup (which I explicitly added to my initramfs to assist with diagnosis) to manually open the LUKS encrypted container, activate my LVMs and mount filesystems (I don't know how to convince the system to exit the ramdisk shell and continue booting up to multi-user after having provided assistance to mount the root filesystem - how to do that would be a side question).

Here's the question: What are the pieces to pursue to investigate why I get no passphrase prompt to decrypt my LUKS volume at boot time?

I'm looking for hints to methodically determine the culprit.

I have configured systems similarly in the past where it worked (including using dracut-sshd to allow for entering the passphrase over the network in addition to the console). I'm just not sure what's different this time.

My initramfs image includes the 'crypt' module (and now some other tools like cryptsetup to try to help diagnose the issue

Update... In my case I do get the prompt for passphrase when the unit is power cycled, but NOT when rebooted.

In the good case, I get the following messages from systemd-cryptsetup:

May 21 17:51:06 localhost systemd[1]: Created slice Slice /system/systemd-cryptsetup.
 .
 .
May 21 17:54:04 localhost systemd-cryptsetup[833]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/sdc7.

In the bad case, I never see that last message from systemd-cryptsetup. Eventually dracut-initqueue just gets tired of waiting for the root filesystem to show up. I get the following where dracut_initqueue seems to say to systemd-cryptsetup that "Ahem... I am waiting here" while impatiently tapping its foot:

May 21 17:39:53 localhost systemd[1]: Created slice Slice /system/systemd-cryptsetup.
 .
 .
May 21 17:42:04 localhost dracut-initqueue[677]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
May 21 17:42:04 localhost dracut-initqueue[677]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2fap_root1.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
May 21 17:42:04 localhost dracut-initqueue[677]:     [ -e "/dev/disk/by-label/ap_root1" ]
May 21 17:42:04 localhost dracut-initqueue[677]: fi"
May 21 17:42:04 localhost dracut-initqueue[677]: Warning: dracut-initqueue: starting timeout scripts
May 21 17:42:05 localhost dracut-initqueue[677]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
May 21 17:42:05 localhost dracut-initqueue[677]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2fap_root1.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
May 21 17:42:05 localhost dracut-initqueue[677]:     [ -e "/dev/disk/by-label/ap_root1" ]
May 21 17:42:05 localhost dracut-initqueue[677]: fi"
May 21 17:42:05 localhost dracut-initqueue[677]: Warning: dracut-initqueue: starting timeout scripts
May 21 17:42:06 localhost dracut-initqueue[677]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
May 21 17:42:06 localhost dracut-initqueue[677]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-label\x2fap_root1.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
May 21 17:42:06 localhost dracut-initqueue[677]:     [ -e "/dev/disk/by-label/ap_root1" ]
May 21 17:42:06 localhost dracut-initqueue[677]: fi"

So now, I am searching for information on how to get debug information about systemd-cryptsetup to see what it's doing.

Answer 1

The fact that the passphrase prompt works after a power cycle but fails after a reboot suggests something (maybe the disk, or the storage controller?) does not get completely reset on reboot.

It might be a timing issue, e.g. the power-up self-tests give the disks enough time to fully spin up, but in a reboot, one of the disks might not complete it's reset by the time it needs to be accessed.

Or it might be a quirk of your system chipset: if your hardware is significantly newer than your kernel version, the kernel might not "know" the specific extra actions your system chipset needs to be done at reboot. These actions are often workarounds to specific hardware or firmware bugs.

The missing cryptsetup message would suggest that whatever the problem is, it seems to affect the /dev/sdc disk. How old is that disk? Is it a HDD or a SSD? Is it connected with SATA or with something else?