Encrypted LVM "is not a valid LUKS device", cannot boot or open

Question

I have an Arch installation on a LUKS-encrypted LVM. After I installed Debian on a different NVME drive I'm no longer able to cryptsetup open the partition from other systems, nor can I boot into Arch.

# cryptsetup open /dev/nvme0n1p3 arch
    Device /dev/nvme0n1p3 is not a valid LUKS device.

Here is a hexdump -C of the first 35kb of said partition.

Notes:

1. When I was installing Arch I first made a partition and assigned it as an LVM partition, then set up and opened LUKS, then set up LVM itself.
2. When Arch was operational, sometimes the decryption process would fail and throw me into GRUB rescue shell saying that the logical volume was not found. I presume it's because the motherboard passes NVMEs in different order yet in GRUB config file the partition was set as /dev/nvme0n1p3:<LVM VG:LV>, not as a UUID. Back then it could be fixed by unplugging the laptop's power cable until the decryption prompt appeared. After the failure nothing works anymore, including removing all other drives.

Is it toast or can I do something to recover it?

Answer 1

The issue was solved using Method #1 created and kindly suggested by frostschutz. Everything worked out smoothly and without any errors. After dumping and repairing a LUKS header file I was able to use it to open the partition. Then I mounted the volume after scanning for it using lvscan.

For more extreme cases there is also Method #2.

From now on I'll be smarter and will start making header backups :)