bare metal raid array with LVM dissapears after reboot

Question

tl;dr:

RAID array + LVM dissapears after reboot.

Detail

I wanted to set up an encrypted raid array, based on this SO.

I set up a RAID array "bare metal" (i.e. on the disks not on the partitions). Once the array was built I didn't format it and ran pvcreate. Then I set up LVM and encryption in the usual way. (See Digital Ocean Guide on RAID, arch guide on LVM on RAID.

After setting up encryption, I ran mdadm --detail --scan | tee -a /etc/mdadm/mdadm.conf then update-initramfs -u and issued a reboot. /dev/md0 is then gone. Where did I go wrong?

Steps to debug.

There are a lot of similar questions, this is what I have tried so far based on my reading.

• Output of mdadm --examine --scan is blank.
• uncommented contents of /etc/mdadm/mdadm.conf: HOMEHOST <system>, MAILADDR root, ARRAY /dev/md0 metadata=1.2 name=server:0 UUID=xxxxx There are no duplicate entries as suggested in some SO answers.
• Issuing service mdadm status yields Loaded: masked (Reason: Unit mdadm.service is masked.)
• Issuing cat /proc/mdstat yeilds Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] \n unused devices: <none>
• I tried running dpkg-reconfigure mdadm and update-initramfs -u as suggested here and I did not get an option to enable "all". I tried rebooting after this, which also did not work.

As a last resort I tried mdadm --assemble --scan.

Strangely, the type is not TYPE="linux_raid_member" as I would have expected from this SO

root@server:/home/user# blkid
/dev/sda1: UUID="xx" TYPE="ext2" PARTUUID="xx-xx"
/dev/sda5: UUID="xx" TYPE="LVM2_member" PARTUUID="xx-xx"
/dev/mapper/server--vg-root: UUID="xx" TYPE="ext4"
/dev/mapper/server--vg-swap_1: UUID="xx" TYPE="swap"
/dev/sdb: PTUUID="xx" PTTYPE="gpt"
/dev/sdc: PTUUID="xx" PTTYPE="gpt"
/dev/sdd: PTUUID="xx" PTTYPE="gpt"

The array is over sd[bcd] and /dev/sda holds the root filesystem. This array drive is for data.

Steps to recreate.

For completeness and in case it is useful.

• mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd Create array
• pvcreate /dev/md0 Make physical volume
• vgcreate data_group /dev/md0 Create vol group
• lvcreate -n secure data_group -L +900G Create volume
• cryptsetup -v luksFormat /dev/data_group/secure Setup encryption
• cryptsetup luksOpen /dev/data_group/secure cryptvault Open encrypted volume
• mkfs.ext4 /dev/mapper/cryptvault Partition
• mkdir /mnt/vault Make mount point
• mount /dev/mapper/cryptvault /mnt/vault Mount
• mdadm --detail --scan | tee -a /etc/mdadm/mdadm.confSave array layout
• update-initramfs -u Update init with new config
• reboot

Answer 1

This is just a theory: mdadm metadata version 1.2 is 4K from the start of the device and the disks already had existing GPT partition table on them when you created the array which was not removed by mdadm (I'd expect mdadm to remove all existing signatures, but that is the only explanation for the GPT signature existence). The array is now not assembled because for all tools (including mdadm) this just looks like and empty disk with GPT partition table (it is also possible that blkid detects the header backup near the end of the disk) not a mdmember, so the disks are ignored.

You can try to remove the partition table using wipefs -- first just plain wipefs /dev/sdb to detect and print all signatures and then wipefs -o <offset> with offset of the GPT signature printed by previous command. This might help the system to detect the disk as a mdmeber device.

If this doesn't help, I'd suggest starting over and picking metadata version 1.1 (--metadata=1.1 option for mdadm, 1.1 places metadata at the start of the device) or creating the array on top of partitions (there is no benefit in using the disks without a partition table IMHO). Or of course you can wait for someone else with some better suggestions :-)

Btw. I assume this is a brand new setup so there are no valuable data on the devices, but it might be useful to backup first few MiBs of the disks before doing any irreversible changes.