Persistent disk name /dev/sd'x', changing with almost every reboot

Question

I'm dealing with an issue for a few days now which start pulling my hair. Hopefully someone here has some ideas to get my issue fixed :-).

I have a big storage server with 24x 3,5" drives. These drives are spread across 3 LSI 9211-8i controllers that are running in IT mode. The drives are part of a large software raid-6 array (mdraid) which works fine. As OS drive I have 2x 1TB SSDs hooked up directly to the motherboards SATA ports @ software raid-1 (mdraid), which work fine except the drive persistent names.
As OS i'm using Ubuntu Server 18.04 LTS.

For some reason almost every reboot is changing the drive names for the SSD drives that are connect directly to the motherboard. At some moment they are named "sda + sdb", while with next reboot it might be "sdy + sdz", or sometimes even a random letter in the middle of the alfabet. It's very annoying because of hotswap drive bay labels. This does not happen to the 24 drives connected to the LSI controller - they are always in the right sequence.

What I would like to have is that the OS SSDs are always identified as either "sda + sdb" or as "sdy + sdz". I don't really mind which name, as long it's static and won't change with reboots.

I already tried setting up a custom udev rule in "/etc/udev/rules.d/01-disk-bay.rules". The rules don't work or the OS just ignoring them as nothing appears to change. I'm probably doing something wrong.

The contents of this file are

########## Map SATA 0 to /dev/sdy ##############


KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata1/host*", NAME="sdy", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"

KERNEL=="sd?*", ATTR{partition}=="1", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata1/host*", NAME="sdy%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

########## Map SATA 1 to /dev/sdz ##############


KERNEL=="sd?", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata2/host*", NAME="sdz", RUN+="/usr/bin/logger My disk ATTR{partition}=$ATTR{partition}, DEVPATH=$devpath, ID_PATH=$ENV{ID_PATH}, ID_SERIAL=$ENV{ID_SERIAL}", GOTO="END_20_PERSISTENT_DISK"

KERNEL=="sd?*", ATTR{partition}=="1", SUBSYSTEM=="block", DEVPATH=="*1f.2/ata2/host*", NAME="sdz%n", RUN+="/usr/bin/logger My partition parent=%p number=%n, ATTR{partition}=$ATTR{partition}"

LABEL="END_20_PERSISTENT_DISK"

The identification info of the SSDs are as follow: first SSD (currently showing up as 'sda')

# udevadm info --name /dev/sda
P: /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda
N: sda
S: disk/by-id/ata-Crucial_CT1024MX200SSD1_1619128D4E19
S: disk/by-id/wwn-0x500a0751128d4e19
S: disk/by-path/pci-0000:00:1f.2-ata-1
E: DEVLINKS=/dev/disk/by-path/pci-0000:00:1f.2-ata-1 /dev/disk/by-id/wwn-0x500a0751128d4e19 /dev/disk/by-id/ata-Crucial_CT1024MX200SSD1_1619128D4E19
E: DEVNAME=/dev/sda
E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda

and the second SSD (currently showing up as 'sdb').

# udevadm info --name /dev/sdb
P: /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb
N: sdb
S: disk/by-id/ata-Crucial_CT1024MX200SSD1_1619128B9EC5
S: disk/by-id/wwn-0x500a0751128b9ec5
S: disk/by-path/pci-0000:00:1f.2-ata-2
E: DEVLINKS=/dev/disk/by-id/wwn-0x500a0751128b9ec5 /dev/disk/by-id/ata-Crucial_CT1024MX200SSD1_1619128B9EC5 /dev/disk/by-path/pci-0000:00:1f.2-ata-2
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb

Is there anyone here with experience on how to deal with such a thing? :-)

Answer 1

/dev/sdX has not been a stable identifier for a drive for a very long time (and indeed probably never was). Those are allocated in the order they're discovered, and different controllers are probed in parallel. Not only that, if a drive drops off the bus and comes back, it'll often get a new letter. Or sometimes when you replace a failed drive (both of these happen because something still has a reference to the old drive, e.g., the "failed" entry in md-raid).

There are stable identifiers; use them instead. Your udevadm info output told you what they are:

S: disk/by-id/ata-Crucial_CT1024MX200SSD1_1619128D4E19
S: disk/by-id/wwn-0x500a0751128d4e19

These two uniquely identify the drive itself, even if you move it to a different port.

S: disk/by-path/pci-0000:00:1f.2-ata-1

This identifies the port where the drive is plugged in. If you replace the drive, the new drive will also have this identifier.

So, e.g., if you want to check SMART status of whatever drive is in that port, you'd use smartctl -x /dev/disk/by-path/pci-0000:00:1f.2-ata-1 and not /dev/sda. If you want to check that particular drive, even if someone has moved it to another port, you'd use smartctl -x /dev/disk/by-id/wwn-0x500a0751128d4e19.

(You can use udev rules to set additional short names if you need to. You do this by SYMLINK+= in your rule; see e.g., /lib/udev/rules.d/60-persistent-storage.rules for examples. But you often don't need short names if it's just going in a config file for example).