My NAS went offline today. I could not even boot up. I have extracted the OS drive (Debian Bookworm) and plugged it into my laptop via an adapter. I failed mounting it, with the error: unknown filesystem type 'LVM2_member'
I suspect the file system got corrupted somehow (we had a power outage few days ago) or the SSD is dying (less than 3 years old). I'd like to understand what is wrong, and fix it if possible. Below are the results of dmesg (just after plugging it in), lvdisplay, pvdisplay, mount, fdisk and lsblk.
I think there is a bad block I need to blacklist/replace but I am out of my depth when it comes to investigate hardware issues. Not sure what to make of the mount results (see below)
neela@hooban:~$ sudo dmesg -w
[ 6789.320784] usb 1-1: new high-speed USB device number 17 using xhci_hcd
[ 6789.469625] usb 1-1: New USB device found, idVendor=2537, idProduct=1066, bcdDevice= 1.00
[ 6789.469631] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 6789.469635] usb 1-1: Product: NS1066
[ 6789.469638] usb 1-1: Manufacturer: Norelsys
[ 6789.469641] usb 1-1: SerialNumber: 0123456789ABCDE
[ 6789.470794] usb-storage 1-1:1.0: USB Mass Storage device detected
[ 6789.471301] scsi host4: usb-storage 1-1:1.0
[ 6792.042555] scsi 4:0:0:0: Direct-Access NORELSYS 106X 00WR PQ: 0 ANSI: 6
[ 6792.043245] sd 4:0:0:0: Attached scsi generic sg3 type 0
[ 6792.043547] sd 4:0:0:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[ 6792.043784] sd 4:0:0:0: [sdd] Write Protect is off
[ 6792.043791] sd 4:0:0:0: [sdd] Mode Sense: 43 00 00 00
[ 6792.044016] sd 4:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 6792.071052] sdd: sdd1 sdd2 sdd3
[ 6792.091541] sd 4:0:0:0: [sdd] Attached SCSI disk
[ 6792.496074] EXT4-fs (sdd2): mounting ext2 file system using the ext4 subsystem
[ 6792.498069] EXT4-fs (sdd2): warning: mounting unchecked fs, running e2fsck is recommended
[ 6792.498359] EXT4-fs (sdd2): mounted filesystem without journal. Opts: (null)
[ 6792.498364] ext2 filesystem being mounted at /media/neela/ffc11704-3b1b-4e02-8b8a-65937fc1e930 supports timestamps until 2038 (0x7fffffff)
[ 6792.542711] Buffer I/O error on dev dm-7, logical block 0, async page read
[ 6792.542721] Buffer I/O error on dev dm-7, logical block 0, async page read
[ 6792.542831] Buffer I/O error on dev dm-7, logical block 0, async page read
[ 6792.542838] Buffer I/O error on dev dm-7, logical block 0, async page read
[ 6792.548115] Buffer I/O error on dev dm-5, logical block 0, async page read
[ 6792.548132] Buffer I/O error on dev dm-5, logical block 0, async page read
[ 6792.548229] Buffer I/O error on dev dm-5, logical block 0, async page read
[ 6792.548236] Buffer I/O error on dev dm-5, logical block 0, async page read
[ 6817.907631] pcieport 0000:00:1c.0: AER: Corrected error received: 0000:00:1c.0
neela@hooban:~$ sudo cryptsetup luksOpen /dev/sdd3 NAS_crypt
Enter passphrase for /dev/sdd3:
neela@hooban:~$ sudo cryptsetup isLuks /dev/sdd3 -v
Command successful.
neela@hooban:~$ sudo lvdisplay
WARNING: Device mismatch detected for emporium-vg/root which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
WARNING: Device mismatch detected for emporium-vg/swap_1 which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
WARNING: Device mismatch detected for emporium-vg/home which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
--- Logical volume ---
LV Path /dev/emporium-vg/root
LV Name root
VG Name emporium-vg
LV UUID paF4f7-0roi-dXJx-Jjhq-QTVw-QPOT-exsdP2
LV Write Access read/write
LV Creation host, time emporium, 2021-11-21 18:00:01 +0000
LV Status available
# open 0
LV Size <27.94 GiB
Current LE 7152
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
--- Logical volume ---
LV Path /dev/emporium-vg/swap_1
LV Name swap_1
VG Name emporium-vg
LV UUID uJMS6C-0vW2-akFp-hn65-j5Hu-IuqY-MBnH06
LV Write Access read/write
LV Creation host, time emporium, 2021-11-21 18:00:01 +0000
LV Status available
# open 0
LV Size 976.00 MiB
Current LE 244
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:6
--- Logical volume ---
LV Path /dev/emporium-vg/home
LV Name home
VG Name emporium-vg
LV UUID 4nXaef-8rkL-VTan-Btvf-PEN6-Y1jY-S5iKj3
LV Write Access read/write
LV Creation host, time emporium, 2021-11-21 18:00:01 +0000
LV Status available
# open 0
LV Size 901.62 GiB
Current LE 230816
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:7
neela@hooban:~$ sudo pvdisplay
WARNING: Device mismatch detected for emporium-vg/root which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
WARNING: Device mismatch detected for emporium-vg/swap_1 which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
WARNING: Device mismatch detected for emporium-vg/home which is accessing /dev/mapper/luks-70895566-5131-4b7b-9cab-a36b9d568c28 instead of /dev/mapper/NAS_crypt.
--- Physical volume ---
PV Name /dev/mapper/NAS_crypt
VG Name emporium-vg
PV Size <930.52 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 238212
Free PE 0
Allocated PE 238212
PV UUID Cr1nHl-B8mA-9ckn-Hq7M-NIDQ-XXJF-YN1GhM
neela@hooban:~$ sudo mount -o ro,noload /dev/emporium-vg/root /media/neela/emporium/
mount: /media/neela/emporium: wrong fs type, bad option, bad superblock on /dev/mapper/emporium--vg-root, missing codepage or helper program, or other error.
neela@hooban:~$ sudo mount /dev/mapper/NAS_crypt /media/neela/emporium/
mount: /media/neela/emporium: unknown filesystem type 'LVM2_member'.
neela@hooban:~$ sudo mount /dev/emporium-vg/root /media/neela/emporium/
mount: /media/neela/emporium: can't read superblock on /dev/mapper/emporium--vg-root.
neela@hooban:~$ sudo lsblk -l | grep -v loop
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
sdb 8:16 0 119.2G 0 disk
sdb1 8:17 0 512M 0 part /boot/efi
sdb2 8:18 0 732M 0 part /boot
sdb3 8:19 0 118G 0 part
sdd 8:48 0 931.5G 0 disk
sdd1 8:49 0 512M 0 part
sdd2 8:50 0 488M 0 part /media/neela/ffc11704-3b1b-4e02-8b8a-65937fc1e930
sdd3 8:51 0 930.5G 0 part
sr0 11:0 1 1024M 0 rom
sdb3_crypt 253:0 0 118G 0 crypt
ubuntu--vg-root 253:1 0 117G 0 lvm /
ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
data 253:3 0 1.8T 0 crypt /media/data
NAS_crypt 253:8 0 930.5G 0 crypt
neela@hooban:~$ sudo fdisk -l /dev/sdd
Disk /dev/sdd: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: 106X
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 41D2987D-9BEE-4EE3-B6D7-A21CBFCF9143
Device Start End Sectors Size Type
/dev/sdd1 2048 1050623 1048576 512M EFI System
/dev/sdd2 1050624 2050047 999424 488M Linux filesystem
/dev/sdd3 2050048 1953523711 1951473664 930.5G Linux filesystem
====== EDIT =====
Running fsck did not help (the device letter changed between reboots)
neela@hooban:~$ sudo fsck /dev/sdc
fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sdc
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
Found a gpt partition table in /dev/sdc
neela@hooban:~$ sudo mke2fs -n /dev/sdc
mke2fs 1.45.5 (07-Jan-2020)
Found a gpt partition table in /dev/sdc
Proceed anyway? (y/N) y
Creating filesystem with 244190646 4k blocks and 61054976 inodes
Filesystem UUID: ad6ecc21-051a-4c29-9921-74d01dec41c7
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848
neela@hooban:~$ sudo e2fsck -b 32768 /dev/sdc
e2fsck 1.45.5 (07-Jan-2020)
e2fsck: Bad magic number in super-block while trying to open /dev/sdc
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
Found a gpt partition table in /dev/sdc
neela@hooban:~$
==== EDIT 2 =====
I have run fsck on the partition instead of the device, and found the following. Anyone knows a sensible answer to the options fsck offers ?
neela@hooban:~$ sudo fsck /dev/sdc1
fsck from util-linux 2.34
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 2
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
? 3
/dev/sdc1: 8 files, 1493/130811 clusters
neela@hooban:~$ sudo fsck /dev/sdc2
fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)
/dev/sdc2 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdc2: 347/124928 files (28.0% non-contiguous), 115619/499712 blocks
neela@hooban:~$ sudo fsck /dev/sdc3
fsck from util-linux 2.34
neela@hooban:~$
fsck?