2

I hope you´re doing well.

I work as a technician in an IT company focused on Windows systems and cloud stuff, hence my knowledge to Linux is sadly very limited. So please excuse any dumb questions but I´ll try to be as helpfull as possible. Also this is my first time posting here, so pleas tell me if I do something wrong.

So here´s the story: A new customer called and said his server is not reachable -> Server is dead (powersupply and and motherboard broke, even with new PS not even a POST beep). His old IT company was a oneman show, he unfortunately died, so no help from that side..

The Server is a 15 Year old Fujitsu with a LSI logic RAID embedded. Inside we found 2x 2TB SATA HDDs connected to the MB. All his data aswell as his software database file are on this server. Of course there´s not a backup.. he doesnt need one because eveything is mirrored.. you know. Also the server OS was setup only 2-3 years ago the customer stated.

So I started with some recoverytools like Diskinternals RAID Recovery but those did not really work out. I only got single files (some of them were functional docs so the Disk itself seems ok) but no folders or such. To have the customers software restored to another system, we need a complete folder, subfolders and files.

But what I found were files and folders only present on a Linux system. So I think the previous technician installed a Linux OS and set up a network share for the customer.

So I pulled a dump from one of the Raid memberdisks to another HDD and set up a Debian machine for further testing. I´m still not sure if he set up a linux / mdadm Raid or if he did it using the onboard LSI Raid controller.

Until now I had no luck mounting or reassembling the disk. Any help would be greatly appreciated.

  • mount /dev/sdb /mnt/mountpoint brings error wrong FS, bad options, corrupted superblock
  • Disk is not shown as md in /dev/

lsblk:

sdb      8:16   0   1,8T  0 disk 
├─sdb1   8:17   0   240G  0 part 
└─sdb2   8:18   0   1,6T  0 part

fdisk -l

Festplatte /dev/sdb: 1,82 TiB, 2000398934016 Bytes, 488378646 Sektoren
Festplattenmodell: EFAX-68FB5N0    
Einheiten: Sektoren von 1 * 4096 = 4096 Bytes
Sektorgröße (logisch/physikalisch): 4096 Bytes / 4096 Bytes
E/A-Größe (minimal/optimal): 4096 Bytes / 4096 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x87c99aec

Gerät      Boot     Anfang       Ende   Sektoren Größe Kn Typ
/dev/sdb1  *          2048   62916607   62914560  240G fd Linux RAID-Autoerkennung
/dev/sdb2         62916608 3897729167 3834812560 14,3T fd Linux RAID-Autoerkennung
/dev/sdb3       3897729168 3907029167    9300000 35,5G fd Linux RAID-Autoerkennung

mdadm --query /dev/sdb

/dev/sdb: is not an md array

mdadm --assemble --scan

mdadm: No arrays found in config file or automatically

mdadm --examine /dev/sdb

mdadm: No md superblock detected on /dev/sdb

Thanks in advance for any help or tips, KofftheHoff

Edit1:

After using

losetup --find --show --read-only --sector-size 512 --partscan /dev/sdb

and

mdadm --examine /dev/loop*

I get this wich looks promising:

/dev/loop0:
   MBR Magic : aa55
Partition[0] :     62914560 sectors at         2048 (type fd)
Partition[1] :   3834812560 sectors at     62916608 (type fd)
Partition[2] :      9300000 sectors at   3897729168 (type fd)
/dev/loop0p1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7e3b9767:71d0e46d:6fe589d3:47671ff3
           Name : schobert-fs:0
  Creation Time : Wed Mar 27 18:49:49 2019
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 62881792 (29.98 GiB 32.20 GB)
     Array Size : 31440896 (29.98 GiB 32.20 GB)
    Data Offset : 32768 sectors
   Super Offset : 8 sectors
   Unused Space : before=32680 sectors, after=0 sectors
          State : clean
    Device UUID : 34f9240c:3f35c4a9:b20f6259:5a6a295e

    Update Time : Fri Dec  2 16:10:26 2022
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9882cebb - correct
         Events : 430


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/loop0p2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 1c73d398:e5404786:1cba7820:fb5e4cd5
           Name : schobert-fs:1
  Creation Time : Wed Mar 27 18:50:09 2019
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3834550416 (1828.46 GiB 1963.29 GB)
     Array Size : 1917275200 (1828.46 GiB 1963.29 GB)
  Used Dev Size : 3834550400 (1828.46 GiB 1963.29 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=16 sectors
          State : clean
    Device UUID : 038f5d97:741a9a29:c4803eec:d5502d4b

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Nov 30 10:19:49 2022
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5dcb018a - correct
         Events : 12662


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/loop0p3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 00bd818b:55eed0df:3ad0c7d3:0c7a3a97
           Name : schobert-fs:2
  Creation Time : Wed Mar 27 18:50:31 2019
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 9291808 (4.43 GiB 4.76 GB)
     Array Size : 4645888 (4.43 GiB 4.76 GB)
  Used Dev Size : 9291776 (4.43 GiB 4.76 GB)
    Data Offset : 8192 sectors
   Super Offset : 8 sectors
   Unused Space : before=8104 sectors, after=32 sectors
          State : clean
    Device UUID : 1577f59e:d2022fbc:d0b79765:f127efbc

    Update Time : Wed Nov 30 00:01:49 2022
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : dc81c5d0 - correct
         Events : 92


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/loop1.
mdadm: No md superblock detected on /dev/loop2.
mdadm: No md superblock detected on /dev/loop3.
mdadm: No md superblock detected on /dev/loop4.
mdadm: No md superblock detected on /dev/loop5.
mdadm: No md superblock detected on /dev/loop6.
mdadm: No md superblock detected on /dev/loop7.
mdadm: cannot open /dev/loop-control: Invalid argument

Thanks @frostschutz, also for the tip with pulling an image. I´m working with a dumped disk right now, the originals stay untouched.

@gabor.zed Do those different names schobert-fs:0 /schobert-fs:1 /schobert-fs:2 prove your assumption? Or is the number after : just a marker what drive it was in the raid?

lsblk now brings this:

NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0       7:0    0   1,8T  1 loop 
├─loop0p1 259:0    0    30G  1 part 
├─loop0p2 259:1    0   1,8T  1 part 
└─loop0p3 259:2    0   4,4G  1 part

fdisk now looks reasonable too:

Festplatte /dev/loop0: 1,82 TiB, 2000398934016 Bytes, 3907029168 Sektoren
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 512 Bytes
E/A-Größe (minimal/optimal): 512 Bytes / 512 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0x87c99aec

Gerät        Boot     Anfang       Ende   Sektoren Größe Kn Typ
/dev/loop0p1 *          2048   62916607   62914560   30G fd Linux RAID-Autoerkennung
/dev/loop0p2        62916608 3897729167 3834812560  1,8T fd Linux RAID-Autoerkennung
/dev/loop0p3      3897729168 3907029167    9300000  4,4G fd Linux RAID-Autoerkennung

I even get 3 md devices now listed in /dev/

md125 md126 md127

mdadm --query --detail  /dev/md126

/dev/md125:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 1
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 1

              Name : schobert-fs:2
              UUID : 00bd818b:55eed0df:3ad0c7d3:0c7a3a97
            Events : 92

    Number   Major   Minor   RaidDevice

       -     259        2        -        /dev/loop0p3


mdadm --query --detail /dev/md126

/dev/md126:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 1
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 1

              Name : schobert-fs:1
              UUID : 1c73d398:e5404786:1cba7820:fb5e4cd5
            Events : 12662

    Number   Major   Minor   RaidDevice

       -     259        1        -        /dev/loop0p2


mdadm --query --detail /dev/md127

/dev/md127:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 1
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 1

              Name : schobert-fs:0
          UUID : 7e3b9767:71d0e46d:6fe589d3:47671ff3
        Events : 430

Number   Major   Minor   RaidDevice

   -     259        0        -        /dev/loop0p1

The md devices cannot be mounted by themself, right? And whats somehow odd is that he states the md devices are Raid0. Don´t know what to make of that right now.

At least when I try:

mount -o ro -t auto /dev/md125 /mnt/raid1

i get:

mount: /mnt/raid1: Der Superblock von /dev/md125 konnte nicht gelesen werden.

Superblock cannot be read

I think i have to assemble the raid somehow before accessing it?

Edit 2: @frostschutz i ran as requested:

file -s /dev/md*

/dev/md125: empty
/dev/md126: empty
/dev/md127: empty

and

blkid

/dev/sda1: UUID="ae7d369d-cf6b-4f84-a010-5d8a4c6fac80" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="248a7be6-01"
/dev/sda5: UUID="e49af320-e5fc-45fa-91b4-528b231f0bbd" TYPE="swap" PARTUUID="248a7be6-05"
/dev/sdb1: PARTUUID="87c99aec-01"
/dev/loop0p1: UUID="7e3b9767-71d0-e46d-6fe5-89d347671ff3" UUID_SUB="34f9240c-3f35-c4a9-b20f-62595a6a295e" LABEL="schobert-fs:0" TYPE="linux_raid_member" PARTUUID="87c99aec-01"
/dev/loop0p2: UUID="1c73d398-e540-4786-1cba-7820fb5e4cd5" UUID_SUB="038f5d97-741a-9a29-c480-3eecd5502d4b" LABEL="schobert-fs:1" TYPE="linux_raid_member" PARTUUID="87c99aec-02"
/dev/loop0p3: UUID="00bd818b-55ee-d0df-3ad0-c7d30c7a3a97" UUID_SUB="1577f59e-d202-2fbc-d0b7-9765f127efbc" LABEL="schobert-fs:2" TYPE="linux_raid_member" PARTUUID="87c99aec-03"

Edit3:

So since I think all the data is located on md126 next up I ran:

mdadm --stop /dev/md126
mdadm: stopped /dev/md126

After that i tried auto assemble and it put the md126 back together and it seems to work since it came up with the raidname and a new device under /dev/md/

mdadm --assemble --scan
mdadm: /dev/md/schobert-fs:1 has been started with 1 drive (out of 2).

I tried mounting it then, but it keeps saying it can´t because its read only. Makes sense because the loop device ist in read only mode. But Should´nt it run when iI mount it in read only mode also with the option -o ro?

mount -o ro /dev/md/schobert-fs\:1 /mnt/raid1
mount: /mnt/raid1: /dev/md126 konnte nicht im Lese-Schreib-Modus eingehängt werden, (Medium) ist schreibgeschützt..

Edit 4:

hoooray, I got it!

Found the last hint in the mount manual:

-r, --read-only

Mount the filesystem read-only. A synonym is -o ro.

Note that, depending on the filesystem type, state and kernel behavior, the system may still write to the device. For example, Ext3 or ext4 will replay its journal if the filesystem is dirty. To prevent this kind of write access, you may want to mount ext3 or ext4 filesystem with ro,noload mount options or set the block device to read-only mode, see command blockdev(8).

[…]

norecovery/noload

Don't load the journal on mounting. Note that if the filesystem was not unmounted cleanly, skipping the journal replay will lead to the filesystem containing inconsistencies that can lead to any number of problems.

So i ran:

mount -o ro,noload /dev/md/schobert-fs\:1 /mnt/raid

and voila, all the files are there!

Massive thanks to @frostschutz and @gabor.zed for helping me out!

Have a good day all.

Improve this question
5
  • I think he mirrored by partition, not full disk. so you have 3 md array. try mdadm --examine /dev/sdb1 If it works, you can "assemble" the 3 array, with the 3 partitions only from one disk. If it doesn't work, try the other disk. Commented Dec 14, 2022 at 17:21
  • good morning @gabor.zed . Thanks for your post. Yeah after inspecting the loop thingies this could be. I´ll post more on that in themain post. Commented Dec 15, 2022 at 8:20
  • file -s /dev/md*, blkid, ...? Commented Dec 15, 2022 at 9:46
  • oh, they're all inactive... you might have to --stop and --assemble --run them yourself, if your system does not run degraded raid by default. you can also check dmesg Commented Dec 15, 2022 at 9:47
  • I ran the commands you named and put them in main post @frostschutz Not sure how to use --stop and other options of mdadm. Do I use them with the md or loop devices? When I just use mdadm --stop it states: mdadm: No devices given. Commented Dec 15, 2022 at 11:50

1 Answer 1

Reset to default
2

The main problem here is not the RAID but a bogus partition table. The partition table is made for 512 byte sectors however the drive is detected as 4K native sectors. So all partition offsets and sizes are completely wrong.

You might be able to work around it with losetup:

losetup --find --show --read-only --sector-size 512 --partscan /dev/sdb

And then see if the loop device has valid partitions and mdadm metadata:

mdadm --examine /dev/loop*

And then go on from there, hopefully with no further problems.

If there's any chance that the drives might be defective, you can also consider pulling an image with ddrescue first. Image files would also default to 512 byte sector handling usually.

Do everything read-only or use copy-on-write overlays.

Improve this answer
2
  • Good morning and thanks for your answer! You were right it seems. I´ll put the results in the main post. But I still don´t get how to access the raid. Commented Dec 15, 2022 at 8:18
  • Thanks for sharing the copy-on-write overlay! Such an awesome capability! Commented May 21, 2023 at 21:02

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.