Rebuilding mdadm RAID 5 array with multiple failed drives

Question

I had a 5 drive Raid5 array and I've had two drives fail. I have fresh drives to swap out, but when I'm trying to start it it complains '/dev/md/0 assembled from 3 drives - not enough to start the array'

the superblock data from one of the original drives:

mdadm -E /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : eece6340:50c5f548:fec6e083:8e175d25
           Name : nas2:0  (local to host nas2)
  Creation Time : Sat Jun 11 15:08:47 2016
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 7813533696 (7451.57 GiB 8001.06 GB)
  Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=176 sectors
          State : clean
    Device UUID : f99f8f44:bbf30563:35183897:5563d56a

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Aug 12 15:01:12 2018
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 3c38edaa - correct
         Events : 364523

   Device Role : Active device 0
   Array State : AA..A ('A' == active, '.' == missing, 'R' == replacing)

and here is the output from attempting to assemble :

mdadm: /dev/sdb is identified as a member of /dev/md/0, slot 0.
mdadm: /dev/sdf is identified as a member of /dev/md/0, slot 4.
mdadm: /dev/sdc is identified as a member of /dev/md/0, slot 1.
mdadm: added /dev/sdc to /dev/md/0 as 1
mdadm: no uptodate device for slot 2 of /dev/md/0
mdadm: no uptodate device for slot 3 of /dev/md/0
mdadm: added /dev/sdf to /dev/md/0 as 4
mdadm: added /dev/sdb to /dev/md/0 as 0
mdadm: /dev/md/0 assembled from 3 drives - not enough to start the array.
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically

Is there anyway to clone superblock data (like dd) to a new drive to allow the array to be assembled?

Answer 1

You can't reassemble without data loss a RAID-5 array with more than one drive dead. In theory, you could read data from the rest of the drives, but you'd have chunks missing in between.

The structure of a 5-disk RAID-5 is something like this: D is data chunks, P parity chunks.

12345
DDDDP
DDDPD
DDPDD
DPDDD
PDDDD

With any two drives dead, you have 8 data chunks and 2 parity chunks missing out of each 20 data chunks and 5 parity chunks. The way the parity works, the system can recover a single missing data chunk if the parity is intact (per each row in the diagram). With two missing data chunks, there's no unique way to reconstruct the chunks. So the remaining parity chunks are useless, and you're left with 12 data chunks out of each 20. You've lost 40 % of the data.

According to the mdadm man page, the chunks are 512 kB by default, so you could find intact pieces of data up to 1.5 MB. But the filesystem code will not be happy to see major pieces of metadata missing and any files larger than that would with very high likelihood be missing parts, even if you could find the correct device blocks the file data was located.

Answer 2

RAID 5

Excerpt from Wikipedia article:

It requires that all drives but one be present to operate. Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is lost.

Therefore my conclusion is, if those two drives are real dead, it is now impossible to assemble the array, sorry.