16

I'm currently in the process of setting up a NAS system mainly for backing up my computer and probably also for streaming media files to my devices.

For these purposes, I decided to use OpenMediaVault with two new 4GB drives as it seems that it's a stable running software and that it has all the needed features already built in.

But it's not very clear yet which filesystem to choose. I'm currently tending to go with BTRFS instead of ext4 as I really want to have the "self-healing" as well as simple snapshots.

Apart from that, I would like to set up my drives in RAID 1 so I can benefit from a higher data availability.

Now when it comes to BTRFS, there are two options on how to set up my RAID 1.

Either one is using the standard Linux software raid with mdadm and the other one would be "native" btrfs raid.

What are the advantages/disadvantages of each?

1
  • 4
    Close voters: this is not primarily opinion based. Commented Nov 7, 2018 at 17:04

4 Answers 4

6

The current status of btrfs can be found on the btrfs wiki's “Status” article. btrfs RAID1 has been stable for a while, and has several useful features that mdraid does not (e.g., mirror is per-file, so you can have unmirrored files too if you want—though tools to do this aren't readily available). Also, adding a third drive (when you need more space) is much easier with btrfs than mdraid.

mdraid's main advantage is it's far more mature. So mdraid+ext4 is probably a little bit safer. Documentation & experts are easier to find. But if you're running btrfs on top of it anyway (because, e.g., you want snapshots)….

Personally, I have a set of external drives I use for one layer of backups; they use btrfs. I started with one drive (back in 2014), then later added a second (larger) drive for redundancy. Later the first drive failed, so I replaced it with another larger drive. Then I was going to run out of space, so I added a third drive, twice as big as the other two. Sometime in the next year or so, I'll have to add another. All of this would be much more painful (or at least time-consuming) to do with mdraid than it was with btrfs RAID1.

So, for your application, I'd suggest btrfs RAID1 is a better choice. If you were running a mission-critical darn-near-100% uptime required server, it'd be different. And of course, as always, keep backups (of your media files, at least if you care if they're lost — you don't want to know how many copies of my music collection I have...)

2
  • The last I heard, the filesystem is capable of having different chunks use different raid levels, but there was no user space method of flagging different files to be stored in a different chunk type. Is there a command for doing that these days? Commented Nov 7, 2018 at 17:26
  • @psusi I feel I've seen an a way to do that somewhere, but can't find it right now ☹. I'm sure the drange filter could be used, but that'd be a PITA. Commented Nov 7, 2018 at 17:33
6

The main tradeoff is pretty simple to understand:

  • BTRFS has better data safety, because the checksumming lets it ID which copy of a block is wrong when only one is wrong, and means it can tell if both copies are bad.
  • MD RAID has better performance, because it does a better job of parallelizing writes and striping reads.

Realistically, raid1 mode is the most well tested of the various multi-device configurations provided by BTRFS, by virtue of the facts that it's one of the oldest supported configurations, and it's also one of the most widely used. Most of the developers consider it stable enough (if you're not using complicated features) to be a realistic option in production usage, provided of course that you stay up to date.

6
  • I used btrfs multi device volume as a main volume for a good while. What's funny is when the data safety has to come into play (something goes south, logical errors reported) and you'd need to "fsck" it (quoting fsck intentionally, since btrfs has different mechanism), it's incredibly painful PITA. You'd think to somehow mount it in a recovery mode and then "fsck", it's almost impossible. Commented Feb 20, 2019 at 21:24
  • 2
    RHEL 8 and Centos 8 dropped BTRFS support. hit this wall so... fyi Commented Oct 20, 2020 at 14:07
  • 2
    @TecHunter And most of that was them deciding they didn’t want to support people who had no idea what they were doing. You shouldn’t be using any software for production workloads if you don’t understand it’s limitations, BTRFS is just a bit less forgiving in that respect than most other options. Commented Oct 21, 2020 at 2:04
  • @AustinHemmelgarn yes, it motivated me to move to MD Raid. Commented Oct 22, 2020 at 7:27
  • 2
    Well, BTRFS has checksumming and MD RAID does not. MD RAID does NOT guaranty data integrity at all. If a one disk in a MD mirror contains/returns corrupted data, this goes unnoticed and you cannot tell wich side of the mirror is the correct one! BTRFS would detect and fix this automatically. You can even scrub it. And BTRFS does support stripping. Due to its cow nature it is always consistent and snapshots are very efficient. From a data integrity and safety point of view, BTRFS is superior to MD+ext4. Commented Jan 5, 2021 at 16:12
5

One advantage of mdadm raid is that it is well understood and stable. Also grub was previously not able to boot from multi device btrfs filesystems. There may be other software bugs with such setups. You also could use mdadm to create a raid10 instead of a raid1. Using the offset layout with a largeish chunk size gets you much of the performance of a raid0 with the redundancy of a raid1.

One advantage of the native btrfs method is that you can decide to convert from raid1 to raid0 later. Another theoretical advantage is that you can mix and match and put some files in raid0 and some in raid1, but the last time I checked at least, the tools weren't up to that yet.

1
  • 2
    It's indeed possible to boot onto a multi-device "RAID0" btrfs volume, this is back from 2016 back askubuntu.com/questions/794763/…, I got that working as the only and main volume. But the installer was not able to understand the situation, failed, needed black magic and tricks to go through. Now 2 generations later I won't use any RAID, because I'm not sure about the speed benefits (phoronix.com/…) and it just complicates everything. Commented Feb 20, 2019 at 21:28
4

checksum

First, md-raid do have checksum and parity checks.

# echo check > /sys/block/md127/md/sync_action
# cat /proc/mdstat 
Personalities : [raid1] 
md127 : active raid1 sdc1[0] sdb1[1]
      1855337472 blocks super 1.2 [2/2] [UU]
      [=================>...]  check = 87.8% (1629808384/1855337472) finish=18.7min speed=200062K/sec
      bitmap: 0/14 pages [0KB], 65536KB chunk

unused devices: <none>
#

So parity validation is not an advantage of either, as both have it. What btrfs have is an advantage over plain ext4 et al. Because it stores duplicate values (profile: DUP) and can (very rarely) recover even with a single disk. But that is irrelevant for raid1 discussion.

Administration

PRO md-raid: mdadm have integrated monitoring. You can add EMAIL or PROGRAM to your mdadm.conf and have it email/trigger a script on any event. Disk failed? progress of rebuild? progress of check? result of important changes? etc.

It is also more explicit on how you assemble it. You must assemble the md device and then you mount whatever is in the md device. With btrfs it will create N blockdevs with the same UUID, and the first time you mount them you must pass mount -o device=/dev/other /dev/main /mnt/point

layout flexibility

Both can be created and new devices added on most combinations. Directly on entire disk (/dev/sda) without partitions is OK. Both can be under a partition (/dev/sda1). Both can be under LVM. Both can be under dm-crypt (cryptsetup, luks). Both can be under dm-parity (but you don't wait it). etc.

PRO md-raid: You can host block devices underneath it. for example, luks or other filesystems.

CON md-raid: it is close to impossible to resize the thing hosting it.

PRO btrfs: you don't have to deal with several layers, mostly because it is all btrfs. Easy to resize (grow) partitions.

CON btrfs: if you plan to use dm-crypt/luks, if must be over btrfs, and I have no idea what will happen if you have corruption. With md-raid you can have luks underneath, and then it is predictable that the corruption will be fixed.

Both are the same in terms of adding more devices and spares.

I won't mention snapshots etc because you can use brtfs under md-raid and also have it.

RAID1 profiles

md-raid is only full copy. It might be better if you care for full reliability. It might be worse if you care for performance.

brtfs guarantee 2 copies at minimum, but not 1 copy on all devices! For example, if you have three disks, md-raid will have three copies of the data. btrfs will still have only two.

you can force more copies if you have 3 or more devices using the new (untested?) raid1c3 and raid1c4 profiles.

I won't mark which one have a pro, since md-raid is a PRO for redundancy and btrfs is a PRO for performance.

rough edges

mdadm et al will have very weird commands and syntax. But it is generally considered stable.

btrfs as everyone mentioned, even in 2025 have some rough edges: for example, you get some scary messages from the tooling:

# btrfs filesystem resize max /mnt/point
Resize device id 1 (/dev/sdb2) from 3.64TiB to max
WARNING: the new size 0 (0.00B) is < 256MiB, this may be rejected by kernel

which is some weird logic handling the word "max" but if you look at the logs it is mostly fine:

BTRFS info (device sdb2): resize device /dev/sdb2 (devid 1) from 4000000770048 to 8100829442048

(Also, keep in mind you must expand each "device" in the RAID, so run for each one btrfs filesystem resize 2:max /mnt/point increasing the number before :max... and afterwards, remember to btrfs rebalance /mnt/point)

So with mdadm you will be constantly looking up commands and conf file syntax. With btrfs you will be constantly looking up error messages to see if they are relevant :)

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.