Check read-only Btrfs volume

Question

On a system with Ubuntu 18.04 there is a Btrfs volume mounted as /home. It suddenly switched to read-only mode out of his own volition. In the fstab file there is the following:

UUID=56b07769-992f-4c55-b620-c35d399a4c55 /home btrfs defaults,subvol=@home 0 2

df reports this volume as the /dev/sda2 device:

/dev/sda2  2,8T  2,1T  686G  76% /home

The logs report problems with two sectors, which is bad, but possibly not catastrophic:

$ dmesg
[...]
[255987.491752] sd 2:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[255987.491754] sd 2:0:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 05 f8 80 00 00 00 08 00 00
[255987.491755] print_req_error: I/O error, dev sda, sector 391296
[255987.491758] Buffer I/O error on dev sda2, logical block 16, async page read
[255995.569797] sd 2:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[255995.569800] sd 2:0:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 00 3f e9 2f 80 00 00 00 20 00 00
[255995.569801] print_req_error: I/O error, dev sda, sector 1072246656
[255995.569806] BTRFS error (device sda2): bdev /dev/sda2 errs: wr 5, rd 3371, flush 1, corrupt 0, gen 0

The first reaction was to remount the volume in read/write mode, but that resulted in an error:

$ sudo mount -o remount,rw /home
mount: /home: mount point not mounted or bad option.

Then I tried checking the health of the volume, but surprisingly, btrfs is not recognising the volume:

$ sudo btrfs check --force -p /dev/sda2
WARNING: filesystem mounted, continuing because of --force
No valid Btrfs found on /dev/sda2
ERROR: cannot open file system

What else can be done to mark those bad sectors and re-mount this volume in read-write mode?

Answer 1

1. Shutdown your system.
2. Boot from a USB-stick (or any other live medium) with a Linux distro
3. After booting install tools for messing around with btrfs. (I am assuming your usb-stick is also a ubuntu): sudo apt install btrfs-progs on Ubuntu 20 or sudo apt install btrfs-tools on older systems. (It will be installed in memory)
4. Install tools to mess around with the disk at hardware level: sudo apt install smartmontools
5. Mount your disk read-only: mkdir /mnt/baddisk && mount -o ro /dev/sdb1 /mnt/baddisk (sdb1 will probably be different in your case)
6. Backup your data: rsync -Havz /mnt/baddisk someothersystem:/dataofbuggydisk/

Only now you can play around.

• Use smartctl -a /dev/sdb to see if reallocation is actually still a option. Reallocation will be handled low level by your disk itself, not by btrfs.
  If the disk can't do this you will certainly not be able to do this.
• You can play around with btrfs rescue and btrfs check if needed. But if you only have a couple of bad sectors this will probably not be needed. A regular Ubuntu config makes sure that the filesystem is always in the best condition possible.
  Normally the only thing you should worry about is what SMART tells you about this disk. Use smartctl to check it from to time. (It could be that nowadays Ubuntu also does this for you and warns you when necessary but I'm not sure about this, also do it manually just to be safe.)

Edit: I just noticed that you mentioned that Ubuntu placed your disk in read-only mode. When it decides to do this you have real problems. This can usually only happen when:

• The system was shutdown uncleanly, but you don't mention it so it's probably not the case
• You have low-level (disk, not filesystem) problems.

I would strongly advice not to use this disk again. In theory it's certainly possible to recover the filesystem and continue using this disk for a while, but you will end up with large problems sooner or later...