Creating a compressed image of an exFAT USB drive using dd

Question

My system is running Ubuntu 22.04.3 LTS. I plug a USB drive into that system. That USB drive (/dev/sdb) contains an Ubuntu installation, mostly in an ext4 partition (/dev/sdb3). That installation fills about 25GB of drive space.

With /dev/sdb3 unmounted, I use sudo zerofree -v /dev/sdb3 to zero free space on that USB drive. Then I use the following command to create a compressed image:

sudo dd if=/dev/sdb bs=16M conv=sync,noerror | pv | sudo pigz -c > /media/TargetDrive/UbuntuImage.dd.gz

This seems effective. The resulting .gz file weighs only 12GB.

Now I remove that USB drive (/dev/sdb) and replace it with a USB drive containing a YUMI exFAT installation.

When I seek information on this /dev/sdb drive using GParted, I get a warning - "Unable to read the contents of this file system!" - and an indication that exfatprogs is required for exfat file system support. I install exfatprogs and restart GParted. That eliminates that warning.

This YUMI exFAT installation fills 43GB on the USB drive. I understand that zerofree and Microsoft's sdelete -z E: don't zerofill on exFAT, so I follow advice that seems to recommend the following:

sudo mount -o rw /dev/sdbX /mnt
sudo dd if=/dev/zero of=/mnt/zero_file bs=32M
sudo sync
sudo rm /mnt/zero_file
sudo umount /dev/sdbX

Those commands duly produce and then remove a zero_file whose size does appear to fill empty space. (I saw but did not try this alternative, which was less clear to me: cat > /dev/sdbX/zeros < /dev/zero ; sync ; rm /dev/sdbX/zeros.)

Then I run substantially the dd command shown above. This time, dd and pigz do not achieve comparable compression. The 43GB exFAT installation is saved in an image file of 42GB.

Granted, the exFAT drive's contents are produced by Windows. But experience with Windows drive imaging software indicates that I should expect substantial compression.

My question: can the dd command be improved to yield better compression for this exFAT drive?

Note: I seem to have obtained similarly poor compression with an NTFS USB drive, but did not document each step in that case.

Answer 1

(This didn't fit in a comment, so trying here.)

Excellent question. Unfortunately I'm not sure - that process is pretty much exactly what I would do. You don't say how big your "zeros" file is. If it's really small there just isn't any space available on the drive. The other alternative is that these disks are somehow automatically encrypted, in which case there won't be any actual zeroes written.

Another option, since you mentioned partitions, is that the partition you're writing to is small and there's a bunch of free space that you can't access but which you capture when you image the entire device. If that's the case, you either need to create a new filesystem that uses the rest of the space on the drive that you can write zeros to, OR write zeroes to the entire device (e.g, /dev/sdb, but be careful!) before you put the YUMI/etc. image on top of it.

Regarding this comment:

cat > /dev/sdbX/zeros < /dev/zero ; sync ; rm /dev/sdbX/zeros

That doesn't work. /dev/sdb is the entire device; /dev/sdbX is a partition on a device; /dev/sdb/whatever doesn't exist. The partition (or alternately, full device) needs a mounted filesystem in order to hold a file called zeros. That aside, the command is the same as cat /dev/zero > /path/to/zeros if that's clearer to you.

I would also suggest using bzip2 or xz in your pipeline if you want better compression. Those are both widely supported compression formats. The YUMI install image is probably already in a disk image that gets uncompressed when it boots so you'll get very little additional compression from imaging it.

Edit to add: zstd appears to be the currently-preferred compression algorithm. It has exceptional speed/size tradeoffs, and if time is no object also has some of the highest compression ratios. More information available here:

• https://etbe.coker.com.au/2020/06/06/comparing-compression/
• https://linuxreviews.org/Comparison_of_Compression_Algorithms