Unable to Transfer Large Files To External Hardrive

Question

When trying to transfer files 200MiB or over to an external drive (WD Elements 20TB) on the Raspberry Pi it freezes and causes the system to crash. This issue doesn't occur when transferring the same file on Windows. It also doesn't happen when copying files off of the drive.

I've looked in /var/log/syslog, /var/log/messages and /var/log/kern.log but can't see any activity around the time of the crash.

From the 2 drives below, there are no troubles transferring to sda the issue only occurs on sdb.

Dec  6 18:35:47 raspberrypi kernel: [   15.755155] .ready
Dec  6 18:35:47 raspberrypi kernel: [   15.756114] sd 1:0:0:0: [sda] 31251759103 512-byte logical blocks: (16.0 TB/14.6 TiB)
Dec  6 18:35:47 raspberrypi kernel: [   15.805511] sd 1:0:0:0: [sda] Write Protect is off
Dec  6 18:35:47 raspberrypi kernel: [   15.805531] sd 1:0:0:0: [sda] Mode Sense: 4f 00 00 00
Dec  6 18:35:47 raspberrypi kernel: [   15.805907] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Dec  6 18:35:47 raspberrypi kernel: [   15.806645] sd 1:0:0:0: [sda] Optimal transfer size 33553920 bytes
Dec  6 18:35:47 raspberrypi kernel: [   15.944304]  sda: sda1 sda2
Dec  6 18:35:47 raspberrypi kernel: [   15.947731] sd 1:0:0:0: [sda] Attached SCSI disk
Dec  6 18:35:47 raspberrypi kernel: [   16.555156] .......ready
Dec  6 18:35:47 raspberrypi kernel: [   22.795753] sd 0:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
Dec  6 18:35:47 raspberrypi kernel: [   22.796071] sd 0:0:0:0: [sdb] 39063584768 512-byte logical blocks: (20.0 TB/18.2 TiB)
Dec  6 18:35:47 raspberrypi kernel: [   22.796091] sd 0:0:0:0: [sdb] 4096-byte physical blocks
Dec  6 18:35:47 raspberrypi kernel: [   22.796873] sd 0:0:0:0: [sdb] Write Protect is off
Dec  6 18:35:47 raspberrypi kernel: [   22.796894] sd 0:0:0:0: [sdb] Mode Sense: 47 00 10 08
Dec  6 18:35:47 raspberrypi kernel: [   22.797738] sd 0:0:0:0: [sdb] No Caching mode page found
Dec  6 18:35:47 raspberrypi kernel: [   22.797829] sd 0:0:0:0: [sdb] Assuming drive cache: write through
Dec  6 18:35:47 raspberrypi kernel: [   23.084630] Alternate GPT is invalid, using primary GPT.
Dec  6 18:35:47 raspberrypi kernel: [   23.084674]  sdb: sdb1
Dec  6 18:35:47 raspberrypi kernel: [   23.088174] sd 0:0:0:0: [sdb] Attached SCSI disk

OS Info:

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

4GB Model

Running the old 32Bit OS

Drive trying to copy to:

WD Elements 20tb formatted for NTFS

I've tried enabling Write Caching when the drive is plugged into a Windows Machine.

Possibly not an isolated issue with these drives and Raspberry Pi's https://community.wd.com/t/how-to-get-wd-elements-drive-to-work-on-raspberry-pi/275999

Answer 1

This was most likely an OS issue. It appears 32bit OS's are funny with anything over 2TB and may not be possible (at least easily) to get them working with anything over 16TB

So I've installed the 64bit version of Raspbian on a spare SD card and copied changes over, and it appears to have worked. I'm also not getting the Alternate GPT is invalid, using primary GPT error in /var/log/syslog

How I came to this conclusion:

• Found a comment on the Amazon review that someone else found it worked once formatting to ext4. Issue is I kind of want it to work with both Linux and Windows, and Windows doesn’t support ext4 (but reading some more modern articles maybe it does now).

• Found the same issue in WD forum

• Another posting in Raspberry Pi forum

However there were some comments that mentioned an issue about the 32bit OS, and the user should upgrade to a 64bit OS. But no comments as to whether that worked or not. This led me onto this posting in SuperUser.

Although it’s aimed at Windows it's likely it could be the same issue for Linux. That the maximum partition size a drive can deal with is 16TB. Which explains why sda works (at it is 16TB) and that sba doesn’t as it’s greater than 16TB.

The next step was to try the 64bit OS, and copy some files over.

So likely not a common issue, as unless you’re running a Raspberry Pi you’re unlikely to be running a 32bit OS these days.