3

I am migrating my home server from Windows to Ubuntu server 22.04. Some of the data I am migrating is on an NTFS drive and I want to transfer this data to a new (second) drive which is using XFS. However I am encountering very low transfer speeds ranging from 40 MB/s to 1 MB/s. Hdparm gives read speeds of around 230 MB/s unbuffered for the drives in question.

The drives are connected via motherboard SATA but I get similar results via an HBA (LSI 9211-8i IT mode) well. The drives are Seagate iron wolf 10Tb and 8Tb spinning disks. The motherboard is a gigabyte X399 aorus xtreme running an AMD threadripper 1920x. The files are mixed: video files (average 4GB) photos (8-20MB) and lots of smaller files >10 Mb.

While the drives were both connected in the Ubuntu server (via sata) I was getting transfer speeds of 1-3 MB/s using cp and around 7-8 MB/s using rsync. Not sure if it makes any difference but I formatted the XFS drives with the reverse mapping b-tree option (-m rmapbt=1).

The fstab entry for the xfs drive has the following options: auto,rw,sync,noexec,nodev,nosuid,nofail,x-gvfs-show,x-gvfs-name=sdisk2

When the ntfs drive was mounted in the Linux server it had the following fstab mount options: auto ro 0 0

From my basic google-fu I have managed to test:

  1. checking write-cache - confirmed enabled on the drives in question.
  2. potential bottleneck with NTFS drivers under Linux - see below.
  3. moving sata ports - this made no difference.

To test point 2 I connected the NTFS drive to a wired network windows box and then made the XFS drive available via samba - this increased the transfer speed to around 30-40MB/s - using explorer. So maybe part of the issue was the Linux NTFS driver. This is the best result thus far, but I assume this is well below what should be expected, at least based on my previous windows-only transfer experience.

I not sure what else to check. I am relatively new to Linux (learning a lot over the past two weeks) and would be grateful for any assistance.

Thanks,

FM

Improve this question
12
  • Linux perfectly supports NTFS using ntfs-3g or native ntfs3 driver. I'm not sure why you need to move data around at all. From your description it's impossible to say whether you're moving files within the same drive or between drivers. If it's the former, the copy speed will naturally be a lot lower. Commented Jul 10, 2022 at 12:12
  • 2
    Can confirm that ntfs-3g is very slow, so slow it's tedious for backup purposes, i.e. for a one-way copy-off-ntfs-to-xfs. That's mostly the result of it being an old userland driver, and maybe also code efficiency issues. The workaround I went with was a minimal Windows VM on the same computer, attaching the NTFS disk to that VM, and as you said, copying via shared folder (in my case, virtiofs, but the cost of samba/CIFS should be tolerable) Commented Jul 10, 2022 at 12:30
  • @ArtemS.Tashkinov between two separate drives - will amend my post to clarify. The reason is that I intend to use mergerfs and pool a number of xfs drives. I have no ongoing need for NTFS on the server and wish to take advantage of native linux filesystems. Commented Jul 10, 2022 at 12:40
  • @MarcusMüller thank you. That is an interesting way I had not considered. Will try this and see if it works any better. Commented Jul 10, 2022 at 12:41
  • 1
    If the OP uses something like Debian 9, I can imagine all sorts of issues. Again, I've offered to use newer or better utilities as well as the native ntfs3 driver which mustn't have this issue. Using a VM with Windows sounds like a total overkill and a complete waste of time. Commented Jul 10, 2022 at 13:01

2 Answers 2

Reset to default
1

Could you try copying files this way?

tar c source/directory | pv -petrab | tar -C /destination/directory -x

(with the hint by @MarcusMüller)

Improve this answer
1
  • I like this approach by the way (+1 from me!), and I think you can, since tar really doesn't care about any behaviour of the output stream, insert | pv between, i.e. tar c source/directory | pv --average-rate | tar -C /destination/directory -x, and get live statistics. Commented Jul 10, 2022 at 13:26
0

Thank you both for your answers. Both have been very useful and I am very pleased to say worked a treat - back up to ~230Mb/s. For good measure I reformatted the target drive to ext4 and tried both options and both gave very comparable results. I've opted for the tar command, mostly because I am increasingly becoming more comfortable operating via cli.

Just so that I understood the tar command you've kindly provided, the start (tar -c) creates an archive and compresses the source folder then pipes this in some streaming fashion to be decompressed (tar -C -x) at the target folder, with the intermediate PV command providing progress, stats etc. Is this correct?

Apologies for putting this in the answer section. I originally posted as an unregistered user and the password/registration email process seems to have created a new user with the same name! And this is the only place I can post this. In any case you have my thanks.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.