Why is rsync so slow when working with MTP devices?

Question

In order to perform regular backup of my phone (Samsung A5 2016, or SM-A510F) using rsync, I used the following command:

rsync --verbose --progress --omit-dir-times --no-perms --recursive --inplace --progress /run/user/1000/gvfs/mtp\:host\=%5Busb%3A002%2C018%5D/Phone/ /data/phone_back_folder/

1000 is, obviously, the UID of my user
mtp\:host\=%5Busb%3A002%2C018%5D being a temporary MTP host name given to my phone when mounted (which is somehow changing every time the device is mounted)
--omit-dir-times and --no-perms are here because MTP does not them apparently

Since I use the --progress flag, I can see the estimated transmitting speed for each file, by instance:

Phone/DCIM/Camera/20180408_184356.jpg
      2,814,075 100%   16.58MB/s    0:00:00 (xfr#265, to-chk=821/1262)

But these figures are hard to believe, since, most of the time, it seems that the system is idling, waiting for the next file to be transferred.

$ dstat -cd --disk-util --disk-tps
----total-cpu-usage---- -dsk/total- sda--sdb--sdc- -dsk/total-
usr sys idl wai hiq siq| read  writ|util:util:util|reads writs
  6   1  93   0   0   0| 344k  166k|0.03:0.37:1.43|   5     6 
  2   0  98   0   0   0|4096B    0 |   0:   0:   0|   0     0 
  3   0  97   0   0   0|   0    52k|   0:1.20:   0|   0     2 
 13   1  87   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   0  97   0   0   0|   0   152k|0.80:   0:   0|   0     2 
  3   0  97   0   0   0|   0   648k|   0:3.20:   0|   0    78 
  4   0  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   0  97   0   0   0|   0     0 |   0:   0:   0|   0     0 
  7   1  92   0   0   0|   0     0 |   0:   0:   0|   0     0 
  5   0  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  6   1  94   0   0   0|   0    12k|   0:0.40:   0|   0     2 
  6   1  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  6   0  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  7   1  91   0   0   0|   0   188k|   0:0.80:   0|   0     3 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   0  96   0   0   0|4096B   44M|8.80:   0:   0|   1   189 
  7   1  92   0   0   0|   0   172k|   0:3.20:   0|   0    18 
  6   1  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   0  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  6   1  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  6   0  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  7   2  91   0   0   0|   0   116k|   0:0.80:   0|   0     4 
  5   1  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   0  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  5   0  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  5   0  94   0   0   0|   0   224k|   0:0.80:   0|   0     3 
  5   1  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  5   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
 12   3  85   0   0   0|   0     0 |   0:   0:   0|   0     0 
  8   1  91   0   0   0|   0   604k|   0:   0:   0|   0    59 
  5   0  94   0   0   0|   0    84k|   0:1.20:   0|   0     2 
  5   0  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   0  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   0  97   0   0   0|   0     0 |   0:   0:   0|   0     0

Update: I ran another time dstat after unmounting and remounting the MTP device, and it started faster and falling back again to idling for most of the time.

$ dstat -cd --disk-util --disk-tps
----total-cpu-usage---- -dsk/total- sda--sdb--sdc- -dsk/total-
usr sys idl wai hiq siq| read  writ|util:util:util|reads writs
  5   1  94   0   0   0| 219k  603k|0.02:0.36:1.59|   3     7 
  6   3  90   1   0   0|  12k   25M|   0:   0:12.8|   3    44 
  5   2  86   7   0   0|   0    41M|   0:   0:64.0|   0    68 
  5   3  92   0   0   0|4096B 2744k|   0:2.00:1.60|   1   101 
  5   2  92   0   0   0|  20k    0 |   0:   0:3.60|   5     0 
  5   2  92   0   0   0|  16k    0 |   0:   0:2.40|   4     0 
  6   3  90   1   0   0|  12k   17M|   0:   0:14.0|   3    27 
  5   2  78  14   0   0|  12k   69M|   0:   0:92.0|   3   113 
  7   2  90   0   0   0|  12k    0 |   0:   0:2.80|   3     0 
  5   3  92   0   0   0|  12k   68k|   0:0.40:2.00|   3     2 
  6   2  92   0   0   0|  20k    0 |   0:   0:2.40|   5     0 
  5   2  94   0   0   0|4096B    0 |   0:   0:0.80|   1     0 
  3   1  91   5   0   0|   0    62M|   0:   0:45.2|   0   106 
  4   1  90   6   0   0|   0  2500k|   0:1.20:49.6|   0    67 
  4   1  95   0   0   0|   0   104k|   0:0.40:   0|   0     2 
  4   1  95   0   0   0|   0   192k|   0:0.80:   0|   0     3 
  4   2  94   0   0   0|   0    28k|0.40:   0:   0|   0     3 
  4   1  95   0   0   0|   0    40k|   0:   0:2.40|   0     2 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   2  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   1  95   0   0   0|   0    48k|   0:0.40:   0|   0     2 
  3   1  95   0   0   0|   0    16k|   0:   0:   0|   0     4 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   2  94   0   0   0|   0   332k|   0:   0:   0|   0    59 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   2  95   0   0   0|   0    16k|   0:0.80:   0|   0     2 
  3   1  96   0   0   0|   0     0 |   0:   0:38.4|   0     0 
  6   2  92   0   0   0|   0   104k|   0:   0:10.4|   0    14 
  5   2  93   0   0   0|   0     0 |   0:   0:   0|   0     0 
  4   1  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  95   0   0   0|   0   448k|   0:   0:   0|   0     1 
  3   1  95   0   0   0|   0    48k|   0:0.80:   0|   0     2 
  3   1  96   0   0   0|   0    32k|   0:   0:   0|   0     7 
  4   1  94   0   0   0|   0   548k|   0:2.00:   0|   0    66 
  4   1  95   0   0   0|   0    92k|   0:1.60:   0|   0     9 
  3   1  96   0   0   0|   0    28k|   0:1.20:   0|   0     3 
  7   1  92   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   2  95   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   1  96   0   0   0|   0   384k|   0:   0:   0|   0    44 
  4   2  95   0   0   0|   0    60k|   0:0.80:   0|   0     2 
  3   1  96   0   0   0|   0     0 |   0:   0:2.40|   0     0 
  3   1  96   0   0   0|   0  2560k|   0:   0:47.6|   0     5 
  4   2  94   0   0   0|   0     0 |   0:   0:   0|   0     0 
  3   2  95   0   0   0|   0   520k|   0:0.40:   0|   0    48 
  3   1  96   0   0   0|   0     0 |   0:   0:   0|   0     0

I was wondering if everything was normal, and, either way, why MTP transfer is so slow, especially via rsync.

At this rate, it seems that it would take several hours to backup my phone data (without previous backup), which is not something I can really do for a total backup. For an incremental backup, that could have been acceptable, but this is not the question here.

For the record, last time I tried to backup my pictures with a tool such as Shotwell (it was a while ago), it did not seem to have taken long (but I was not very paying close attention to it).

What would make a difference here, apart from using different libraries to handle MTP access?
What is the state of MTP support on Linux currently? I heard of some people were using jmtpfs to mount their MTP devices with FUSE, but not always without issues.
Correct me if I am wrong, but I thought gvfs was using libmtp under the hood.

System and machine details

$ uname -svro
Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) GNU/Linux

$ lsusb | grep MTP
Bus 002 Device 018: ID 04e8:6860 Samsung Electronics Co., Ltd Galaxy (MTP)

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 238.5G  0 disk 
├─sda1   8:1    0 107.8G  0 part /
├─sda2   8:2    0  13.5G  0 part [SWAP]
├─sda3   8:3    0   500M  0 part 
└─sda4   8:4    0 116.7G  0 part 
sdb      8:16   0 119.2G  0 disk 
└─sdb1   8:17   0 119.2G  0 part /home
sdc      8:32   0   2.7T  0 disk 
└─sdc1   8:33   0   2.7T  0 part /data

The data from my phone are written on sdc1, the solely partition of an HDD.
As you can see, sda is where the root filesystem and the swap live (along with other partitions which are irrelevant here, sda3 and sda4) while sdb is dedicated to my home directory.

I can confirm that things are no better under jmtpfs, possibly worse. The mount isn't even showing up in df, gparted or dstat. Incremental backups take an age just to go through all of the files that don't need updating. For backing up files from a phone to a computer, I'd recommend adbackup: android.stackexchange.com/questions/28296/… — craq
– craq, Commented May 25, 2019 at 6:45

Gabriel Staples · Accepted Answer · 2025-01-27 19:26:59Z

1

MTP is an ancient protocol and phone manufacturers are lazy and typically implements the reference implementation (ship fast!). MTP is slow because it wasn't designed to be fast to begin with.

Atop of that rsync needs to traverse through all the files to find them in order to compare it to the existing files, and since the driver tries to emulate file listing as a filesystem you also adding more layers before you hit the actual MTP listing, which also takes time. For each file rsync finds, it adds to the list of files to copy and it re-estimates the time left. Next find is found and the list gets longer and estimation rises.

Maybe copy all images to a temporary folder and then rsync between the temporary folder and your storage folder instead?

edited Jan 27 at 19:26

Gabriel Staples

3,0323 gold badges34 silver badges52 bronze badges

answered Mar 24, 2022 at 10:06

Fredrik

1664 bronze badges

This does not explain why the transfer starts fast (no buffering, files list already done, etc.) and keeps slowing down from here.

Paradox
– Paradox

2022-03-24 14:56:30 +00:00
Commented Mar 24, 2022 at 14:56
That's the estimation from rsync, when there are a list of files it will start out fast because estimating one file, then it finds the next etc etc

Fredrik
– Fredrik

2022-03-25 08:22:07 +00:00
Commented Mar 25, 2022 at 8:22
The indexing part would not perpetually decrease the actual transfer, this is simply not how rsync and what its behavior displays here for example.

Paradox
– Paradox

2022-03-25 10:26:54 +00:00
Commented Mar 25, 2022 at 10:26

Add a comment |

Toni Andjelkovic · Accepted Answer · 2023-04-28 00:03:39Z

0

rsync uses a POSIX stream-based input-output model with system calls like open(2), lseek(2), read(2) and write(2), while MTP is an object-based protocol with higher-level operations like "CopyObject". MTP support for partially modifying files is poor, and the corresponding POSIX functions must be emulated inefficiently with MTP protocol means.

For copying lots of files to a MTP drive you are better off using the GNOME file manager, right clicking on a folder and choosing "Copy to". This uses the higher-level MTP file copying operations and is much faster than rsync.

Because of this and other compatibility reasons (filenames, permissions, timestamps etc.) MTP is a poor choice for backing up data from a Linux system.

edited Apr 28, 2023 at 0:03

answered Apr 27, 2023 at 23:10

Toni Andjelkovic

11 bronze badge

As you might know native MTP support was way more "jankier" 4 years ago, and that's a euphemism. ;)

Paradox
– Paradox

2023-04-28 17:08:15 +00:00
Commented Apr 28, 2023 at 17:08

Add a comment |

Stack Exchange Network

Why is rsync so slow when working with MTP devices?

System and machine details

2 Answers 2

You must log in to answer this question.

Hot Network Questions

Why is rsync so slow when working with MTP devices?

System and machine details

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions