Skip to main content
added 416 characters in body
Source Link
Sir Muffington
  • 1.3k
  • 3
  • 11
  • 24

I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.

Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync.

I did the following before this:

  1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
  2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files.
  3. I checked my kernel log and found nothing (sudo dmesg)

I found a reconnect flag for my mount, but it would not be instant.

  • There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?

How could I best fix this problem at hand?

UPDATE 1

Correct me if I'm wrong but I think the issue at hand was that the storage had the owners and groups of different users and groups than in rsync. To elaborate: cp -r copied the files and changed their ownership and group ownership to the user copying, whereas rsync seems to copy the file 1:1 with the same user and group ownership... That's probably why the transfer was overwriting old files...

I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.

Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync.

I did the following before this:

  1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
  2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files.
  3. I checked my kernel log and found nothing (sudo dmesg)

I found a reconnect flag for my mount, but it would not be instant.

  • There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?

How could I best fix this problem at hand?

I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.

Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync.

I did the following before this:

  1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
  2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files.
  3. I checked my kernel log and found nothing (sudo dmesg)

I found a reconnect flag for my mount, but it would not be instant.

  • There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?

How could I best fix this problem at hand?

UPDATE 1

Correct me if I'm wrong but I think the issue at hand was that the storage had the owners and groups of different users and groups than in rsync. To elaborate: cp -r copied the files and changed their ownership and group ownership to the user copying, whereas rsync seems to copy the file 1:1 with the same user and group ownership... That's probably why the transfer was overwriting old files...

Source Link
Sir Muffington
  • 1.3k
  • 3
  • 11
  • 24

How do you synchronize files with partially failed download using rsync?

I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.

Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync.

I did the following before this:

  1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
  2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files.
  3. I checked my kernel log and found nothing (sudo dmesg)

I found a reconnect flag for my mount, but it would not be instant.

  • There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?

How could I best fix this problem at hand?