Skip to main content
added example for transferring listed files
Source Link
Chris Davies
  • 128k
  • 16
  • 178
  • 323

In terms of ease of transfer, I'd suggest rsync.

Advantages

  • One command to do it all, one command to find them
  • Restartable when the network drops in the middle of the transfer
  • Less effort - you don't need to split/tar then transfer/untar
  • Source server can concentrate on reading files, leaving target servers to deal with writes (fewer potential filesystem IO conflicts updating metadata, etc.)

Disadvantages

  • You discard all the effort you've taken so far
  • It's a different solution to the one you have considered
  • Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solutionsolutions

cd /path/to/top/source/directory
rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory
  1. all files in the directory tree

     cd /path/to/top/source/directory
     rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory
    
  2. filenames listed in a file

     cd /path/to/top/source/directory
     rsync -avPR -e ssh --files-from=/path/to/filelist . remoteuser@remotehost:/path/to/top/destination/directory
    

    This second solution requires the file names to be listed relative to the top of the source directory, because the filenames as obtained from the filelist file will be applied in the target filesystem too.

If you're going to do thiseither of these as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.

In terms of ease of transfer, I'd suggest rsync.

Advantages

  • One command to do it all, one command to find them
  • Restartable when the network drops in the middle of the transfer
  • Less effort - you don't need to split/tar then transfer/untar

Disadvantages

  • You discard all the effort you've taken so far
  • It's a different solution to the one you have considered
  • Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solution

cd /path/to/top/source/directory
rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory

If you're going to do this as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.

In terms of ease of transfer, I'd suggest rsync.

Advantages

  • One command to do it all, one command to find them
  • Restartable when the network drops in the middle of the transfer
  • Less effort - you don't need to split/tar then transfer/untar
  • Source server can concentrate on reading files, leaving target servers to deal with writes (fewer potential filesystem IO conflicts updating metadata, etc.)

Disadvantages

  • You discard all the effort you've taken so far
  • It's a different solution to the one you have considered
  • Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solutions

  1. all files in the directory tree

     cd /path/to/top/source/directory
     rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory
    
  2. filenames listed in a file

     cd /path/to/top/source/directory
     rsync -avPR -e ssh --files-from=/path/to/filelist . remoteuser@remotehost:/path/to/top/destination/directory
    

    This second solution requires the file names to be listed relative to the top of the source directory, because the filenames as obtained from the filelist file will be applied in the target filesystem too.

If you're going to do either of these as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.

Source Link
Chris Davies
  • 128k
  • 16
  • 178
  • 323

In terms of ease of transfer, I'd suggest rsync.

Advantages

  • One command to do it all, one command to find them
  • Restartable when the network drops in the middle of the transfer
  • Less effort - you don't need to split/tar then transfer/untar

Disadvantages

  • You discard all the effort you've taken so far
  • It's a different solution to the one you have considered
  • Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solution

cd /path/to/top/source/directory
rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory

If you're going to do this as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.