Combining multiple files without tar

Question

I have a directory that contains 400 million files. Using find, I created a list of all the files, which looks like this:

/output/custom/31/7/31767937
/output/custom/31/7/317537a
/output/custom/31/7/317537
/output/custom/31/7/317ab
/output/custom/31/7/317bo
/output/custom/31/7/317je
/output/custom/31/7/317ma
/output/custom/31/7/31763

I then split the file into 20 different files, and ran a script to create 20 different tarballs:

for i in $(ls x*)
do
    tar -cf /tar/$i.tar -T $i &
done

The input files are on a different drive than the /tar mount point. The script has now been running for 2 days, and it's about 1/4 of the way done. I'll probably just leave it running at this point. However, for future reference, I'm wondering if there's a better way to do this than using tar?

My end goal here is to move these tarballs to 20 different servers, untar them and run some scripts on the files. Oh, and since I then have the tarballs I'll be putting them on S3 storage too.

Answer 1

In terms of ease of transfer, I'd suggest rsync.

Advantages

• One command to do it all, one command to find them
• Restartable when the network drops in the middle of the transfer
• Less effort - you don't need to split/tar then transfer/untar
• Source server can concentrate on reading files, leaving target servers to deal with writes (fewer potential filesystem IO conflicts updating metadata, etc.)

Disadvantages

• You discard all the effort you've taken so far
• It's a different solution to the one you have considered
• Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solutions

1. all files in the directory tree

   cd /path/to/top/source/directory
   rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory
   

2. filenames listed in a file

   cd /path/to/top/source/directory
   rsync -avPR -e ssh --files-from=/path/to/filelist . remoteuser@remotehost:/path/to/top/destination/directory
   

   This second solution requires the file names to be listed relative to the top of the source directory, because the filenames as obtained from the filelist file will be applied in the target filesystem too.

If you're going to do either of these as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.