Uploading files to remote server and batch rename

Question

I have a large set of files and need to upload them to a remote machine with specific destinations. I have a map (can be re-organised) as

1/1/1/file.jpg -> 2/3/4/image1.jpg
2/12/2/file.jpg -> 5/6/7/image2.jpg
3/31/31/file.jpg -> 8/9/1/image3.jpg

I can think of two solutions:

Uploading to specific destinations

scp /folder/1/1/1/file.jpg user@ip:/dir/2/3/4/image1.jpg
scp /folder/2/12/2/file.jpg user@ip:/dir/5/6/7/image2.jpg
scp /folder/3/31/31/file.jpg user@ip:/dir/8/9/1/image2.jpg

but it is not efficient to have thousands of scp connections.

Question: How can I upload multiple files in one connection (for the sake of speed) with a method like scp?

Uploading with rsync and then batch rename

I can upload the entire folder with rsync, which is quite fast, to a temporary folder. Then, rename the files according to the above map.

Question: How can I batch rename with a given map?

Answer 1

A third solution requires that all of the source files are on the same filesystem, and thus hard links can be made arbitrarily to any of the files.

Store your file map in file filemap.txt:

$ cat << EOF > filemap.txt
1/1/1/file.jpg -> 2/3/4/image1.jpg
2/12/2/file.jpg -> 5/6/7/image2.jpg
3/31/31/file.jpg -> 8/9/1/image3.jpg
EOF

Then use this script to create a temporary output directory named temp which will contain sub-directories matching the destination paths of your file map. Each source filename will in turn be hard-linked into its proper destination path under directory temp. Once temp is fully populated, it will be rsynced to user:ip and then deleted from the local machine.

#!/usr/bin/env bash

srcD="/folder/"
destD="/dir/"

rm   -rf temp

while read  src X dest
do

        dir="$(dirname "$dest")"

        mkdir -p "temp/$dir"
        ln "$srcD$src" "temp/$dest"

done < filemap.txt

rsync -av temp/ user@ip:"$destD"

rm -rf temp