2

So, in my work we've had to copy a large number of files from my old server to new. There are two data backups I've had to transfer each having ~200000 files cumulatively over 20GBs. There are large number of small files and no large files, so scp I think is the right way to do this transfer.

However, when I remote ssh login into my server2 and scp into it the files from my server1, I see the files being accessed in my terminal. However, I've had to break the wifi connection when I left office, so the terminal now shows a broken pipe. So, I have no way to know if the scp process is still running or how much is remaining. I can change the directory size over time, but I was wondering if there was a better way? Could we get the display of the running process back on my terminal?

I did this for my first data set, I'm yet to start the process for the second data set.. ANything I can do before/while launching the command to prevent/solve such issues of large SCPs?

Also, how many days is this likely to run? 1 server is AWS and other is Google, if that even matters.

Improve this question
1
  • 1
    That sounds like you want to run screen (or tmux) on server2, which let you detach from your shell and re-connect later. Commented Aug 17, 2017 at 5:39

3 Answers 3

Reset to default
3

Use rsync.

rsync -av --progress /from/dir/ server:/to/dir/
Improve this answer
3
  • To expand on the answer - this will allow you to resume if the session breaks since it synchronises the directories rather than doing deep copy Commented Aug 17, 2017 at 10:35
  • @JackLeo But this is an entire different copy procedure right? But the current scp is still working in the background I think.. doing this would simply re-work all of that right? How do I simply check the status? Commented Aug 17, 2017 at 11:21
  • @DeepakM.P. You are right, it is a different way of doing the same. I don't think it would re-do all of the work if you would just stop the scp, but you probably would lose time overall, because if file indexing is done at the start of rsync. I would suggest this only if the current process crashes. I'm pretty sure you can't check status of a running process because it simply yelds files and pushes them one a the time without tracking how much it will need to do. Rsync is capable of that calculation because it tries to synchronise thus knows the total amount of work it will need to do Commented Aug 17, 2017 at 13:02
0

I'm going to answer different parts left unanswered by @hschou.

So, I have no way to know if the scp process is still running or how much is remaining.
Could we get the display of the running process back on my terminal?

to get a list of running processes you can use command ps (man page)

ps aux | grep scp

To monitor it, you could use command watch or just do a simple loop yourself

while true; do clear; ps aux | grep scp; sleep 5; done

This will run ps aux | grep scp every 5 seconds.

Improve this answer
0

You can send a large number of (small) files with scp from a shell script that uses background processes to achieve good performance.

Use semaphore (marker) files, like filename and filename.done. When the receiving job sees filename.done it knows that filename is completely transferred.

The shell script can take a pair of files from input directory, move to a working directory, and exec a background function to transfer the pair.

The background function transfer the pair with one scp command, so the operation is [quasi] atomic, then check the scp return code. On success, move the pair to a backup location and exit. On failure, move the pair back to the original input directory, where it can be picked up for another attempt. Handy way to deal with soft errors.

Configure a limit for the number of concurrent child processes. Start a few, check how many are running (pstree), wait if necessary to keep the number of child pids within the the limit.

Generous use of a logging function is helpful.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.