Periodically transfer files from two remote servers to single host using shell script

Question

• I have two remote data acquisition machines that are continuously writing data to their own local data.txt files.
• The remote machines begin acquiring data and appending their data.txt with new data upon powering up/reboot via a systemd service.
• The host machine enables/starts the service on each remote machine.

I would like to periodically and automatically transfer their respective data files to my host machine using a shell script.

Note: I currently only have access to one remote machine, and will have access to the second in the future. I'm trying to prepare for when this machine comes online. This is why I can't test out my "thoughts" on how to accomplish this.

Here is what I have cooked up to accomplish this for a single remote machine:

#!/usr/bin/env bash

#Enable data acquisition service on remote machine
REMOTE1="user@remote1"
REMOTE2="user@remote2"

ssh $REMOTE1 "systemctl start my.service"
while [ $? -ne 0 ]; do !!; done # waits for service to start, returns 0

ssh $REMOTE2 "systemctl start my.service"
while [ $? -ne 0 ]; do !!; done

# transfer data file every 60 seconds, until SIGINT from host
while sleep 60; do scp $REMOTE1:/data1.txt /host/; done

exit 0

I have run this successfully using one remote machine; I have not noticed any missing data/corruption either. To get this working with two remote machines, my first thought is to add another instance of the while loop:

while sleep 60; do scp $REMOTE1:/data1.txt /host/; done
while sleep 60; do scp $REMOTE2:/data2.txt /host/; done

However, I can see this may not work... given that the first loop will never break to begin the second one (?). My second thought would be to use a single while loop, but adding the another scp command for the second machine:

while sleep 60; do scp $REMOTE1:/data1.txt /host/; scp $REMOTE2:/data2.txt /host/; done

I am aware that this method could end up "drifting" if the command execution time varies. Though I am not worried about the transfer occurring exactly every X amount of time, it just needs to happen (sudo) periodically. Would love to hear comments/suggestions on how to accomplish this.

Answer 1

Use an array to contain the pair of hosts. Initially the array can contain just a single element, and you can add the second host later. Here I've included two sets of credentials so you can see the syntax. For the copying I've chosen to use rsync but you could revert to scp if you preferred.

#!/bin/bash

# Remote credentials
remotes=('user@remote1' 'user@remote2')

# Enable data acquisition service
for remote in "${remotes[@]}"
do
    ssh -n "$remote" 'systemctl start my.service'
done

# Transfer data file with an interval of 60 seconds, until SIGINT
while sleep 60
do
    for remote in "${remotes[@]}"
    do
        rsync -tz --contimeout=10 "$remote:/data1.txt" /host/
        # scp -pC "$remote:/data1.txt" /host/
    done
done

# Forever

The loop delays 60 seconds between executions, so if the copies take 15 seconds each you'll actually loop every 90 seconds. If you want the loop to aim for a minute cycle regardless of the copying duration you can use bash's built-in clock:

interval=60 start=$SECONDS
while sleep $(( interval - ((SECONDS - start) % interval) ))
do
    …
done