efficiently rsync multiple clients (which are behind firewalls) to a server

Question

Like many teams, we now have people working from home. These remote clients are behind firewalls (which we do not control), and they do not have static IP addresses. In short, we cannot directly SSH into these clients. However, the clients can SSH into our server. (Hardened SSH is already set up on all clients and the server for other reasons.)

Our requirement is to keep a set of files (in a few different directories) in sync on each client, and to do it efficiently. I am trying to avoid having each client run an rsync command every NN seconds. It is preferable for the client to be notified when any of the relevant files on the server have been changed.

Furthermore, our implementation can use only SSH, rsync, inotify tools, and either bash or Python (as well as tools like awk, cut, etc.). Specifically, we cannot use NextCloud, OwnCloud, SyncThing, SeaFile, etc.

The only open incoming port on the server is for SSH and the only packages we wish to maintain or update are core packages from our distro's repository.

Our idea is to have each client establish a reverse SSH tunnel to the server. Then the server can run a script like this:

#!/bin/bash
while true; do
    inotifywait -r -e modify,attrib,close_write,move,create,delete /path/to/source/folder
    for port_user in "$(netstat -Wpet | grep "ESTABLISHED" | grep 'localhost.localdomain:' | grep 'sshd:' | cut -d ':' -f2-3 | cut -d ' ' -f1,4)"; do
        uport=$(echo $port_user | cut -d ' ' -f1)
        uu=$(echo $port_user | cut -d ' ' -f2)
        sudo -u $uu rsync -avz -e "ssh -p $uport -i /home/$uu/.ssh/id_ed25519"  /path/to/source/folder $uu@localhost:/path/to/destination/folder
    done
done

I am seeking feedback. First, can the above bash script be improved or cleaned up? It seems like I had to use far too many cut statements, for example.

EDIT: here are the responses to the excellent questions and comments by roaima.

1. The script on the file server is running as root. The script on the client is not.

2. & 7. This is example output of my netstat command

netstat -Wpetl
tcp 0 0 localhost.localdomain:22222 0.0.0.0:* LISTEN  myuser 42137  8381/sshd: myuser

3. "You have a race condition..." - Thank you. We are going to ignore this issue for now.

4. "You have an omission problem..." - Thank you again. I believe this is easily remedied on the client side. Here is the client side script that will launch upon user login:

#!/bin/bash

synchost=sync.example.com
syncpath="path/to/sync/folder"
uu=$(logname)
uport=222222 #hard code per client device
# initial sync upon connecting:
rsync -avzz -e "ssh -i /home/$uu/.ssh/id_ed25519"  /"$syncpath"/ $uu@$synchost:/"$syncpath"
# loop until script is stopped when user logs out
while true; do
    inotifywait -r -e modify,attrib,close_write,move,create,delete /"$syncpath"
    rsync -avzz -e "ssh -i /home/$uu/.ssh/id_ed25519"  /"$syncpath"/ $uu@$synchost:/"$syncpath"
done

There is also an on-demand script the user can run at any time for force a sync. It is the above script without the while loop.

6. Here is the current version of the server script:

syncpath="path/to/sync/folder"
while true; do
    inotifywait -r -e modify,attrib,close_write,move,create,delete /"$syncpath"
    netstat -Wpetl | grep "LISTEN" | grep 'localhost.localdomain:' | grep 'sshd:' | while read proto rq sq local remote state uu inode prog
    do
        uport=${local#*:}
        sudo -u $uu rsync -avzz -e "ssh -p $uport -i /home/$uu/.ssh/id_ed25519"  /"$syncpath"/ $uu@localhost:/"$syncpath"
    done
done

5. "You should consider a timeout for each ssh/rsync to the client so that if they disconnect while you're attempting a transfer you don't end up blocking everyone else".

This is a good suggestion. However, some valid rsync updates may run for a much longer time than average. Can you suggest a proper way to deal with the normal & necessary long rsync updates at times while also handling the rare situation of a client disconnecting during an update?

I have an idea for one approach that may solve the timeout as well as (most of) the race condition in a very, very simple way. First, the initial client side sync upon each user login should take care of long-running update operations. Therefore, server side sync operation times will not have such a long right tail. We can optimize the timeout parameters and sleep time and use an approach like the following:

syncpath="path/to/sync/folder"
while true; do
    inotifywait -r -e modify,attrib,close_write,move,create,delete /"$syncpath"
    netstat -Wpetl | grep "LISTEN" | grep 'localhost.localdomain:' | grep 'sshd:' | while read proto rq sq local remote state uu inode prog
    do
        uport=${local#*:}
        timeout 300s sudo -u $uu rsync -avzz -e "ssh -p $uport -i /home/$uu/.ssh/id_ed25519"  /"$syncpath"/ $uu@localhost:/"$syncpath"
    done

    sleep 90

    netstat -Wpetl | grep "LISTEN" | grep 'localhost.localdomain:' | grep 'sshd:' | while read proto rq sq local remote state uu inode prog
    do
        uport=${local#*:}
        timeout 900s sudo -u $uu rsync -avzz -e "ssh -p $uport -i /home/$uu/.ssh/id_ed25519"  /"$syncpath"/ $uu@localhost:/"$syncpath"
    done
done

One last comment. The parameters shown for the rsync commands are not final. Suggestions are appreciated, but we also intend to spend some time evaluating all the options for the rsync commands.

Answer 1

A number of thoughts

1. Your script is (presumably) running as root, so that netstat -Wpet can run and sudo -u ${user} operation is simplified.

2. Using a reverse connection such as ssh -R 20202:localhost:22 centralserver I cannot get a port and user combination from the netstat | grep | grep | cut ... line.

   netstat -Wpet | grep "ESTABLISHED"  | grep sshd:
   tcp   0   36   centralserver:ssh   client:37226   ESTABLISHED   root   238622975   15198/sshd:   roaima
   

   As a result I can't usefully test possible changes to your script. What are you expecting to see here?

3. You have a race condition, such that if a second file is changed after the inotifywait has completed it may not get propagated to all your target systems until another file has been changed.

   A fix for this might be to listen for events from a single instance of inotifywait and run the set of rsync transfers on each event. However, depending on the frequency of updates this might saturate your clients' network connections

4. You have an omission problem, in that a client connecting after a set of changes will not receive those changes until the next file change. If the updates are this critical you need to consider some way of updating the client copy immediately they have connected

5. You should consider a timeout for each ssh/rsync to the client so that if they disconnect while you're attempting a transfer you don't end up blocking everyone else

6. Given a snippet of bash code such as this, you may be able to replace cut statements with variable manipulation (%, #, and / operators)

    while read -r proto recvq sendq localaddrport foreignaddrport state user inode pidprogram name
    do
        localaddr="${localaddrport%:*}" localport="${localaddrport#*:}"
        foreignaddr="${foreignaddrport%:*}" foreignport="${foreignaddrport#*:}"
        pid="${pidprogram%/*}" program="${pidprogram#*/}"; program="${program%:}"
   
        echo "Foreign address = $foreignaddr and port = $foreignport"
        echo "PID = $pid, program = $program"
        echo "Name = $name"
   
    done < <(netstat -Wpet | grep '\<localhost.localdomain:.*\<ESTABLISHED\>.*/sshd:')
   

7. If we could see expected output of your netstat command it might be possible to use awk to simplify the line processing