ssh agent forwarding - what's going on under the hood?

Question

When I ask Google how ssh agent forwarding works it gives me lots of links to SEO optimized rubbish explaining how to provision ssh-agent. That is NOT what I am asking.

I currently have an issue where jobs started in a screen session on the far side of a VPN connection fail because they can't connect out via ssh after the VPN fails.

Normally these jobs rely on agent forwarding from the origin client to connect. I have suspicions about what is going wrong here, but a better understanding of the whole agent forwarding would help here.

When I connect from host0 to hosta, the ssh-agent on host0 provides my private key to the ssh client on host0. On hosta I see SSH_AUTH_SOCK populated referencing a local socket. If, on hosta I then ssh hostb, the ssh client somehow connects to ssh-agent on host0. presumably that is using an alternate channel in the host0-hosta ssh connection.

what is happenning at $SSH_AUTH_SOCK on hosta?

(fuser $SSH_AUTH_SOCK suggests that nothing has that open)

In the case of my screen session, if the ssh session which started the screen session has ended, and I start a new ssh session from host0 to hosta, will the key requests from the screen session be sent over the new connection?

Answer 1

Well, anything that needs the SSH agent, looks for the environment variable SSH_AUTH_SOCK to figure out the path to a Unix socket used to reach the agent. This is mentioned at least in the man page for ssh-add. (The path might look e.g. something like /tmp/ssh-XXXXi1AqzV/agent.1149174.)

When you start the agent, it prints out a set of commands to set that and other environment variables. Also, when you connect somewhere with ssh -A, the client asks for agent forwarding, and the remote SSH server sets that env var for the session.

However... when you run screen, the processes inside the screen session don't see the env vars set by the session outside of screen. All they'll have is SSH_AUTH_SOCK set by when the screen session was started (if any). Env vars are inherited from the parent when a process starts, and they can't be changed from the outside, so you can't automatically give the new path to the processes inside screen. (You could check the new path and assign it separately to each shell, but that would get weary really quick.)

Any solution would require the env var to be constant over the lifetime of the screen session. Luckily, Unix sockets are just files, so you can make the env var point to a symlink with a known path, and change the link target when opening a new SSH session.

The solution I have is pretty much this:

~/.screenrc:

setenv SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock

~/.ssh/rc:

if [ -S "$SSH_AUTH_SOCK" ]; then
    ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi

There may be other ways. One downside of this is that the link always points to the socket created by the last SSH session opened (with agent forwarding). So if I open two sessions and close the second one, then the processes inside screen can't find the agent.

Answer 2

While @ikkachu pointed me to a solution, I am sharing the details of what I implemented here.

First thing to note is the the ssh-agent running on your local machine is not simply a repository for keys; it handles the asymmetric encryption/decryption used for negotiating the (symmetric) session key. Hence your private key is not exposed anywhere other than your local machine (I mention this as I did not see this in the results Google found for me). It's worth bearing in mind that ssh will renegotiate keys every hour of use.

To allow for recovery, I added this to my bash_profile on the ssh server:

if [ -n "$SSH_AUTH_SOCK" ]; then
   if [ ! -L "$HOME/.ssh/ssh_auth_sock" ]; then
      # No current provision - insert symbolic link in front of socket
      ln -s "$SSH_AUTH_SOCK" "$HOME/.ssh/ssh_auth_sock"
      SSH_AUTH_SOCK="$HOME/.ssh/ssh_auth_sock"
   else
      # test if existing provision is working
      BACKUP_SSH_AUTH_SOCK="$SSH_AUTH_SOCK"
      SSH_AUTH_SOCK="$HOME/.ssh/ssh_auth_sock"
      ssh-add -l >/dev/null 2>&1
      if [ 2 -eq "$?" ]; then
        # reprovision using current connection
        rm "$HOME/.ssh/ssh_auth_sock"
        ln -s "$BACKUP_SSH_AUTH_SOCK" "$HOME/.ssh/ssh_auth_sock"
      fi
   fi
fi

To facilitate maintaining connectivity, I wrote a wrapper around ssh which will automatically reconnect if:

• ssh exits with a non-zero exit code
• after at least 20 seconds of operation

In addition, I set the connectTimeout to 10 seconds and clientAliveInterval to 10 seconds. The other useful bit of functionality here was to incorporate speech synthesis to alert me when things are not going to plan:

function say_out_load () {
   local msg
   msg="$1"
   if [ -f "/etc/wsl.conf" ]; then
       powershell.exe -c "Add-Type -AssemblyName System.speech; \$speak = New-Object System.Speech.Synthesis.SpeechSynthesizer; \$speak.Speak('$msg');"
       # powershell.exe -c '(New-Object Media.SoundPlayer "C:\Windows\Media\Windows Ringout.wav").PlaySync();'
   elif which say >/dev/null ; then
       say "$msg"
   fi
}

Collectively these don't prevent onward travel ssh from the remote host from failing but massively reduce the windows during which this can occur. And I should be aware almost immediately when a task in a minimized/background window is failing.