4

This is the weirdest thing.

In a script, I start an SSH tunnel like so:

ssh -o StrictHostkeyChecking=no -fND 8080 foo@bar

This starts an ssh instance that goes into the background, and script execution continues. Next, I save its PID (for killing it later) by using bash's $! variable. For this to work, I append & to the ssh command even though it already goes into the background by itself (otherwise $! doesn't contain anything). Thus, for example the following script:

#!/bin/bash

ssh -o StrictHostkeyChecking=no -fND 8080 foo@bar &
echo $!
pgrep -f "ssh -o StrictHostkeyChecking=no -fND 8080 foo@bar"

outputs

(some ssh output)
28062
28062

...two times the same PID, as expected. But, now, when I execute this exact sequence of commands from the terminal, the PID output by $! is wrong (in the sense that it is not the PID of the ssh instance). From the terminal:

$ ssh -o StrictHostkeyChecking=no -fND 8080 foo@bar &
[1] 28178
(some ssh output)
$ echo $!
28178
$ pgrep -f "ssh -o StrictHostkeyChecking=no -fND 8080 foo@bar"
28181

It's not always 3 numbers apart, either. I've also observed a difference of 1 or 2. But it is never the same PID, as I would have expected and as is indeed the case when this sequence of commands is run within a script.

  1. Can someone explain why this is happening? I thought it might be due to the initial ssh call actually forking another process, but then why does it work from within a script?

  2. This also made me doubt whether using $! in my script to get the ssh PID as described above will indeed always work (though it has so far). Is this indeed reliable? I felt it was "cleaner" than using pgrep...

Improve this question
6
  • The PID may not be sequential, e.g. on OpenBSD, or if your linux vendor has applied kernel patches to randomize the PID. Commented Sep 18, 2015 at 20:18
  • Does the process with the original PID exist? Do you have some commands in PROMPT_COMMAND or similar? Commented Sep 18, 2015 at 20:21
  • @thrig I'm on Ubuntu, and there's no PID randomization that I'm aware of. Anyhow, this would not explain why it works as expected from within a script? Commented Sep 18, 2015 at 20:23
  • @choroba No, the process with the original PID does not exist anymore. I indeed had $PROMPT_COMMAND set, but only with some bash internal commands, no executables. However, I now unset $PROMPT_COMMAND to be sure and this still happens. Commented Sep 18, 2015 at 20:25
  • 1
    using -f makes the command going to background after authentication itself even without & character. The original pid disapears because the command is forked and daemonized at this point. Commented Sep 18, 2015 at 20:26

1 Answer 1

Reset to default
7

The shell's $! variable only knows the pid of the process started by the shell. As you suspected, the ssh call using -f forks its own process so it can go to background, so the overall process tree looks like [1]:

shell
|
+--ssh<1> (pid is $!)
   |
   +--ssh<2> (pid is different)

ssh<1> exits very shortly after invocation; therefore, the value in $! is unlikely to be useful. It's ssh<2> which is carrying on the remote communication and doing your tunneling for you, and the only way to reliably get its PID is by examining the process table, as you're doing with pgrep [2]. The pgrep method is likely to be the correct one here.

As to why it works in the script but not interactively, this is probably a race condition. Because you put the first ssh in the background, the shell and ssh are executing concurrently, and ssh does some moderately CPU-heavy cryptographic authentication and some network roundtrips. It's likely that the pgrep you run in the script is simply running before ssh<1> forks itself to go to background. To get around this, run pgrep later, either by a sleep call, or just by calling it only when you actually need the PID later on.

[1]: Technically, it might be more complicated than this, if ssh is using a classic double-fork to background. In that case, there would be another, ephemeral ssh process between the two.

[2]: Unless you're systemd and you're using cgroups or something to keep track of all your children. Which you aren't.

Improve this answer
1
  • Perfect! Thank you. Yes, of course, the race condition -- you are absolutely right. I put a sleep in there and observed the same behavior as interactively. So, for my script, pgrep it is... Commented Sep 18, 2015 at 20:48

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.