Duplicate stdout to pipe into another command with named pipe in POSIX shell script function

Question

mkfifo foo
printf %s\\n bar | tee foo &
tr -s '[:lower:]' '[:upper:]' <foo
wait
rm foo

This is a working POSIX shell script of what I want to do:

• printf %s\\n bar is symbolic for an external program producing stdout
• tr -s '[:lower:]' '[:upper:]' is symbolic for another command that is supposed to receive the stdout and do something with it
• tee duplicates stdout to named pipe foo

And the output is as expected:

bar
BAR

Now I'd like to tidy up the code so it becomes external_program | my_function. Something like this:

f() (
  mkfifo foo
  tee foo &
  tr -s '[:lower:]' '[:upper:]' <foo
  wait
  rm foo
)
printf %s\\n bar | f

But now there is no output at all.

Answer 1

The core of the problem is that POSIX has that (generally unhelpful) requirement for sh that asynchronous commands must have their stdin redirected to /dev/null unless there's an explicit stdin redirection (well, technically, that implicit /dev/null redirection happens before explicit redirections if any).

See for instance on Linux-based systems:

$ sh -c 'realpath /dev/stdin & wait'
/dev/null

A common work around for the asynchronous command to have its stdin left alone is to do something like:

{ cmd <&3 3<&- & } 3<&0

Where the original stdin is made available on fd 3 in addition to 0 in a command group with 3<&0, and within the command group cmd's stdin, which was reopened on /dev/null because of the & is being redirected back to the oringal stdin through that fd 3 (which we then close as it's no longer needed).

In:

f() (
  mkfifo foo
  tee foo &
  tr -s '[:lower:]' '[:upper:]' <foo
  rm foo
)
printf %s\\n bar | f

tee's stdin will be /dev/null, not the reading end of the pipe from printf. Changing it to:

f() (
  mkfifo foo
  { tee foo <&3 3<&- & } 3<&0
  tr -s '[:lower:]' '[:upper:]' <foo
  rm foo
  wait
)
printf '%s\n' bar | f

Would address it, but as you found out, so would

f() (
  mkfifo foo
  tr -s '[:lower:]' '[:upper:]' <foo &
  tee foo
  rm foo
)
printf %s\\n bar | f

As then tee is not run asynchronously so doesn't have its stdin redirected to /dev/null and tr's stdin is being redirected explicitly, it doesn't matter that it was redirected to /dev/null beforehand.

We also don't need the wait, as tee is the process running synchronously (behind waited for implicitly by the shell) and would normally not terminated before tr as it's waiting for eof on its stdin (and tr's stdout which is the other end of that pipe is only closed upon exit).

You may still want to wait for tr though to retrieve its exit status:

f() (
  ret=0
  mkfifo foo || exit
  tr -s '[:lower:]' '[:upper:]' <foo &
  tee foo || ret=$?
  rm -f foo
  wait "$!" || ret=$?
  exit "$ret"
)
printf '%s\n' bar | f

foo won't be removed if the subshell that's the body of the function is killed. You could reduce the window during which the fifo exists and make it even more like an unnamed pipe (where the named pipe is just an ephemeral meeting place for those two processes to establish a pipe), by removing it as soon as it's been opened both in read and write mode by the two processes.

f() (
  ret=0
  mkfifo foo || exit
  { tee foo <&3 3<&- & } 3<&0
  {
    # at that point, foo will have been opened here in read-only mode
    # on stdin which can only happen if tee has opened it in write-only 
    # mode already.
    rm -f foo
    tr -s '[:lower:]' '[:upper:]' || ret=$?
  } < foo
  wait "$!" || ret=$?
  exit "$ret"
)
printf '%s\n' bar | f

Answer 2

It looks like the issue is, as suspected, trying to pipe external_program into a detached tee.

In the original script, there was:

printf %s\\n bar | tee foo &

Which doesn't detach tee (only), but the whole pipe sequence in one. So what I originally did was a pipe like this, which gets resolved immediately, i.e. both sleep and noop : get detached together:

$ time sh -c 'sleep 10 | : &'

real    0m0.001s
user    0m0.000s
sys     0m0.001s

However, when detaching tee inside the function, it was only tee that got detached, not the external program at the beginning of the pipe. Re-using the example with sleep, we can see that only noop : gets detached, but sleep does not, and the script takes the full 10 seconds:

$ time sh -c 'sleep 10 | { : & }'

real    0m10.002s
user    0m0.002s
sys     0m0.000s

The solution is to not detach tee, but the other command:

f() (
  mkfifo foo
  tr -s '[:lower:]' '[:upper:]' <foo &
  tee foo
  wait
  rm foo
)
printf %s\\n bar | f

Now tee can receive stdout from the pipe, write to the FIFO, and the output of the function is again:

bar
BAR