78

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second.

But how to make bidirectional pipe, i.e. cross-bind stdin and stdout of two programs? Is there easy way to do it in a shell?

Improve this question
0

8 Answers 8

Reset to default
50

Well, its fairly "easy" with named pipes (mkfifo). I put easy in quotes because unless the programs are designed for this, deadlock is likely.

mkfifo fifo0 fifo1
( prog1 > fifo0 < fifo1 ) &
( prog2 > fifo1 < fifo0 ) &
( exec 30<fifo0 31<fifo1 )      # write can't open until there is a reader
                                # and vice versa if we did it the other way

Now, there is normally buffering involved in writing stdout. So, for example, if both programs were:

#!/usr/bin/perl
use 5.010;
say 1;
print while (<>);

you'd expect a infinite loop. But instead, both would deadlock; you would need to add $| = 1 (or equivalent) to turn off output buffering. The deadlock is caused because both programs are waiting for something on stdin, but they're not seeing it because its sitting in the stdout buffer of the other program, and hasn't yet been written to the pipe.

Update: incorporating suggestions from Stéphane Charzelas and Joost:

mkfifo fifo0 fifo1
prog1 > fifo0 < fifo1 &
prog2 < fifo0 > fifo1

does the same, is shorter, and more portable.

Improve this answer
7
  • 36
    One named pipe is sufficient: prog1 < fifo | prog2 > fifo. Commented Nov 2, 2012 at 16:47
  • 4
    @AndreyVihrov that's true, you can substitute an anonymous pipe for one of the named ones. But I like the symmetry :-P Commented Nov 2, 2012 at 17:26
  • 4
    @user14284: On Linux, you can probably do it with something like prog1 < fifo | tee /dev/stderr | prog2 | tee /dev/stderr > fifo. Commented Nov 6, 2012 at 10:28
  • 5
    If you make it prog2 < fifo0 > fifo1, you can avoid your little dance with exec 30< ... (which by the way only works with bash or yash for fds over 10 like that). Commented Apr 5, 2014 at 20:08
  • 1
    @Joost Hmmm, appears you're correct that there is no need, at least in bash. I was probably worried that since the shell performs the redirects (including opening the pipes), it might block—but at least bash forks before opening the fifos. dash seems OK too (but behaves a little differently) Commented Nov 9, 2016 at 17:27
33

If pipes on your system are bidirectional (as they are on Solaris 11 and some BSDs at least, but not Linux):

cmd1 <&1 | cmd2 >&0

Beware of deadlocks though.

Also note that some versions of ksh93 on some systems implement pipes (|) using a socket pair. socket pairs are bidirectional, but ksh93 explicitly shuts down the reverse direction, so the command above wouldn't work with those ksh93s even on systems where pipes (as created by the pipe(2) system call) are bidirectional.

Improve this answer
2
14

I am not sure if this is what you are trying to do:

nc -l -p 8096 -c second &
nc -c first 127.0.0.1 8096 &

This begins by opening a listening socket on port 8096, and once a connection is established, spawns program second with its stdin as the stream output and stdout as the stream input.

Then, a second nc is launched which connects to the listening port and spawns program first with its stdout as the stream input and its stdin as the stream output.

This is not exactly done using a pipe, but it seems to do what you need.

As this uses the network, this can be done on 2 remote computers. This is almost the way a web server (second) and a web browser (first) work.

Improve this answer
2
  • 1
    And nc -U for UNIX domain sockets that only take filesystem address space. Commented May 30, 2017 at 5:51
  • 2
    -c option is not available in Linux. My happiness was a short lived one :-( Commented Mar 30, 2018 at 14:08
14

bash version 4 has coproc command that allows this done in pure bash without named pipes.

From bash man page about coproc command:

coproc [NAME] command [redirections]

This creates a coprocess named NAME. If NAME is not supplied, the default name is COPROC.

The standard output of command is connected via a pipe to a file descriptor in the executing shell, and that file descriptor is assigned to NAME[0]. The standard input of command is connected via a pipe to a file descriptor in the executing shell, and that file descriptor is assigned to NAME[1].

Start with a version easy to understand (but does not work). Start entire pipeline under coproc and use extra cat for plumbing.

coproc {
  cmd1 | cmd2
}
cat <&${COPROC[0]} >&${COPROC[1]}

Buffering is obviously something that needs to be taken care of. Here plain cat will likely break it due to buffering. So use stdbuf to disable buffering by cat:

# this is same as above
coproc {
  cmd1 | cmd2
}
# replace last line from sample above with:
stdbuf -i0 -o0 cat <&${COPROC[0]} >&${COPROC[1]}

Better still, do away with cat. Break up pipeline into two steps, run 1st section under coproc and 2nd section to connect with 1st.

coproc cmd1
cmd2 <&${COPROC[0]} >&${COPROC[1]}

This still assumes that cmd1 and cmd2 handle buffering correctly to make it work.

Some other shells also can do coproc as well.

That is all the answer above.

Below is just some exploration and tests.

Here example chains three commands only to make it a little more interesting.

# start pipeline
coproc {
    cmd1 | cmd2 | cmd3
}

# reconnect STDOUT from `cmd3` to STDIN of `cmd1`
stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}

Get rid of cat and stdbuf and stay with pure bash. Break it up into two parts, launch the first pipeline under coproc, then launch second part (either a single command or a pipeline), reconnecting it to the first:

coproc {
    cmd 1 | cmd2
}
cmd3 <&${COPROC[0]} >&${COPROC[1]}

Proof of concept

File prog - a worker program participating in looped IO. Just a dummy prog to consume, tag and re-print lines. It is using subshells to avoid buffering problems maybe overkill, it's not the point here.

#!/bin/bash

# Start this prog with only param "1" to produce some output

let c=0
sleep 2

[ "$1" == "1" ] && ( echo start )

while : ; do
  read line
  echo "$1:${c} ${line}" 1>&2
  sleep 2
  ( echo "$1:${c} ${line}" )
  let c++
  [ $c -eq 3 ] && exit
done

File start_io_loop_with_cat - a demo launcher. This is a version using bash, cat and stdbuf

#!/bin/bash

# start all 3 commands as one pipeline
coproc {
  ./prog 1 \
  | ./prog 2 \
  | ./prog 3
}

# start cat without buffering to connect IO loop
stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}

File start_io_loop_pure_bash - another demo launcher. This is a version using pure bash only.

#!/bin/bash

# start first 2 of 3 commands in the pipeline
coproc {
  ./prog 1 \
  | ./prog 2
}

# start 3rd command connecting IO to 1 and 2 to make the IO loop
./prog 3 <&${COPROC[0]} >&${COPROC[1]}

Output:

> ./start_io_loop_pure_bash
2:0 start
3:0 2:0 start
1:0 3:0 2:0 start
2:1 1:0 3:0 2:0 start
3:1 2:1 1:0 3:0 2:0 start
1:1 3:1 2:1 1:0 3:0 2:0 start
2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
1:2 3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start

That does it.

Improve this answer
13

You can use pipexec:

$ pipexec -- '[A' cmd1 ] '[B' cmd2 ] '{A:1>B:0}' '{B:1>A:0}'
Improve this answer
0
10

A convenient building block for writing such bidirectional pipes is something that connects the stdout and stdin of the current process together. Let us call it ioloop. After calling this function, you only need to start a regular pipe:

ioloop &&     # stdout -> stdin 
cmd1 | cmd2   # stdin -> cmd1 -> cmd2 -> stdout (-> back to stdin)

If you do not want to modify the descriptors of the top-level shell, run this in a subshell:

( ioloop && cmd1 | cmd2 )

Here is a portable implementation of ioloop using a named pipe:

ioloop() {
    FIFO=$(mktemp -u /tmp/ioloop_$$_XXXXXX ) &&
    trap "rm -f $FIFO" EXIT &&
    mkfifo $FIFO &&
    ( : <$FIFO & ) &&    # avoid deadlock on opening pipe
    exec >$FIFO <$FIFO
}

The named pipe exists in the filesystem only briefly during ioloop setup. This function is not quite POSIX because mktemp is deprecated (and potentially vulnerable to a race attack).

A linux-specific implementation using /proc/ is possible that does not require a named pipe, but I think this one is more than good enough.

Improve this answer
4
  • Interesting function, +1. Could probably use a sentence or 2 added explaining ( : <$FIFO & ) in more detail. Thank you for posting. Commented Nov 9, 2016 at 19:47
  • I did a little looking around and came up blank; where can I find out information about the deprecation of mktemp? I use it extensively, and if a newer tool has taken its place, I would like to start using it. Commented Nov 9, 2016 at 20:16
  • 1
    Alex: the open(2) syscall on a naned pipe is blocking. if you try to "exec <$PIPE >$PIPE" it will get stuck waiting for another process to open the other side. The command ": <$FIFO &" is executed in a subshell in the background and enables the bidirectional redirection to complete successfully. Commented Dec 4, 2016 at 5:52
  • 1
    DopeGhoti: the mktemp(3) C library function is deprecated. The mktemp(1) utility is not. Commented Dec 4, 2016 at 5:55
7

There is also

As @StéphaneChazelas correctly notes in the comments, above examples are the "base form", he has nice examples with options on his answer for a similar question.

Improve this answer
1
  • 1
    Note that by default, socat uses sockets instead of pipes (you can change that with commtype=pipes). You may want to add the nofork option to avoid an extra socat process shoving data between the pipes/sockets. (thanks for the edit on my answer btw) Commented Jul 17, 2016 at 15:27
0

There are many great answers here. So I just want to add something to easy play around with them. I assume stderr is not redirected anywhere. Create two scripts (let say a.sh and b.sh):

#!/bin/bash
echo "foo" # change to 'bar' in second file

for i in {1..10}; do
  read input
  echo ${input}
  echo ${i} ${0} got: ${input} >&2
done

Then when you connect them any good way you should see on the console:

1 ./a.sh got: bar
1 ./b.sh got: foo
2 ./a.sh got: foo
2 ./b.sh got: bar
3 ./a.sh got: bar
3 ./b.sh got: foo
4 ./a.sh got: foo
4 ./b.sh got: bar
...
Improve this answer

You must log in to answer this question.