Split output and rejoin again with named pipes on linux

Question

My question is related to https://serverfault.com/questions/171095/how-do-i-join-two-named-pipes-into-single-input-stream-in-linux but with a slightly more convoluted setup.

I have three programs, cmd1, cmd2 and cmd3;

cmd1 takes no input and writes to stdout

cmd2 reads stdin or a given file and writes to stdout

cmd3 reads two files

The dataflow for these programs is the following: cmd2 consumes data produced by cmd1, and cmd3 consumes data produced by both cmd1 and cmd2:

cmd1 ---+-----> cmd2 --->
        |                  cmd3
        +--------------->

How can I achieve this dataflow with a single command line using >(), pipes and tee?

My best guess is cmd1 | tee >(cmd2) > >(cmd3).

possible duplicate of tee + cat: use an output several times and then concatenate results — Stéphane Chazelas
– Stéphane Chazelas, Commented Jun 22, 2018 at 18:05
or Split an input for different command and combine the result — Stéphane Chazelas
– Stéphane Chazelas, Commented Jun 22, 2018 at 18:05

Kusalananda · Accepted Answer · 2018-06-22 21:38:59Z

mkfifo thepipe
cmd3 <( cmd1 | tee thepipe ) <( cmd2 thepipe )

This uses a named pipe, thepipe, to transfer data between tee and cmd2.

Using your diagram:

cmd1 ---(tee)---(thepipe)--- cmd2 --->
          |                            cmd3
          +-------------------------->

Example with

cmd1 = echo 'hello world', writes a string to standard output.
cmd2 = rev, reverses the order of characters on each line, reads a file or from standard input.
cmd3 = paste, takes input from two files (in this case) and produces two columns.

mkfifo thepipe
paste <( echo 'hello world' | tee thepipe ) <( rev thepipe )

Result:

hello world     dlrow olleh

The same thing, but putting the named pipe on the other branch in your diagram:

cmd1 ---(tee)--------------- cmd2 --->
          |                            cmd3
          +-----(thepipe)------------>

cmd3 thepipe <( cmd1 | tee thepipe | cmd2 )

With our example commands:

paste thepipe <( echo 'hello world' | tee thepipe | rev )

This produces the same output as above.

There are obviously other possibilities, such as

cmd1 | tee >( cmd2 >thepipe ) | cmd3 /dev/stdin thepipe

but I don't think you can get away from having to use a named pipe unless you're happy writing intermediate results to a temporary file and breaking it down into two sets of commands.

Stack Exchange Network

Split output and rejoin again with named pipes on linux

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Split output and rejoin again with named pipes on linux

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions