Can I use named pipes to achieve temporal uncoupling?

Question

I have 2 applications that pipe their data:

application1 | application2

Basically, application1 generates a log with events that application2 processes. The issue is that I frequently update application2. Meaning, I need to stop these, update the binaries, and restart it. In that small duration application1 can miss data.

I read about using named pipes using mkfifo and thought that could be an ideal solution. Keep application1 running and have it write to the file backed named pipe so that when application is updated no data is lost and once application2 starts is gets the data.

Testing this with cat to emulate reader/writer is works until there is no longer a reader. This is unexpected.

An alternative would be to use an actual file but that has issues:

• It remains on disk and does not act like FIFO
• It requires some form of rotation to prevent files growing super large
• AFAIK when the reader is at the end (tail?) if will need to probe behind a timer if the file has grown in size which increased the processing latency

I'm in control of the reader, current behavior is already that it will auto restart but I'm not in control of the writer. I can only pipe its output to something else.

1. Can named pipes be configured in a way so that these are durable?
2. I read about "pinning" to pipe by the writer, but I fail to get that to work
3. Can I prevent a pipe from getting closed once the reader exits?
4. Are there alternatives that behave like a pipe?

---

Solution https://unix.stackexchange.com/a/784153/585995 works!

mkfifo /tmp/mypipe
writerapp 1<>/tmp/mypipe

The writer keeps running while I would restart the reader:

readerapp </tmp/mypipe

btw, you can test this yourself by using cat

Answer 1

You can look at named pipes as sort of anchors for regular pipes to be created.

Upon:

application1 > fifo

The shell does a open("fifo", O_WRONLY). That open() blocks until another process (or the same process) does a open("fifo", O_RDONLY)¹:

application2 < fifo

At that point, a pipe (very similar to the unnamed ones created by pipe()) is instantiated.

If application2 terminates, the reading end of that pipe is closed, that becomes a broken pipe, and application1 gets a SIGPIPE the next time it tries to write to it.

To avoid that, you'd want to make sure the pipe is kept alive by having for instance another process that has a fd opened for reading on the pipe, or you can have application1 open the fifo in O_RDWR mode¹ so that it becomes itself both a writer and reader of that pipe even if it's not going to actually read from it.

With:

application1 1<> fifo

Stdout is opened in O_RDWR and the pipe is immediately instantiated. application1 will start writing to the pipe until the pipe buffer is full. If it wanted to, application1 could read from its stdout fd and consume what it has written earlier, but applications normally don't read from their stdout.

Then, you do:

application2 < fifo

application2 will become another reader of that pipe. If it terminates, the pipe will not become broken because there's still a reader to it (application1), and you can start the new version of application2 as: application2 < fifo which will carry on where the previous run left it.

With unnamed pipes, you could also do something like:

./application1 |
  while ./application2; do
    until [ -f new-application2 ]; do
      sleep 1
    done
    mv new-application2 application2 || break
  done

Where you'll supply new versions of your application2 as new-application2² which is automatically renamed and run in a loop.

The loop stops when either application2 or mv fail at which point the pipe will become broken and application1 will get a SIGPIPE the next time it tries to write to it.

---

^{¹ IIRC, that doesn't work with all Unices, but I couldn't tell you which ones it doesn't work on.}

^{² You'll want to make sure new-application2 appears as whole (for instance by renaming it from a previous name/location) to avoid the risk of it being executed whilst not fully created.}

Answer 2

Can named pipes be configured in a way so that these are durable?

Depending on your definition of durable, yes. mkfifo creates the fifo or named pipe, and it remains there until you delete it.

I read about "pinning" to pipe by the writer, but I fail to get that to work

Not sure what you mean by that. However, if you are not in control of the writer, this would not be a solution.

Can I prevent a pipe from getting closed once the reader exits?

You can use tail -f on the fifo, instead of a cat. Or even tail -n +1 -f fifo_name | command

Some additional information: whan cat encounters the end-of-file, it exits. So, if you do

$ mkfifo fifootje
$ cat fifootje

and in another terminal

echo hop > fifootje

you get the hop from the cat, and cat exits. If your writer then does

$ echo plop > fifootje

it hangs until a new cat fifootje is issued.

tail -f ("follow" or "forever") just waits until some lines are added and it does not exit until killed. So,

$ tail -f fifootje
hop
plop
and more text
entered via cat
^C

and in the writer terminal:

$ echo hop > fifootje 
ljm@verlaine[/tmp]$ echo plop > fifootje 
ljm@verlaine[/tmp]$ cat > fifootje
and more text
entered via cat
^D

Alternatively, you could:

while : ; do
    cat fifootje
done

Are there alternatives that behave like a pipe?

Yes, sockets.