3

I don't understand the following:

$ cat <(sleep 60) & jobs -l
[1] 63813
[1]+ 63813 Running                 cat <(sleep 60) &
/proc/63799/fd$ ps --forest
    PID TTY          TIME CMD
  63055 pts/2    00:00:00 bash
  63813 pts/2    00:00:00  \_ cat
  63814 pts/2    00:00:00  |   \_ bash
  63815 pts/2    00:00:00  |       \_ sleep
  63816 pts/2    00:00:00  \_ ps
/proc/63799/fd$ cd /proc/63813/fd
/proc/63813/fd$ ll
total 0
dr-x------ 2 me   me    0 Aug 17 16:14 ./
dr-xr-xr-x 9 me   me    0 Aug 17 16:14 ../
lrwx------ 1 me   me   64 Aug 17 16:14 0 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:14 1 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:14 2 -> /dev/pts/2
lr-x------ 1 me   me   64 Aug 17 16:14 3 -> 'pipe:[834851]'
lr-x------ 1 me   me   64 Aug 17 16:14 63 -> 'pipe:[834851]'
/proc/63813/fd$ cd /proc/63814/fd
/proc/63814/fd$ ll
total 0
dr-x------ 2 me   me    0 Aug 17 16:14 ./
dr-xr-xr-x 9 me   me    0 Aug 17 16:14 ../
lrwx------ 1 me   me   64 Aug 17 16:14 0 -> /dev/pts/2
l-wx------ 1 me   me   64 Aug 17 16:14 1 -> 'pipe:[834851]'
lrwx------ 1 me   me   64 Aug 17 16:14 2 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:14 255 -> /dev/pts/2
/proc/63814/fd$ cd /proc/63815/fd
/proc/63815/fd$ ll
total 0
dr-x------ 2 me   me    0 Aug 17 16:14 ./
dr-xr-xr-x 9 me   me    0 Aug 17 16:14 ../
lrwx------ 1 me   me   64 Aug 17 16:14 0 -> /dev/pts/2
l-wx------ 1 me   me   64 Aug 17 16:14 1 -> 'pipe:[834851]'
lrwx------ 1 me   me   64 Aug 17 16:14 2 -> /dev/pts/2

So I understand that process substitution <(sleep 60) involves the creation of a pipe ('pipe:[834851]') and that the stdout (file descriptor 1) of sleep 60 is linked to that pipe (1 -> 'pipe:[832422]') in writing mode (l-wx------).

But I do not understand the other file descriptors from the other two processes involved here: the cat child process (63813) and the bash child-child process (63814):

  • Why has the cat process two file descriptors (3 and 63) linked to that same pipe open in reading mode (3 -> 'pipe:[832422]' and 63 -> 'pipe:[832422]')? What does each correspond to?
  • Why has the bash child-child process its stdout linked to that same pipe open in writing mode?

Additional comment: For the two file descriptors of cat, at first I thought there was one for each of the child processes (bash and sleep), but replacing sleep by the read builtin, which doesn't spawn an additional child process, gives the same pattern:

$ cat <(read) & jobs -l
[1] 63905
[1]+ 63905 Running                 cat <(read) &
~$ ps --forest
    PID TTY          TIME CMD
  63055 pts/2    00:00:00 bash
  63905 pts/2    00:00:00  \_ cat
  63906 pts/2    00:00:00  |   \_ bash
  63908 pts/2    00:00:00  \_ ps

[1]+  Stopped                 cat <(read)
~$ ps -o pid,stat,comm
    PID STAT COMMAND
  63055 Ss   bash
  63905 T    cat
  63906 T    bash
  63909 R+   ps
~$ cd /proc/63905/fd
/proc/63905/fd$ ll
total 0
dr-x------ 2 me   me    0 Aug 17 16:31 ./
dr-xr-xr-x 9 me   me    0 Aug 17 16:31 ../
lrwx------ 1 me   me   64 Aug 17 16:32 0 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:32 1 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:32 2 -> /dev/pts/2
lr-x------ 1 me   me   64 Aug 17 16:32 3 -> 'pipe:[836281]'
lr-x------ 1 me   me   64 Aug 17 16:31 63 -> 'pipe:[836281]'
/proc/63905/fd$ cd /proc/63906/fd
/proc/63906/fd$ ll
total 0
dr-x------ 2 me   me    0 Aug 17 16:32 ./
dr-xr-xr-x 9 me   me    0 Aug 17 16:32 ../
lrwx------ 1 me   me   64 Aug 17 16:32 0 -> /dev/pts/2
l-wx------ 1 me   me   64 Aug 17 16:32 1 -> 'pipe:[836281]'
lrwx------ 1 me   me   64 Aug 17 16:32 2 -> /dev/pts/2
lrwx------ 1 me   me   64 Aug 17 16:32 255 -> /dev/pts/2

Edit

After more tests, it seems the pattern of obtaining file descriptor 3 on top of file descriptor 63 is not systematic with cat <(read) & jobs -l. Maybe some race condition... However, it seems systematic with cat <(sleep 60) & jobs -l.

Improve this question
3
  • Where did you get this crazy code from? Assuming you are using bash, <( somecomand ) redirects the output to a file descriptor, so it can be used in place of a file. sleep does not produce any output. Commented Aug 17 at 14:59
  • @GyroGearloose Those commands are my own. They are just to test my understanding of what is happening. I know that sleep doesn't produce any output, I only used it to have some time to have a look at the file descriptors associated to the different processes. I then realized I could use read instead in order to "block" the execution of the command in the background while looking into the /proc/<PID> directories. Commented Aug 17 at 15:11
  • @GyroGearloose, sleep still inherits an stdout file descriptor from its parent, and the fact that it doesn't print anything isn't really relevant to how the plumbing is set up. (Consider that the program inside could be something like find, which may or may not print something.) Commented Aug 17 at 15:16

1 Answer 1

Reset to default
10

So, a process substitution like outer <(inner) makes the output of inner available as a named file that outer can then open. That could be done through named pipes (FIFOs), similarly to something like this (but done by the shell behind the scenes in a better way):

mkfifo tmpfifo     # with a random name
inner > tmpfifo &
outer tmpfifo
rm -f tmpfifo

Another way is for the shell to connect inner to an anonymous pipe (as used in a regular pipeline like left | right), letting outer inherit a file descriptor (fd) to that pipe, and have the process substitution expand to a filename that actually refers to that already-opened fd outer already has.

This requires a system that allows reopening/duplicating open fds through some name. Usually that's /dev/fd/NN (which is linked to /proc/self/fd on Linux). If you try something like echo <(true) in Bash, you get the output /dev/fd/63, showing that Bash used this method and chose to use fd 63 for this. (Other shells might choose other numbers for that fd.)

So, in cat <(...), cat inherits stdin, stdout, stderr (fds 0-2) as normal, but also the pipe from the process substitution at fd 63 and gets passed the filename /dev/fd/63 as a command line argument. It then opens the filename it got, creating a new fd for what it had at fd 63. This new fd gets the number 3, since that's the first available one. The result is that it has copies of the pipe at fds 3 and 63. The second handle doesn't matter, they're both closed when cat exits.

(Using an anonymous pipe instead of a named one is better here, since then everything gets cleaned up automatically when the processes exit, and unrelated processes can't also open the same pipe even by accident.)

Now, the command within the process substitution (inner) is a shell command, so the shell launches a copy of itself to run it. That shell gets its stdout connected to the write end of the pipe, and then runs whatever there is to run, i.e. your sleep command. With just a single command inside the process substitution, the shell could just replace itself with sleep, and then you wouldn't see the intermediate bash process. That's what I get with Bash 5.2 on Ubuntu:

$ cat <(sleep 60) &
$ ps --forest
    PID TTY          TIME CMD
1562963 pts/0    00:00:00 bash
1562972 pts/0    00:00:00  \_ cat
1562973 pts/0    00:00:00  |   \_ sleep
1562975 pts/0    00:00:00  \_ ps

If the command inside is more complex, then the shell needs to keep hanging around until sleep exits, e.g. to run the next command:

$ cat <(sleep 60; true)  & 
$ ps --forest
    PID TTY          TIME CMD
1562963 pts/0    00:00:00 bash
1562983 pts/0    00:00:00  \_ cat
1562984 pts/0    00:00:00  |   \_ bash
1562985 pts/0    00:00:00  |       \_ sleep
1562986 pts/0    00:00:00  \_ ps

That inner shell needs to keep a copy of the stdout file descriptor to be able to pass it to any subsequent commands.

Improve this answer
3
  • Mm... So, looking into the process folder of the "root" bash process (63055) in either case (with sleep or with read), I cannot see any file descriptor open on the pipes. So now I am not sure of my understanding that cat "inherits" its fd 63. Is it rather that this fd 63 is "added" to its list of fd's from the mere presence of the <(...) term in the command line? Or is it that cat indeed inherited its fd 63, but from a temporary bash child process, in which the fd 63 was first added/created, and which later on became replaced altogether by its cat child process? Commented Aug 18 at 10:33
  • 1
    @TheQuark, right, the interactive shell likely forks a copy of itself to set up any redirections, and then exec's the main program (cat), replacing itself. With cat < somefile that setup is just opening the file to fd 0, but with the process substitution it needs to fork another copy of itself to handle the inner command. So the process substitution shell ends up a child of cat (after the exec), and cat gets the fd 63 from the shell program. Commented Aug 18 at 12:51
  • 1
    It's not likely a temporary process as such, since exec replaces the program code (and everything) within the same process, so e.g. your PID 63813 probably was bash to begin with, for a moment. (Would need to check the code or strace the shell to make sure. But I think that's how it goes.) Commented Aug 18 at 12:51

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.