Skip to main content
1 of 5
Stéphane Chazelas
  • 584.7k
  • 96
  • 1.1k
  • 1.7k

The whole point of piping commands is to run them concurrently with one reading the output of the other. If you want to run them concurrently, and if we keep the plumbing metaphor, you'll need to pipe the output of the first command to a bucket (store it) and then empty the bucket into the other command.

But doing it with pipes means having two processes for the first command (the command and another process reading its output from the other end of the pipe to store in the bucket), and two for the second one (one emptying the bucket into one end of the pipe for the command to read it from the other end).

For the bucket, you'll need either memory of the file system. Memory doesn't scale well and you need the pipes. The filesystem makes much more sense. That's what /tmp is for. Note that the disks are likely not to ever see the data anyway as the data may not be flushed there until much later (after you remove the temp file), and even if it is, it will likely still remain in memory (cached). And when it's not, that's when the data would have been too big to fit in memory in the first place.

Note that temporary files are used all the time in shells. In most shells, here documents and here strings are implemented with temp files.

In:

cat << EOF
foo
EOF

Most shells create a tempfile, open it for writing and for reading, delete it, fill it up with foo, and then run cat with its stdin duplicated from the fd open for reading. The file is deleted even before it filled up (that gives the system a clue that it whatever is written there doesn't need to survive a power loss).

You could do the same here with:

tmp=$(mktemp) && {
  rm -f "$tmp" &&
    cmd1 >&3 3>&- 4<&- &&
    cmd2 <&4 4<&- 3>&-
} 3> "$tmp" 4< "$tmp"

If you wanted to store the output in memory, you could do:

(set -o pipefail # ksh/zsh/bash
out=$(cmd1 | base64) &&
  printf '%s\n' "$out" | base64 -d | cmd2)

(or use uuencode -/uudecode if you don't have a base64 command).

Stéphane Chazelas
  • 584.7k
  • 96
  • 1.1k
  • 1.7k