tee /dev/stderr inside script truncates output file if you redirect stderr

Question

Given a script like below if I try to redirect stderr to a file, then the file is truncated at the point tee is used:

$ cat test.sh 
#!/bin/bash

set -eux
echo before
echo '{ "foo": "bar" }' | tee /dev/stderr | jq .foo
echo after
$ ./test.sh 2> log
before
"bar"
after
$ cat log 
{ "foo": "bar" }
+ echo after

The log file should have all of the stderr output. Everything I see if I run the same script without redirecting:

$ ./test.sh
+ echo before
before
+ echo '{ "foo": "bar" }'
+ tee /dev/stderr
+ jq .foo
{ "foo": "bar" }
"bar"
+ echo after
after

So why do I only see the lines that come after tee? Using 2>> doesn't seem to help.

I Don't understand why is this happening. How can I tee the output while still having the ability to redirect the whole script stderr to a file?

Answer 1

This is due to how /dev/stderr, or rather, /proc/$pid/fd/$num works in Linux. There, opening /dev/stderr does not duplicate file descriptor 2, but instead accesses the resource that fd is connected to directly.

So, since tee normally truncates the file it writes to, so it does with tee /dev/stderr. In your case, it's pretty much the same as doing tee log.

See What's wrong with var=$(</dev/stdin) to read stdin into a variable? for further details.

---

That's only an issue on Linux. E.g. on macOS, it works as you'd assume it does. A somewhat compressed example:

linux$ bash -xc 'false; echo "truncated?" | tee /dev/stderr >/dev/null; true' 2>log
linux$ cat log
truncated?
+ true

vs.

mac$ bash -xc 'false; echo "truncated?" | tee /dev/stderr >/dev/null; true' 2>log
mac$ cat log
+ false
+ tee /dev/stderr
+ echo 'truncated?'
truncated?
+ true

Now, to make that work, you'd need to avoid using /dev/stderr, and instead somehow use something like >&2 which tells the shell to duplicate file descriptor 2.

I think you could do that with tee >( cat >&2 ). That's a bit convoluted, but you can't tell tee to use an existing fd, you need to give it a filename, and due to the above issue, it can't be one that refers to the script's stderr directly.

However, it has the issue that the cat runs in the background, which can make the output from it delayed, like here:

linux$ bash -xc 'false; echo "truncated?" | tee >(cat >&2) >/dev/null; true' 2>log
$ cat log
+ false
+ echo 'truncated?'
+ tee /dev/fd/63
++ cat
+ true
truncated?

Note that just tee -a /dev/stderr doesn't help, since while that means anything tee writes goes to the end of the file, anything written through the original stderr does not, but follows the write position of that file description. So you get things like this:

$ bash -xc 'false; echo "truncated?" | tee -a /dev/stderr >/dev/null; true' 2>log
$ cat log
+ false
+ echo 'truncated?'
+ tee -a /dev/stderr
+ true
ed?

where the last + true<nl> (from the script's stderr) is written over the truncated? (from tee). You need to also make the original redirection an appending one, i.e. bash ... 2>>log.

Or just make the original redirection go to a pipe instead of the file. With a pipe, /dev/stderr works more like you'd think, since pipes don't have a write position, and can't be truncated. Just the way to do that is a bit convoluted, you need something like 2> >(cat > log).

Answer 2

As I wrote this I realized I probably can use the tee -a option.

Still I don't quite understand why doing tee /dev/stderr would truncate the redirect file