I'm redirecting stdout and stderr to the same file by using > and >> respectively:
rsync -a --exclude cache/ src_folder/ target_folder/ 1>out_err.log 2>>out_err.log
However the stderror is not logged in the file. What happens to the stderror when using >> instead of > to redirect both to the same file?
My intention was to write stdout to a new file and then append stderr to that same file.
The trouble with using 1>out_err.log 2>>out_err.log is that it creates two separate open file descriptions for your target file, one for stdout and one for stderr. They both have their own write position and write mode (regular vs. append).
The exact outcome in this case depends on the order in which messages are written, just as if two separate commands were writing to that file. If stderr is used to write to the file, the write goes to the end of the file due to append mode, but the write position of stdout does not get updated. Then, when the file is written using stdout, it will overwrite what was written previously. (Furthermore, "first" might not match what you would see if you ran the same command on the console.)
The conventional and safe approach in this situation is to first redirect stdout to your desired log file, and then use 2>&1 to redirect stderr to the same open file description as stdout. This only opens the file once, creating a duplicate file handle for stderr, so that both stdout and stderr share the write position.
rsync -a --exclude cache/ src_folder/ target_folder/ > out_err.log 2>&1
open(), the one that holds the seek position and the write mode (regular vs. append), while the second is just a pointer to the OFD.
dup() (or dup2() or the shell's n>&m) produces a new file descriptor pointing to the same open file description, while another open() creates a new open file description (along with a new fd pointing to it). The issue here is exactly that doing > foo 2> foo does two open() calls leading into two OFDs and two separate seek positions, while > foo 2>&1 does one open() and one dup(), leading into a single OFD and a single seek position meaning that writes on one fd update the seek position on the other (they're the same) -> no overwriting.
My intention was to write stdout to a new file and then append stderr to that same file.
The rsync utility interleaves its stdout and stderr, so in this situation you need to separate them, saving stderr until you've written everything from stdout. The sponge utility (from moreutils) will let you do this.
rsync… 2>&1 >out_err.log | sponge -a out_err.log
Here we write stdout directly from rsync, piping the stderr into the stdin of sponge. This tool accepts everything from stdin and writes nothing (holding everything in memory) until the pipe is closed, at which point it's all appended to the log file.
(A note on the ordering of 2>&1 >out_err.log. Initially rsync's file descriptor #1 (stdout) is attached to the pipe |. Then we start redirecting file descriptors. First, the 2>&1 duplicates file descriptor #1 for stdout to fd #2 (stderr), also attaching it to the pipe. Next, we redirect fd #1 (stdout) to the output file. Now stderr is (still) attached to the pipe so error output goes through it to sponge.)
