bash pipe loses data when command crashed

Question

Consider this simple program (abort.py) that writes some text to stdout and stderr and then crashed (abort()).

import os
import sys

print("end stdout")
print("end stderr", file=sys.stderr)
os.abort()

When I run it in terminal, both output to stdout and stderr are correctly produced.

$ python3 abort.py 
end stdout
end stderr
Aborted (core dumped)

However, if I were to redirect stdout and/or stderr to another program (perhaps for logging purpose), it doesn't work any more.

$ python3 abort.py | cat
end stderr

$ python3 abort.py |& cat

$ python3 abort.py | tee -a logs
end stderr

$ python3 abort.py |& tee -a logs

# file `logs` is unchanged

In fact, if the program (abort.py) produce a lot of texts to stdout and stderr, only the last section of them is lost from the perspective of the receiving program on the other end of the pipe. I also tried running the program in a bash script and run that script, but the result is the same.

Why is this happening and how should I fix it?

Background

The example above is obviously contrived. The real world scenario is that I was debugging a large program that occasionally crash (segfault or aborted) after running for days. It produces a lot of logs to both stdout and stderr, but the most important piece of information regarding its crash (such as stack trace) is printed in the end right before the crash happens. I have a logging pipeline setup in a bash script (involves some tee and gzip for example), but I found that last section of the log saved is always missing, which is frustrating.

Answer 1

This is a buffering "problem". Telling python not to buffer results in the desired result.

$ python3 -u abort.py |& cat
end stdout
end stderr

As to what to do about it, you should flush the output every time something is written that you care about enough to take the performance hit, e.g.

import os
import sys

print("end stdout", flush=True)
print("end stderr", file=sys.stderr, flush=True)
os.abort()

Note that the Aborted or Aborted (core dumped) messages come from the shell. If you are running bash or any other POSIX shell then the exit status of a pipeline is that of the end of the pipeline. As the cat is not aborting or dumping core you don't see a message. For bash, use set -o pipefail to see the aborted message:

$ set -o pipefail
$ python3 -u abort.py |& cat
end stdout
end stderr
Aborted (core dumped)