We develop and distribute an application that controls industrial robots (non real-time typically). We run it on Kubuntu 22.04 in a Konsole window. It is runs continuously for several months at a time.
Since we need to compare several alternative possible movements of the robot, CPU and memory usage is significant. The application intermittently will use available memory, but we typically see OOM-killer messages in logs. Core files are also enabled and written if the software crashes due to bugs.
On one project we are seeing sudden disappearances of our software. Even the containing terminal emulator (KDE Konsole) disappears. There are no traces of OOM-killer and while there is a core file written it appears that it is from a bug due to unexpected shutdown rather than the root cause.
The chain of software is:
Konsole > Bash > roslaunch (Python process manager) > our software (several C++ binaries).
roslaunch logs "Keyboard Interrupt". When our software crashes or gets killed we see a log message in roslaunch. Here everything is gone.
Disappearance happens after two hours, 2-5, on our test system in the office. RAM usage is about 20 GB out of 32 GB, so nothing to worry about.
I'm really at loss what is going on. We ran the test again in gnome-terminal and did not observe a "disappearance" within 20 hours. So I opened a bug at bugs.kde.org but I still wonder if anything else could go amiss?