What's the right way to monitor systemd services status

Question

I'm trying to figure what's the right way for me to be notified when some systemd service is having problems. Manually checking systemctl status isn't a sustainable option; at the very least I want to have a simple tray icon, like just green or red. Some sounds and/or visual notifications can be useful too. Surprisingly I don't seem to find any existing tools for that, so the best I can think of is to write my own, maybe some simple bash script which would poll systemctl once in a while, and update the icon status.

Pretty poor but alas.

Before I actually start spending time on that, want to make sure, am I by any chance missing some better options?

Answer 1

sudo systemctl --type=service

sudo systemctl --state=active  

sudo systemctl --state=failed 

journalctl -b -p err

These should be enough to monitor and to find any misbehaving services.

Answer 2

Use the check_systemd command line tool from the Nagios monitoring plugin collection. You can just call this without parameter or setting up Nagios (which is a hell to do).

On Ubuntu you only need to call:

sudo apt -y install monitoring-plugins-systemd 

and then you can check

/usr/lib/nagios/plugins/check_systemd 

If $? is 0 all is ok, if 1 its bad and 2 if its critical

All other return values should tell you to get the heck out because the world is on fire.

Answer 3

Systemd's internal monitoring for services is really basic and will only report "this service has stopped and returned a non-zero exit code", which is also one of the conditions it can "fix" by restarting the service.

Systemd services are also expected to close down on their own when they are no longer needed, which is not an error, however systemd does not usually have an insight if that is what happened, because once a service is started, systemd is no longer involved as a mediator.

If you want actual service monitoring, you need a monitoring solution like nagios that can also verify that services are responding in appropriate time, give meaningful answers, and follow protocol conventions (e.g. an HTTPS server should present a valid certificate, and deliver a static file with an error code indicating success within a short time).

No graphical monitoring tools exist because there is no group of users for which such alerts would be actionable (i.e. they are well enough acquainted with systemd to actually be able to investigate), and who are at the same time content with the limited monitoring implemented here.