How to restart systemd service if it is taking too long to run?

Question

I have a service that runs a slow and buggy one-shot program once a week that may hang sometimes for no apparent reason. I want this service to be restarted if it took longer than 4 hours to run. I tried to set it up to restart after 4 hours if it is still running with the following unit definition:

[Unit]
Description=Some buggy software

[Service]
WorkingDirectory=/home/buggy
ExecStart=/home/buggy/run
Environment=NODE_VERSION=14
Restart=on-failure
RestartSec=4 h
StartLimitBurst=4
StartLimitInterval=1 s

And a timer:

[Unit]
Description=Some buggy software schedule timer

[Timer]
Unit=buggy.service
OnCalendar=Sat *-*-* 22:00:00

[Install]
WantedBy=timers.target

But it still can hang indefinitely, I came to see my service two days after it run and it was still hung.

What did I do wrong? Is there even a way to do what I need with systemd?

Answer 1

Use the RuntimeMaxSec directive. You cannot use a watchdog because buggy program will need to be modified.

From man systemd.service:

RuntimeMaxSec=

Configures a maximum time for the service to run. If this is used and the service has been active for longer than the specified time it is terminated and put into a failure state. Note that this setting does not have any effect on Type=oneshot services, as they terminate immediately after activation completed. Pass infinity (the default) to configure no runtime limit.

If a service of Type=notify/Type=notify-reload sends EXTEND_TIMEOUT_USEC=..., this may cause the runtime to be extended beyond RuntimeMaxSec=. The first receipt of this message must occur before RuntimeMaxSec= is exceeded, and once the runtime has extended beyond RuntimeMaxSec=, the service manager will allow the service to continue to run, provided the service repeats EXTEND_TIMEOUT_USEC=... within the interval specified until the service shutdown is achieved by STOPPING=1 (or termination). (see sd_notify(3)).