0

I am trying to to get my first systemd service working as desired. Here is what I have:

gateway-watchdog.service:

# A systemd service for the Gateway Watchdog

[Unit]
Description=gateway-watchdog: Watchdog to ensure that Gateway (gateway.py) is running

[Service]
Type=simple
User=root
WorkingDirectory=/home/ubuntu/lora_gateway/
ExecStart=/home/ubuntu/lora_gateway/watchdog.sh

[Install]
WantedBy=multi-user.target

gateway-watchdog.timer:

[Unit]
Description=gateway-watchdog: Timer to run the associated gateway-watchdog service

[Timer]
# Run service 1 minute after boot
OnBootSec=1min
# Run service 15 minutes after the last run
OnUnitActiveSec=30s

[Install]
WantedBy=timers.target

watchdog.sh has these lines (contained in some if statements):

#!/bin/bash
...
if [[condition]]; then
    command="python3 gateway.py"
        # Process needs to be run in the background with '&'. If it isn't, the systemd service will treat this script
        # as still running and so not re-run it at the time specified by the timer service.
    echo "2 $(date +'%F %T'): Running: $command" >> $LOG_FILE_PATH
    eval "$command"
        # Command must be run using eval. If it isn't, the ampersand at the end of the command is effectively ignored
    exit
fi

What I need:

The service will run watchdog.sh. If the if conditions in the script are true, watchdog.sh will launch python3 gateway.py as a separate process and then exit, and the service will no longer be considered active so that it will run again 30 seconds later.

Through various combinations of different service Types (I've tried simple, oneshot, and forking), using & at the end of ExecStart and/or my command line, and/or using nohup, I am able to either launch and keep gateway.py alive, or have the service run every 30 seconds, but not both.

When python3 gateway.py stays alive, it always stays part of the service's CGroup, so the service stays active (running).

How can I change this behavior to achieve what I'm trying to do? Any thoughts are greatly appreciated.

EDIT: I have updated the file contents above to reflect changes made based on suggestions from U.

With the files as given above, sudo systemctl status gateway-watchdog.service gives:

● gateway-watchdog.service - gateway-watchdog: Watchdog to ensure that Gateway (gateway.py) is running
     Loaded: loaded (/etc/systemd/system/gateway-watchdog.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-01-13 01:45:25 UTC; 8s ago
TriggeredBy: ● gateway-watchdog.timer
   Main PID: 30684 (watchdog.sh)
      Tasks: 3 (limit: 4435)
     CGroup: /system.slice/gateway-watchdog.service
             ├─30684 /bin/bash /home/ubuntu/lora_gateway/watchdog.sh
             └─30696 python3 gateway.py

Jan 13 01:45:25 ubuntu systemd[1]: Started gateway-watchdog: Watchdog to ensure that Gateway (gateway.py) is running.

Good: The gateway.py process stays running. Bad: The service does not run again 30 seconds later, because the previous run is still active.

If I put the & back into command="python3 gateway.py &", sudo systemctl status gateway-watchdog.service gives:

● gateway-watchdog.service - gateway-watchdog: Watchdog to ensure that Gateway (gateway.py) is running
     Loaded: loaded (/etc/systemd/system/gateway-watchdog.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2022-01-13 01:49:05 UTC; 6s ago
TriggeredBy: ● gateway-watchdog.timer
    Process: 33724 ExecStart=/home/ubuntu/lora_gateway/watchdog.sh (code=exited, status=0/SUCCESS)
   Main PID: 33724 (code=exited, status=0/SUCCESS)

Jan 13 01:49:05 ubuntu systemd[1]: Started gateway-watchdog: Watchdog to ensure that Gateway (gateway.py) is running.
Jan 13 01:49:05 ubuntu systemd[1]: gateway-watchdog.service: Succeeded.

Good: The service is inactive, so it does run again 30 seconds later. Bad: The python3 gateway.py process dies immediately.

3
  • 1
    Please avoid "bash script.sh"; instead make the script executable using "shebang". Likewise avoid "command &" commands; let systemd handle it. Assist the community to help you by providing the journal output your commands produce. Commented Jan 12, 2022 at 9:37
  • Thanks for your reply. watchdog.sh does contain #!/bin/bash. I have changed ExecStart=bash watchdog.sh to ExecStart=sudo /home/ubuntu/lora_gateway/watchdog.sh. Despite having WorkingDirectory set, it seems that I have to give the full path here. Commented Jan 13, 2022 at 1:29
  • I have updated the post with systemctl status output. Commented Jan 13, 2022 at 1:51

2 Answers 2

0

I have arrived at a solution by referencing https://stackoverflow.com/a/57041270/2909854. The command line in watchdog.sh is now:

command="systemd-run --scope -E setsid nohup python3 $GATEWAY_SCRIPT_PATH"

I also had to change my service Type set to forking, and add a TimeoutSec=1. Not sure if the timeout is necessary, but it helps at least when debugging.

1
  • Glad you found a solution. As we do not have the programs you are using, it's hard to reproduce the problem. Maybe it would have been useful to set up a primitive service that basically does the same (just some daemon process shell script (watchdog) and the timer setup). If you just wanted to run the service after boot, there are easier ways: Just add the required systemd dependencies; no timer needed. Commented Jan 13, 2022 at 14:59
0

You can use the KillMode option.

Using

[Service]
KillMode=process

in gateway-watchdog.service will only kill the bash script (which doesn't matter since it exited anyway) and leave the python process.

The default is KillMode=control-group, which kills every process in the CGroup.

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.