Including ExecStop commands makes service (consul) fail to start properly

Question

I'm monkeying with consul trying to set up a small CoreOS cluster. If I save the following file to /etc/systemd/system/consul.service, enable the service, and reboot the VM, all 3 vms in the cluster will happily start up properly and join together

[Unit]
Description=consul
After=etcd2.service
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill consul
ExecStartPre=-/usr/bin/docker rm consul
ExecStart=/usr/bin/docker run -d --name consul --network host consul agent -server -bootstrap-expect=3 -data-dir /tmp/consul -bind BIND_IPADDR --node NODE_NAME -retry-join IPADDR1 -retry-join IPADDR2
# ExecStop=/usr/bin/docker container exec consul consul leave
# ExecStopPost=/usr/bin/docker container stop consul
# ExecStopPost=/usr/bin/docker container rm consul

[Install]
WantedBy=multi-user.target

If, however, I remove the comments from the ExecStop commands, the startup will fail, in the sense of no consul containers are running after the vms reboot. systemctl --failed doesn't report any services as having failed though.

What am I doing wrong? Am I misunderstanding ExecStop? Consul?

Answer 1

The problem (or at least one problem) is that you are using docker's -d flag (for "detached") in your ExecStart=.

systemd expects the command it runs to stay running while the service is up, in other words, to run in foreground. At least when setting up services of Type=simple, which is the default type and the one that applies to your unit.

When you're running a command that exits right away (like docker run -d does), systemd assumes your service starts and stays up for only a moment until it's completed. So while your container is still up, the systemd service will think it's not. You can confirm that with a command like systemctl status consul.service. (Check this command often to help you understand what systemd thinks the status of your service is. Post its output here to help us diagnose any further issues you're having too.)

What is causing you trouble when you have some ExecStopPost= commands is that systemd will execute these commands, since it assumes the service is finished, therefore killing your container.

Just removing the -d from your docker run in the ExecStart= command might be enough to solve your problem.