Running a systemd service once a day (on a CoreOS cluster)

Question

I currently have a CoreOS cluster on AWS. The cluster runs multiples containers, mainly for Rails applications. However, one of this container is a pure Ruby app processing a lot of data from specific external APIs.

Actually the Docker container for that application is run everyday at 4am UTC.

myapp.service:

[Unit]
Description=MyApp service
Requires=docker.service

[Service]
ExecStart=/home/core/sc/myapp_start.sh
User=core

myapp.timer:

[Unit]
Description=MyApp Timer
Requires=docker.service

[Timer]
OnCalendar=*-*-* 04:00:00
Persistent=true

The shell script which is executed by the service could be sum up to:

/usr/bin/docker run --rm --name=myapp omg/myapp:$tag

Since the container is deployed to a cluster via CircleCI, the container could be on one of the servers present in the cluster. However if the server on which the container is running run out of memory because of another container running on the same server and taking to much RAM, or if there's no free space on disk available, etc - then the container is stopped and run again on another server of the cluster.

It's kind of problematic in the context of that Ruby application which should be run only once a day and not be run again in case of server's failure.

In that situation, how could I proceed?

Thank you.

Answer 1

You need to teach your app to keep track of its own execution using a method that survives a container restart/move (could be a file on a persistent/shared filesystem, some info in a database, some external coordinating service/application, etc - whatever is convenient for your context).

Upon restart if the app sees it has already executed in the specified interval then it would just exit without doing anything. It might be possible to even teach it to pick up and complete previously unfinished work, if needed :)

If it's not possible to do it inside the app then you could wrap the app in a custom wrapper and do it inside the wrapper.