Warn in crontab if command has failed multiple consecutive times

Question

I have a command in my crontab to monitor a service (specifically, check whether the Tor version of my website can still be accessed): this monitoring command succeeds if it can access the site, and fails otherwise (and I get an email). However, because of intermittent failures of Tor, I get emails every now and then, ever for fairly short downtimes.

I would like to be notified if this monitoring command in my crontab has been failing for multiple consecutive times (say, 10 times), so I would only be notified for longer outages.

Of course, I could write a custom script to do this, storing the number of failures in a temporary file, etc., but as this looks like a pretty common need, I thought some standard solution for this may already exist (in the same way that moreutils' chronic already exists to serve a similar but different purpose.)

Is there a wrapper script such that issuing wrapper COMMAND will run COMMAND and succeed unless the last 10 invocations of COMMAND have failed, in which case it should return the last error code and the output of the failed invocations?

Answer 1

The following script can be used as the wrapper that you describe. It saves the standard output and standard error streams of the given command to a state directory ($HOME/states) and also stores the number of failed runs.

If the number of failed runs of the command exceeds 10 (or whatever number is given to the -t command line flag), it will provide some output (on its standard error stream). In all other cases, no output will be provided. The script exits with the same exit status as the given command.

Example use:

$ sh ./script.sh -t 2 sh -c 'echo "this will fail"; cd /nowhere'
$ sh ./script.sh -t 2 sh -c 'echo "this will fail"; cd /nowhere'
FAILED 2 times: sh -c echo "this will fail"; cd /nowhere
f88eff95bba49f6dd35a2e5ba744718d
stdout --------------------
this will fail
stderr --------------------
sh: cd: /nowhere - No such file or directory
END

The script itself (relies on md5sum from GNU coreutils):

#!/bin/sh

statedir="$HOME/states"

if ! mkdir -p "$statedir"; then
        printf 'Failed creating "%s"\n' "$statedir" >&2
        exit 1
fi

max_tries=10

while getopts 't:' opt; do
        case "$opt" in
                t) max_tries=$OPTARG ;;
                *) echo 'error' >&2
                   exit 1
        esac
done

shift "$(( OPTIND - 1 ))"

hash=$( printf '%s\n' "$@" | md5sum | cut -d ' ' -f 1 )

"$@" >"$statedir/$hash".out 2>"$statedir/$hash".err
code=$?

if [ -f "$statedir/$hash" ]; then
        read tries <"$statedir/$hash"
else
        tries=0
fi

if [ "$code" -eq 0 ]; then
        echo 0 >"$statedir/$hash"
        exit 0
fi

tries=$(( tries + 1 ))
printf '%d\n' "$tries" >"$statedir/$hash"

if [ "$tries" -ge "$max_tries" ]; then
        cat >&2 <<END_MESSAGE
FAILED $tries times: $@
stdout --------------------
$(cat "$statedir/$hash".out)
stderr --------------------
$(cat "$statedir/$hash".err)
END
END_MESSAGE
fi

exit "$code"