Submitting non-interactive jobs over ssh [duplicate]

Question

I have SSH remote access to a machine I'd like to use for long-running jobs. What I currently do is simply

ssh user@remote command-to-run

This has several drawbacks:

• I can't simply suspend my local machine - when I do that, SIGHUP will be sent to the remote process, effectively killing it. I could use nohup to prevent that.
• The output may be long, I'd rather have it redirected to files. Of course, I can do it manually, but it gets clumsy with a series of commands.
• The process may run a really long time. It would be ideal that the submitting program only confirms that the command (script) has been successfully submitted and terminates.
• I'd like to get a mail notification, when the process terminates, with its exitcode. Of course, I could use a shell script and a terminal command to send it manually, one more hack.
• I want to be able to schedule multiple scripts at once safely. In particular, I want to be able to push multiple scripts with the same name without manual renaming. I don't want to worry about possible files which already exist on the file system.

This is very similar to what SLURM does, but I don't have any administrative rights on the remote side. Besides, when I have the access to all cores of the remote machine, it makes no sense to declare, how many cores I need.

Is there anything I could use for this? What I described seems like a common usecase.

Answer 1

If you can put scripts that run these long-running jobs for you on the remote machine, this becomes very easy:

#!/bin/bash
# This script will run a long-running-job (if it's not already running)
# and email when it completes.
lockfile=/var/run/long-job-1.lock
logfile=$(mktemp)
errfile=$(mktemp)
if [[ -f "$lockfile" ]]; then
    echo "This job is already running." 1>&2
    exit 1
else
    echo $$ > "$lockfile"
    trap 'rm -f "$lockfile" "$logfile" "$errfile"' EXIT
fi

/path/to/some/really/longrunning/job.sh
returncode=$?

if [[ 0 -ne "$returncode" ]]; then
    cat "$errfile" | mailx -s "Job failed with exit code $returncode" -a "$logfile" [email protected]
else
    cat "$logfile" | mailx -s "Job succeeded" [email protected]
fi

Put that script on the remote server in your home directory as longjob1.sh. Then, locally, you can:

ssh username@remotehost "screen -dmS LongJob1 ./longjob1.sh"

The script (and the job it invokes) will run in a screen session on the remote server, and email you when it is done. If it exits in error, you will be emailed the error log, with the standard log attached to the email.