8

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's

for file in ./folder/*;
do
    ./bin/myProgram $file > ./done/$file
done

(Please Ignore syntax errors, it's just pseudo code).

I now wanted to run this script twice at the same time. Obviously, the execution is unnecessary if ./done/$file exists. So I changed the script to

for file in ./folder/*;
do
    [ -f ./done/$file ] || ./bin/myProgram $file >./done/$file
done

So basically the question is: Is it possible that both scripts (or in general more than one script) actually are at the same point and check for the existance of the done file which fails and the command runs twice?

it would be just perfect, but I highly doubt it. This would be too easy :D If it can happen that they process the same file, is it possible to somehow "synchronize" the scripts?

Improve this question
4
  • If you have a version of xargs with the -P option available, see this question. Commented May 10, 2012 at 8:58
  • 2
    GNU Make supports parallel execution, too; the done/$file markers seem a little like make targets to me. Commented May 10, 2012 at 9:11
  • 2
    The (pseudo-)code you posted doesn't actually run two instances of your program in parallel. If you have either xargs or GNU make or some version of parallel, then there is no need to reinvent this particular wheel. Commented May 10, 2012 at 9:20
  • It will run two instances if the above script is executed twice Commented May 10, 2012 at 9:22

4 Answers 4

Reset to default
4

This is possible and does occur in reality. Use a lock file to avoid this situation. An example, from said page:

if mkdir /var/lock/mylock; then
    echo "Locking succeeded" >&2
else
    echo "Lock failed - exit" >&2
    exit 1
fi

# ... program code ...

rmdir /var/lock/mylock
Improve this answer
4
  • 1
    "As an aside, you almost definitely want to quote $file." It wasn't necessary for my simple job, but obviously your right, that it would be better to do so Commented May 10, 2012 at 9:02
  • @stefan - I removed it once I saw "ignore syntax errors"... :-) Commented May 10, 2012 at 9:04
  • :D it's perfectly fine if you remind me of this stuff. I tend to forget it since I'm not yet used to it Commented May 10, 2012 at 9:08
  • I absolutely LOVE the simplicity of the mkdir-locking. Thanks for the link! (even though I would have preferred it reading about it on this site, maybe you want to extend your answer a bit?) Commented May 10, 2012 at 9:11
2

The two instances of your script can certainly interact in this way, causing the command to run twice. This is called a race condition.

One way to avoid this race condition would be if each instance grabbed its input file by moving it to another directory. Moving a file (inside the same filesystem) is atomic. Moving the input files may not be desirable, and this is already getting a bit complicated.

mkdir staging-$$ making-$$
for input in folder/*; do
  name=${x#folder/}
  staging=staging-$$/$name
  output=making-$$/$name
  destination=done/$name
  if mv -- "$input" "$staging" 2>/dev/null; then
    bin/myProgram "$staging" >"$output"
    mv -- "$output" "$destination"
    mv -- "$staging" "$input"
  fi
done

A simple way to process the files in parallel using a widely-available tool is GNU make, using the -j flag for parallel execution. Here's a makefile for this task (remember to use tabs to indent commands):

all: $(patsubst folder/%,done/%,$(wildcard folder/*))
done/%: folder/%
        ./bin/myProgram $< >[email protected]
        mv [email protected] $@

Run make -j 3 to run 3 instances in parallel.

See also Four tasks in parallel... how do I do that?

Improve this answer
1
  • Thank you very much for this answer. If I could accept two answers, I would! Commented May 11, 2012 at 10:42
1

I have the feeling you are really trying to run multiple jobs in parallel and that the lock file is simply a means to an end.

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel ./bin/myProgram ::: ./folder/*

It will run myProgram on each core in parallel.

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Improve this answer
0

The problem with locking is that you need a method that creates a lock which is uninterruptable (sometimes called atomar). As Chris has wrote in his answer mkdir is such an uninterruptable operation (creating a file is no such operation).

There is also a high-level command - usally hidden in the procmail package: lockfile. That command has some nice features and can easily be used in your own scripts without the need to "reinvent the wheel" (for instance writing your own function that locks based on directory creation).

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.