Using `sem` to make a script run in parallel

Question

I have the following shell script (one liner), which I wanted to use to identify directories that have the exact same contents. I'm using it, to identify and remove duplicate (child) directories.

When I try to run the same script with sem, I encounter No such file or directory errors.

Example - no parallel threads

find -type d -links 2 | while read i; do \
   find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \
   | sort | md5sum | xargs -I {} echo {} $i ; \
   done

Gives me:

e94d32e2a683d46d49c7580d649f7888  - ./Daft Punk/Alive 2007 2
e94d32e2a683d46d49c7580d649f7888  - ./Daft Punk/Alive 2007

---

Example - using sem

find -type d -links 2 | while read i; do sem -j+0 \
   find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \
   | sort | md5sum | xargs -I {} echo {} $i ; \
   done; sem --wait

Gives me:

find: `./Daft': No such file or directory
find: `Punk/Alive': No such file or directory
find: `2007': No such file or directory
find: `2': No such file or directory
d41d8cd98f00b204e9800998ecf8427e  - ./Daft Punk/Alive 2007 2
find: `./Daft': No such file or directory
find: `Punk/Alive': No such file or directory
find: `2007': No such file or directory
d41d8cd98f00b204e9800998ecf8427e  - ./Daft Punk/Alive 2007

Questions:

1. Why the difference in behaviour?
2. How do I remove/fix the No such file or directory from the sem script?
3. Are there any other improvements I could make in the script? (there is lots of awk and xargs)

Answer 1

The simplest solution is to quote the command passed to sem:

sem -j+0 "find \"$i\" -type f -print0"

You can see the difference in this example

$ sem -j 8 echo "a  a" 
a a
$ sem -j 8 'echo "a  a"' 
a  a

Answer 2

Use parallel instead:

doit() {
  i="$1"
  find "$i" -type f -print0 | xargs -r0 cat | md5sum |
    awk '{ print $1 }' |
    sort | md5sum
}
export -f doit
find -type d -links 2 | parallel --tag doit