Use files from find command in parallel batches

Question

I have this code to search for dirs in current folder and then tar it

find . -type d -maxdepth 1 -mindepth 1 -print -exec tar czf {}.tar.gz {} \;

I have 100s of folders and currently it does one by one folder.

Is it possible that I can run in batch of 5 in parallel

Answer 1

Using xargs you may achieve a degree of parallelism:

find . -type d -maxdepth 1 -mindepth 1 -print0 |
xargs -t0 -n 1 -P 5 -I XX tar czf XX.tar.gz XX

It's the -P 5 flag that limits the number of parallel processes started by xargs to five, while -n 1 is used to only execute the utility with one input line at a time. With -I XX we say that the input line, which will be a directory path, should be substituted in for the string XX in the command.

xargs will print out the invokations of tar as these are performed. Remove -t to turn this off.

The directory pathnames are passed from find with -print0 and received by xargs with -0 (i.e. using \0 as delimiter) to allow for possibly exotic names.

---

As pointed out in an answer elsewhere on this site:

Note that the bottleneck of the operation would likely be the hard drive. For that reason, even if you did split the task in two or more processes, it would not go faster unless they operate on different drives.