Skip to main content
Unix & Linux

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

7
  • Tar is fine. Running 20 i/o-intensive programs on the same disk(s) probably isn't. With no tars running, look at the output of iostat 5 while gradually starting additional tar processes until one of the disks' throughput tops out. Commented Sep 13, 2017 at 14:43
  • Will it be faster if you split it into 200 files instead of 20? Commented Sep 13, 2017 at 14:47
  • 1
    You can gradually reduce the workload on the disk head, without having to restart completely, by using ionice on all the tar processes except one, or even kill -stop on most of the tar processes. Try to ensure only one process is doing i/o on one disk. Commented Sep 13, 2017 at 15:42
  • @MarkPlotnick Here's the thing. To create the files I ran some Java code that I wrote which spawned 20 threads, read some input files and then created the output files. Watching iostat/iotop while running that code, I was consistently getting 150MB/s write speed. Running these tar processes, I'm only getting 100MB/s. Running a single process the other day, to create one tar file, I ended the day with less written to the disk than running all 20 at the same time. If I run a single process, I'm only getting around 20MB/s write speed. Commented Sep 13, 2017 at 15:48
  • 2
    Never parse the output of ls! If you don't quote its output, your script breaks when any filename contains whitespace; when you do quote it, you can't process more than one file. You can't win. Just use for i in * instead and make it a habit to quote every variable you use,always: tar -cf "/tar/$i.tar" -T "$i" & Commented Sep 14, 2017 at 7:22