Skip to main content
Unix & Linux

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

8
  • Well I have to download 3 million files with wget and I'm noticing that the CPU/Network/IO is not being taxed at all, this is why I'm trying to increase the number of processes to see where it goes. Commented Apr 10, 2014 at 6:08
  • So you are telling me that it would be faster to run 8 wget processes simultaneously(I have 8 threads) that to run more? Commented Apr 10, 2014 at 6:09
  • 3
    @Dominique maybe the whole thing is waiting for the servers you wget from to respond? If that is the case, your CPUs and the rest of your system are unlikely to get taxed. Commented Apr 10, 2014 at 6:22
  • "The CPU/Network/IO" is not being taxed... -> These are two or three different things. In the case of downloading, you have, in order of the significance of the bottleneck, 1) Network I/O, 2) Disk I/O, presuming you are saving to disk, and 3) CPU time. In other words, in most cases the CPU will have to wait on disk I/O and disk I/O will have to wait on network I/O. So in theory doing this in parallel will probably not make any difference at all. However, waiting on the network involves variables outside the system independent to each download, so doing many of them at the same time: Commented Apr 10, 2014 at 6:26
  • A) Should be faster since the faster downloads won't have to wait for the slower ones, B) Certainly won't incurr what I called the "additional cost" of parallel execution because the CPU will be relatively idle in any case. All it will do is occupy more RAM, but in the case of wget that won't be very significant. To be clear: don't expect this to occupy much CPU time no matter how many downloads you execute simultaneously. It simply is not a processor instensive activity. Commented Apr 10, 2014 at 6:29