Timeline for Can GNU Parallel execute more parallel processes?
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 10, 2014 at 16:32 | comment | added | Dominique | Yes it is still low. | |
| Apr 10, 2014 at 7:00 | comment | added | goldilocks |
That makes sense since again, this means the fast downloads won't have to wait on the slow downloads -- you will be able to make the most of your available bandwidth. What I've said in the answer is really about processor usage (I would guess your CPU % is still low w/ -j 120?).
|
|
| Apr 10, 2014 at 6:56 | comment | added | Dominique |
I have tried using -j 8 instead of 100 but only 8 of the 60 scripts are being executed, and its going slower than before. I'm running 120 scripts now with -j 120, its already faster but lets see how it goes.
|
|
| Apr 10, 2014 at 6:29 | comment | added | goldilocks |
A) Should be faster since the faster downloads won't have to wait for the slower ones, B) Certainly won't incurr what I called the "additional cost" of parallel execution because the CPU will be relatively idle in any case. All it will do is occupy more RAM, but in the case of wget that won't be very significant. To be clear: don't expect this to occupy much CPU time no matter how many downloads you execute simultaneously. It simply is not a processor instensive activity.
|
|
| Apr 10, 2014 at 6:26 | comment | added | goldilocks | "The CPU/Network/IO" is not being taxed... -> These are two or three different things. In the case of downloading, you have, in order of the significance of the bottleneck, 1) Network I/O, 2) Disk I/O, presuming you are saving to disk, and 3) CPU time. In other words, in most cases the CPU will have to wait on disk I/O and disk I/O will have to wait on network I/O. So in theory doing this in parallel will probably not make any difference at all. However, waiting on the network involves variables outside the system independent to each download, so doing many of them at the same time: | |
| Apr 10, 2014 at 6:22 | comment | added | Anthon |
@Dominique maybe the whole thing is waiting for the servers you wget from to respond? If that is the case, your CPUs and the rest of your system are unlikely to get taxed.
|
|
| Apr 10, 2014 at 6:17 | history | edited | goldilocks | CC BY-SA 3.0 |
added 718 characters in body
|
| Apr 10, 2014 at 6:09 | comment | added | Dominique | So you are telling me that it would be faster to run 8 wget processes simultaneously(I have 8 threads) that to run more? | |
| Apr 10, 2014 at 6:08 | comment | added | Dominique | Well I have to download 3 million files with wget and I'm noticing that the CPU/Network/IO is not being taxed at all, this is why I'm trying to increase the number of processes to see where it goes. | |
| Apr 10, 2014 at 5:56 | history | answered | goldilocks | CC BY-SA 3.0 |