Skip to main content
deleted 29 characters in body
Source Link
Ole Tange
  • 37.5k
  • 34
  • 119
  • 228

Following the question strict:

mycurl() {
    START=$(date +%s)
    curl -s "http://some_url_here/"$1  > $1.txt
    END=$(date +%s)
    DIFF=$(( $END - $START ))
    echo "It took $DIFF seconds"
}
export -f mycurl

seq 100000 | parallel -j0 mycurl

Shorter if you do not need the boilerplate text around the timings:

seq 100000 | parallel -j0 --joblog log curl -s http://some_url_here/{} ">" {}.txt
cat log | parallelcut --colsep '\t' echof {4} log

If you want to run 1000s in parallel you will hit some limits (such as file handles). Raising ulimit -n or /etc/security/limits.conf may help.

Following the question strict:

mycurl() {
    START=$(date +%s)
    curl -s "http://some_url_here/"$1  > $1.txt
    END=$(date +%s)
    DIFF=$(( $END - $START ))
    echo "It took $DIFF seconds"
}
export -f mycurl

seq 100000 | parallel -j0 mycurl

Shorter if you do not need the boilerplate text around the timings:

seq 100000 | parallel -j0 --joblog log curl -s http://some_url_here/{} ">" {}.txt
cat log | parallel --colsep '\t' echo {4}

If you want to run 1000s in parallel you will hit some limits (such as file handles). Raising ulimit -n or /etc/security/limits.conf may help.

Following the question strict:

mycurl() {
    START=$(date +%s)
    curl -s "http://some_url_here/"$1  > $1.txt
    END=$(date +%s)
    DIFF=$(( $END - $START ))
    echo "It took $DIFF seconds"
}
export -f mycurl

seq 100000 | parallel -j0 mycurl

Shorter if you do not need the boilerplate text around the timings:

seq 100000 | parallel -j0 --joblog log curl -s http://some_url_here/{} ">" {}.txt
cut -f 4 log

If you want to run 1000s in parallel you will hit some limits (such as file handles). Raising ulimit -n or /etc/security/limits.conf may help.

Source Link
Ole Tange
  • 37.5k
  • 34
  • 119
  • 228

Following the question strict:

mycurl() {
    START=$(date +%s)
    curl -s "http://some_url_here/"$1  > $1.txt
    END=$(date +%s)
    DIFF=$(( $END - $START ))
    echo "It took $DIFF seconds"
}
export -f mycurl

seq 100000 | parallel -j0 mycurl

Shorter if you do not need the boilerplate text around the timings:

seq 100000 | parallel -j0 --joblog log curl -s http://some_url_here/{} ">" {}.txt
cat log | parallel --colsep '\t' echo {4}

If you want to run 1000s in parallel you will hit some limits (such as file handles). Raising ulimit -n or /etc/security/limits.conf may help.