Skip to main content
5 of 16
added 99 characters in body
acgbox
  • 1k
  • 5
  • 15
  • 40

Launching Multiple Queries With Bash Script

I have this script (based on this post) to check the http-status of domains list (000, 418, 404 etc):

#!/bin/bash
advance=$(cat advance.txt 2>/dev/null || echo 1)
while read LINE; do
  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
  echo " $LINE"
  advance=$(($advance+1))
  echo $advance > advance.txt
done < <(tail -n +$advance url-list) >> out

If the script is interrupted for any reason, it starts from the last processed line. The problem is that it is very slow since it verifies line by line. I have already tried other alternatives, such as fping (very limited given the size of the list), pyfunceble (freezes PC), wget (doesn't do what i expect), etc etc. So I would like to improve the script.

Question: How can I launch multiple queries with my script so that I could process many lines at the same time (if it would be possible to set the number of lines to be processed manually, avoiding freeze or blocking the script or PC)?

acgbox
  • 1k
  • 5
  • 15
  • 40