I wanted to leave this as a comment but stackexchange thinks I suck :'(.

The main php file seems to be doing a sh** ton of work and a few of your functions are as well. I'd recommend refactoring code http://refactoring.com/catalog/. I see a few hard-coded values that could be better store in a configuration file such as JSON or INI for easy use with PHP.

As for why it slows down, you should use curl_multi not just curl when using a crawler, curl_multi allows concurrent requests whereas running curl_exec() waits for the response the more requests you make the slower it will run.

The way I wrote scrapers for the wide6.com site was one process (download) download web documents while another (scraper) scraped the downloaded documents for video information then deleted the document(s). Do note in my example I ran into a few problems filesystems limit the amount of files in one directory which I unsurprisingly hit, my approach to a solution was to create a sub-tree using the first character of an identifier as the directory name. The download process was basically a background process using curl_multi downloading up to 15 documents at a time per instance with a max of 10 instances. The site was on a dedicated server so there was no major impact on it.

The reason I took that approach is because downloads take time and processing/scraping documents using regular expressions or using the DOMDocument and DOMXPath class take time, but truly they are different processes with a simple relationship in which the scraper relies on the download but the download has no reason to wait for me to update the database or process the downloaded document, it's only reason for existence is to download something while the scraper had to wait on a document to be scraped. Keep in mind this was years ago and my approach would be much more elegant now.

I'd like to note I do not think PHP is a great language for something like this, perhaps take a bit time to learn a language with means of multithreading python is great but I hate it due to it's ugly syntax but it's still easy to learn.

Either way you code works and you are on your way to learning to develop beautiful code, my main takeaways would be study object oriented analysis and design and refactoring which the former would make the later less necessary.

Good luck on your journey my friend.

The main PHP file seems to be doing a lot of work and a few of your functions are as well. I'd recommend refactoring code. I see a few hard-coded values that could be better store in a configuration file such as JSON or INI for easy use with PHP.

As for why it slows down, you should use curl_multi, not just curl when using a crawler. curl_multi allows concurrent requests whereas running curl_exec() waits for the response. The more requests you make, the slower it will run.

The way I wrote scrapers for the wide6.com site was one process (download) download web documents while another (scraper) scraped the downloaded documents for video information then deleted the document(s). Do note in my example that I ran into a few problems. Filesystems limit the amount of files in one directory which I unsurprisingly hit. My approach to a solution was to create a sub-tree using the first character of an identifier as the directory name. The download process was basically a background process using curl_multi downloading up to 15 documents at a time per instance with a max of 10 instances. The site was on a dedicated server so there was no major impact on it.

The reason I took that approach is because downloads take time and processing/scraping documents using regular expressions or using the DOMDocument and DOMXPath class take time. But truly they are different processes with a simple relationship in which the scraper relies on the download but the download has no reason to wait for me to update the database or process the downloaded document. Its only reason for existence is to download something while the scraper had to wait on a document to be scraped. Keep in mind this was years ago and my approach would be much more elegant now.

I'd like to note that I do not think PHP is a great language for something like this. Perhaps take a bit time to learn a language with means of multithreading. Python is great, but I hate it due to its ugly syntax, but it's still easy to learn.

Either way, your code works and you are on your way to learning to develop beautiful code. My main takeaways would be to study object oriented analysis and design and refactoring, the former of which would make the later less necessary.