Pinned

1. Crawling-Infrastructure

   Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.

   TypeScript 146 6

2. GoogleScraper

   A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

   HTML 2.2k 659

3. se-scraper

   Javascript scraping module based on puppeteer for many different search engines...

   HTML 383 90

4. struktur

   Module that extracts structured information from a rendered html site and outputs JSON. HTML to JSON.

   JavaScript 20 5

5. scrapeulous

   Cloud crawler functions for scrapeulous

   JavaScript 22 8

6. Dragonfly-SAE

   Implementation of rfc7664 dragonfly key exchange using ECC

   Python 8 4