crawling

Scrapy, a fast high-level web crawling & scraping framework for Python.

Elegant Scraper and Crawler Framework for Golang

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

List of libraries, tools and APIs for web scraping and data processing.

Distributed crawler powered by Headless Chrome

Declarative web scraping

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

A Devtools driver for web automation and scraping

Apache Nutch is an extensible and scalable web crawler

Web Scraping Framework

A curated list of awesome puppeteer resources.

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

The complete web scraping toolkit for PHP.

🤖 Scrape data from HTML websites automatically by just providing examples

[Unmaintained] A simple and clean video/music/image downloader 👾

Scrapy middleware to handle javascript pages using selenium

HTTP API for Scrapy spiders

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more