Scrapy, a fast high-level web crawling & scraping framework for Python.
crawler
Repositories 2,213
A Powerful Spider(Web Crawler) System in Python.
Python
Updated May 15, 2018
A scalable web crawler framework for Java.
Java
Updated May 6, 2018
News, full-text, and article metadata extraction in Python 3. Advanced docs:
Python
Updated May 21, 2018
Elegant Scraper and Crawler Framework for Golang
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
crawler
spider
multi-interface
golang
distributed-crawler
high-concurrency-crawler
fastest-crawler
cross-platform-crawler
Go
Updated May 23, 2018
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
JavaScript
Updated May 25, 2018
Python爬虫代理IP池(proxy pool)
Redis-based components for Scrapy.
Python
Updated May 5, 2018
Distributed crawler powered by Headless Chrome
JavaScript
Updated May 24, 2018
A collection of awesome web crawler,spider in different languages
Updated Apr 4, 2018
Every web site provides APIs.
Python
Updated May 2, 2018
基于搜狗微信搜索的微信公众号爬虫接口
Python
Updated May 5, 2018
Web Application Security Scanner Framework
Python
Updated Apr 8, 2018
Polite, slim and concurrent web crawler.
Go
Updated Apr 29, 2018
Web crawling framework based on asyncio.
Python
Updated Mar 19, 2018
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be ex…
Go
Updated Nov 16, 2017
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
Java
Updated Dec 4, 2017
The DomCrawler component eases DOM navigation for HTML and XML documents.
PHP
Updated May 26, 2018
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Ruby
Updated Nov 24, 2017
Crawl a website and run it through Google lighthouse
JavaScript
Updated Feb 22, 2018
DotnetSpider, a .NET Standard web crawling library similar to WebMagic and Scrapy. It is a lightweight ,efficient and…
C#
Updated May 24, 2018
简单易用的Python爬虫框架，QQ交流群：597510560
Python
Updated May 10, 2018
Python脚本。模拟登录知乎， 爬虫，操作excel
Python
Updated Mar 21, 2018
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
proxy
anonymity
privacy
socks
http-proxy
crawler
proxy-server
anonymous
proxy-checker
proxy-list
proxypool
proxies
Python
Updated May 19, 2018
Python爬虫，京东自动登录，在线抢购商品
Python
Updated Jan 18, 2018