爬虫集合
Updated May 30, 2019
#
spider
Repositories 1,736
Elegant Scraper and Crawler Framework for Golang
Go
Updated Jul 25, 2019
Python爬虫代理IP池(proxy pool)
Python
Updated Jul 22, 2019
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
crawler
spider
multi-interface
golang
distributed-crawler
high-concurrency-crawler
fastest-crawler
cross-platform-crawler
Go
Updated Apr 30, 2019
Incredibly fast crawler designed for OSINT.
Python
Updated Jun 3, 2019
越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
HTML
Updated Jan 11, 2019
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Good first issues
#115 opened almost 5 years ago by
JavaScript
Updated Jun 10, 2019
AV电影管理系统， avmoo , javbus , javlibrary 爬虫，线上AV影片图书馆，AV磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - …
javbus
avmoo
javlibrary
spider
crawler
laravel
scraper
adult
magnet-link
magnet
database
adult-video
guzzlehttp
PHP
Updated Jul 19, 2019
Python
Updated Jul 23, 2019
A collection of awesome web crawler,spider in different languages
Updated Apr 18, 2019
Every web site provides APIs.
Python
Updated Dec 6, 2018
一些有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。
crawler
spider
taobao
tmall
example
python
selenium
pyquery
stock
fund
multithreading
agent-pool
wechat
wechat-report
Python
Updated Jun 26, 2019
BitTorrent DHT Protocol && DHT Spider.
Go
Updated Mar 20, 2019
Web crawling framework based on asyncio.
Python
Updated Jun 1, 2019
admin ui for scrapy/open source scrapinghub
Python
Updated Mar 21, 2019
PHP
Updated May 31, 2019
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be ex…
Go
Updated Nov 16, 2017
owllook-在线网络小说阅读网站&小说搜索引擎&小说推荐系统[搜索、追书、收藏、追更、小说API]
Python
Updated Jun 18, 2019
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js
JavaScript
Updated Jun 3, 2019
简单易用的Python爬虫框架，QQ交流群：597510560
Python
Updated Jul 26, 2019
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Email notic…
scrapy
scrapyd
scrapyd-ui
scrapyd-api
scrapyd-admin
scrapyd-manage
log-parsing
log-analysis
scrapyd-monitor
scrapyd-keeper
scrapyd-control
scrapy-log-analysis
scrapyd-log-analysis
scrapy-visualization
scrapyd-visualization
dashboard
spider
scrapyd-cluster-management
Good first issues
Python
Updated Jul 25, 2019
链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁
Python
Updated Jul 24, 2019
PHP
Updated Jun 14, 2019
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
PHP
Updated Aug 30, 2018
Async Python 3.6+ web scraping micro-framework based on asyncio.
Python
Updated Jul 12, 2019
Creating Scrapy scrapers via the Django admin interface
Python
Updated Jul 18, 2019
A configurable web spider with a easy-to-use web console
Java
Updated Aug 21, 2018
Geziyor, a blazing fast web crawling & scraping framework for Go
Go
Updated Jul 21, 2019
Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )
JavaScript
Updated Dec 28, 2018
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Java
Updated Apr 2, 2019