The Wayback Machine - https://web.archive.org/web/20230721054300/https://github.com/topics/crawler

#

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 6,408 public repositories matching this topic...

scrapy

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

Updated Jul 20, 2023
Python

iawia002 / lux

👾 Fast and simple video download library and CLI tool written in Go

go golang crawler scraper downloader youtube video download tumblr bilibili qq youku iqiyi

Updated Jul 6, 2023
Go

gocolly / colly

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

Updated Jul 20, 2023
Go

jhao104 / proxy_pool

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Jun 20, 2023
Python

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.

Updated Jul 5, 2023
Python

NaiboWang / EasySpider

Sponsor

A visual no-code/code-free web crawler/spider易采集：一个可视化爬虫软件，可以无代码图形化的设计和执行爬虫任务

Updated Jul 21, 2023
JavaScript

codelucas / newspaper

News, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

Updated Apr 25, 2023
Python

shengqiangzhang / examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report wereader

Updated Apr 27, 2022
Python

code4craft / webmagic

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Jul 13, 2023
Java

crawlab

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management

Updated Jul 8, 2023
Go

s0md3v / Photon

Sponsor

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Dec 26, 2022
Python

guyueyingmu / avbook

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Mar 3, 2023
PHP

crawlee

apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

Updated Jul 21, 2023
TypeScript

injetlee / Python

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

python crawler excel wechat

Updated May 6, 2023
Python

ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow

Updated Jun 14, 2023
Java

projectdiscovery / katana

A next-generation crawling and spidering framework.

cli crawler headless web-spider spider-framework gocrawler

Updated Jul 17, 2023
Go

bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Feb 27, 2023
JavaScript

lorien / awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

Updated Jul 9, 2023
Makefile

BruceDone / awesome-crawler

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Jun 14, 2023

chyroc / WechatSogou

Sponsor

基于搜狗微信搜索的微信公众号爬虫接口

python crawler pypi scrapy wechat sogou

Updated Apr 17, 2023
Python

Followers: 291 followers
Wikipedia: Wikipedia