List of libraries, tools and APIs for web scraping and data processing.
Makefile
Updated Nov 1, 2018
#
web-scraping
Repositories 679
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
php
curl
class
api
api-client
client
framework
http
http-client
http-proxy
json
php-curl
php-curl-library
proxy
requests
restful
web-scraper
web-scraping
web-service
xml
PHP
Updated Oct 31, 2018
Web Scraping Framework
Python
Updated Sep 15, 2018
Apify SDK — The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and…
General Assembly's 2015 Data Science course in Washington, DC
data-science
machine-learning
scikit-learn
data-analysis
pandas
jupyter-notebook
python
course
linear-regression
logistic-regression
model-evaluation
naive-bayes
natural-language-processing
decision-trees
ensemble-learning
clustering
regular-expressions
web-scraping
data-visualization
data-cleaning
Jupyter Notebook
Updated Apr 18, 2016
A framework for creating semi-automatic web content extractors
python
css-selector
xpath-expression
web-scraper
web-scraping
scrapers
scraping
scrapy
selector
extractor
crawler
selector-expression
tutorial
lxml
beautifulsoup
Python
Updated May 1, 2018
Random User-Agent middleware based on fake-useragent
An unofficial API for Quora.
Python
Updated Oct 9, 2016
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
spider
golang
crawler
golang-application
lightweight
go
language-indepedent
elasticsearch
all-in-one
restful-api
web-crawler
crawling
web-spider
web-scraping
scraping
not-a-framework
no-need-to-code
cross-platform
builtin-ui
easy-to-use
Go
Updated Sep 1, 2018
Modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or si…
web-scraping
ruby
headless-chromium
phantomjs
scrapy
spider-framework
crawling-framework
nokogiri
mechanize
spider
capybara
kimurai
Ruby
Updated Oct 12, 2018
ACHE is a web crawler for domain-specific search.
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
Python
Updated Mar 22, 2017
Simple Query Scraping with CSS and Go Reflection
Go
Updated Oct 21, 2016
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Python
Updated May 24, 2017
A JavaScript library for generating random user agents with data that's updated daily.
user-agent
user-agent-spoofer
random
randomization
javascript
navigator
browsers
browser-automation
web-scraping
JavaScript
Updated Nov 6, 2018
Kantu for Chrome and Firefox - Modern Web Browser Automation plus Selenium IDE
MetaData html scraper and parser for Node.js (supports Promises and callback style)
JavaScript
Updated Oct 5, 2018
Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.o…
JavaScript
Updated Jul 17, 2018
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
machine-learning
sportsanalytics
sports-stats
machine-learning-algorithms
predictive-analysis
algorithms
selenium
beautifulsoup
python
python-2
scikit-learn
web-scraping
web-crawling
machinelearning
Jupyter Notebook
Updated Feb 12, 2017
Android app for saving webpages for offline reading.
java
android
android-application
offline
offline-storage
web-scraping
html-parser
html-files
viewer
parser
Java
Updated Jun 22, 2018
Python bindings to Modest engine (fast HTML5 parser with CSS selectors).
Python
Updated Sep 30, 2018
Zillow Scraper for Python using Selenium
Python
Updated Nov 4, 2018
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Python
Updated Apr 6, 2017
Scrapy Training companion code
Python
Updated Jul 19, 2017
Tutorial: Web scraping in Python with Beautiful Soup
Jupyter Notebook
Updated Aug 26, 2017
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
Python
Updated Oct 31, 2018
Collection of scripts corresponding to LucidProgramming YouTube tutorials
python-tutorial
web-scraping
youtube-tutorial
lucidprogramming
ctci-solutions
technical-interview
python
python3
python3-tutorial
Python
Updated Nov 5, 2018
Web scrapping and related analytics using Python tools
analytics
python
sqlite3
xml-parser
json
json-parser
sql
web-scraping
beautifulsoup4
database
regular-expression
nlp
natural-language-processing
Jupyter Notebook
Updated Nov 6, 2018
R
Updated Aug 14, 2018
Headless 'Chrome' Orchestration in R
R
Updated Oct 7, 2018