List of libraries, tools and APIs for web scraping and data processing.
Makefile
Updated Feb 17, 2019
#
web-scraping
Repositories 816
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
php
curl
class
api
api-client
client
framework
http
http-client
http-proxy
json
php-curl
php-curl-library
proxy
requests
restful
web-scraper
web-scraping
web-service
xml
PHP
Updated Mar 7, 2019
Web Scraping Framework
Python
Updated Mar 7, 2019
Apify SDK — The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extrac…
General Assembly's 2015 Data Science course in Washington, DC
data-science
machine-learning
scikit-learn
data-analysis
pandas
jupyter-notebook
python
course
linear-regression
logistic-regression
model-evaluation
naive-bayes
natural-language-processing
decision-trees
ensemble-learning
clustering
regular-expressions
web-scraping
data-visualization
data-cleaning
Jupyter Notebook
Updated Apr 18, 2016
Simple web scraping for R
Nextjs server to query websites with GraphQL
JavaScript
Updated Jan 26, 2019
A framework for creating semi-automatic web content extractors
python
css-selector
xpath-expression
web-scraper
web-scraping
scrapers
scraping
scrapy
selector
extractor
crawler
selector-expression
tutorial
lxml
beautifulsoup
Python
Updated Jan 7, 2019
Random User-Agent middleware based on fake-useragent
Modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or si…
web-scraping
ruby
headless-chromium
phantomjs
scrapy
spider-framework
crawling-framework
nokogiri
mechanize
spider
capybara
kimurai
Ruby
Updated Jan 30, 2019
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Go
Updated Feb 23, 2019
ACHE is a web crawler for domain-specific search.
An unofficial API for Quora.
Python
Updated Oct 9, 2016
Python scripts for building 'Short Jokes' dataset, featured on Kaggle
Python
Updated Mar 22, 2017
A JavaScript library for generating random user agents with data that's updated daily.
user-agent
user-agent-spoofer
random
randomization
javascript
navigator
browsers
browser-automation
web-scraping
JavaScript
Updated Mar 22, 2019
Kantu for Chrome and Firefox - Modern Web Browser Automation plus Selenium IDE
Collection of scripts corresponding to LucidProgramming YouTube tutorials
python-tutorial
web-scraping
youtube-tutorial
lucidprogramming
ctci-solutions
technical-interview
python
python3
python3-tutorial
Python
Updated Mar 15, 2019
Simple Query Scraping with CSS and Go Reflection
Go
Updated Oct 21, 2016
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Python
Updated Jan 4, 2019
Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)
machine-learning
sportsanalytics
sports-stats
machine-learning-algorithms
predictive-analysis
algorithms
selenium
beautifulsoup
python
python-2
scikit-learn
web-scraping
web-crawling
machinelearning
Jupyter Notebook
Updated Feb 12, 2017
Python bindings to Modest engine (fast HTML5 parser with CSS selectors).
Tutorial: Web scraping in Python with Beautiful Soup
Jupyter Notebook
Updated Nov 18, 2018
Scrapy Training companion code
Python
Updated Jan 30, 2019
Zillow Scraper for Python using Selenium
Python
Updated Nov 4, 2018
Apify actor that opens a web page in headless Chrome and analyzes the HTML and JavaScript objects, looks for schema.o…
JavaScript
Updated Jul 17, 2018
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Python
Updated Apr 6, 2017
MetaData html scraper and parser for Node.js (supports Promises and callback style)
JavaScript
Updated Oct 5, 2018
Android app for saving webpages for offline reading.
java
android
android-application
offline
offline-storage
web-scraping
html-parser
html-files
viewer
parser
Java
Updated Jun 22, 2018
Twitter Intelligence OSINT project performs tracking and analysis of the Twitter
Python
Updated Jan 16, 2019
Web scrapping and related analytics using Python tools
analytics
python
sqlite3
xml-parser
json
json-parser
sql
web-scraping
beautifulsoup4
database
regular-expression
nlp
natural-language-processing
data-science
data-wrangling
Jupyter Notebook
Updated Mar 7, 2019