COLLECTED BY

Organization: Archive Team

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.

History is littered with hundreds of conflicts over the future of a community, group, location or business that were "resolved" when one of the parties stepped ahead and destroyed what was there. With the original point of contention destroyed, the debates would fall to the wayside. Archive Team believes that by duplicated condemned data, the conversation and debate can continue, as well as the richness and insight gained by keeping the materials. Our projects have ranged in size from a single volunteer downloading the data to a small-but-critical site, to over 100 volunteers stepping forward to acquire terabytes of user-created data to save for future generations.

The main site for Archive Team is at archiveteam.org and contains up to the date information on various projects, manifestos, plans and walkthroughs.

This collection contains the output of many Archive Team projects, both ongoing and completed. Thanks to the generous providing of disk space by the Internet Archive, multi-terabyte datasets can be made available, as well as in use by the Wayback Machine, providing a path back to lost websites and work.

Our collection has grown to the point of having sub-collections for the type of data we acquire. If you are seeking to browse the contents of these collections, the Wayback Machine is the best first stop. Otherwise, you are free to dig into the stacks to see what you may find.

The Archive Team Panic Downloads are full pulldowns of currently extant websites, meant to serve as emergency backups for needed sites that are in danger of closing, or which will be missed dearly if suddenly lost due to hard drive crashes or server failures.

Collection: Archive Team: URLs

TIMESTAMPS

The Wayback Machine - https://web.archive.org/web/20220709111801/https://github.com/topics/scrape

#

scrape

Here are 374 public repositories matching this topic...

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

python elasticsearch kibana osint twitter tweets scrape pineapple twint tweep scrape-followers scrape-likes scrape-following

Updated May 19, 2022
Python

autoscraper

alirezamika / autoscraper

Sponsor

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

Updated Feb 3, 2021
Python

Anorov / cloudflare-scrape

A Python module to bypass Cloudflare's anti-bot page.

cloudflare anti-bot-page protected-page scrape scraping-websites

Updated Dec 21, 2021
Python

realsirjoe / instagram-scraper

scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot

python bot instagram crawler scraper scrape ig igramscraper

Updated Jul 6, 2022
Python

metascraper

microlinkhq / metascraper

Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

metadata parse scrape

Updated Jul 4, 2022
HTML

Altimis / Scweet

A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...

python scraper twitter tweets scraping scrape selenium-webdriver followers twitter-scraper scrape-tweets following save-image scrape-images scrape-followers scrape-likes scrape-following dowload-images

Updated Jun 23, 2022
Python

austinoboyle / scrape-linkedin-selenium

Open

Experiences Broken with latest UI

5

sviluppare7486 commented Mar 9, 2022

It seems the scrapper doesn't work at all anymore to retrieve experiences for linkedIn profiles.

Read more

bug help wanted good first issue

Open

Certifications return empty []

2

Open

Companyscraper doesn't work and returns error 'NoneType'

3

Find more good first issues

instamancer

ScriptSmith / instamancer

Scrape Instagram's API with Puppeteer

instagram data-mining instagram-scraper scrape instagram-api puppeteer

Updated Jun 23, 2022
TypeScript

Anonyfox / elixir-scrape

Scrape any website, article or RSS/Atom Feed with ease!

html rss data-science information-retrieval elixir scraping feed readability scrape

Updated Jul 25, 2020
Elixir

glebarez / cero

Scrape domain names from SSL certificates of arbitrary hosts

tls ssl scrape recon websecurity domain-names

Updated May 12, 2022
Go

ultralytics / google-images-download

Google/Bing Images Web Downloader

scraper download images scrape google-image-search google-images-downloader bing-images-downloader google-image-downloader

Updated Jan 17, 2021
Python

JaredLGillespie / proxyscrape

Python library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5).

python scraper proxy python3 scrape

Updated Sep 22, 2021
Python

evyatarmeged / Humanoid

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

bot scraping anti-bot-page web-scraping scrape anti-bot bypass humanoid bypass-waf

Updated Feb 6, 2022
JavaScript

luengwaiban / instagram-python-scraper

A instagram scraper wrote in python. Similar to instagram-php-scraper.Usages are in example.py. Enjoy it!

python instagram scraper scrape ig ins clawer

Updated Jun 1, 2021
Python

fansly

Avnsx / fansly

Simply scrape / download all the media from an fansly account

python windows macos linux lightweight scraper downloader image video download archive scrape user-friendly datascraping fansly fansly-scraper fansly-downloader fansly-download

Updated May 12, 2022
Python

unixfox / pupflare

A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

docker koa proxy chromium cloudflare anti-bot-page protected-page scrape scraping-websites puppeteer cloudflare-bypass cloudflare-scrape

Updated Jun 15, 2022
JavaScript

markowanga / stweet

Advanced python library to scrap Twitter (tweets, users) from unofficial API, fully covered by integration tests

python search api scraper twitter tweets twitter-api users unofficial crawl scrape scrap user tweet scrapper twint scrap-tweet searchrunner

Updated Oct 26, 2021
Python

rocketlaunchr / google-search

Sponsor

scrape google search results

search go api golang google scrape

Updated May 24, 2022
Go

badoux / goscraper

Golang pkg to quickly return a preview of a webpage (title/description/images)

url go golang website crawler scraper image webpage engine icon preview scrape description page title scrapper

Updated Jan 18, 2021
Go

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

scraper rest sensor scraping scrape home-assistant hacs home-assistant-custom

Updated Jul 5, 2022
Python

serp-spider / core

🕷️ The PHP SERP Spider - A search engine scraper

scraping scrape scrap serp

Updated Feb 5, 2021
PHP

html2rss

html2rss / html2rss

Open

Remove reverse option

gildesmarais commented Jun 21, 2022

It prevents potential optimizations and the order of items is - according to spec - irrelevant.

remove from code
remove from docs/readme

Read more

good first issue

Jimut123 / jimutmap

API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.

Updated Jun 24, 2022
Python

DrKain / scrape-youtube

A lightning fast package to scrape YouTube search results

nodejs search bot npm playlist youtube video movie discord scrape

Updated Jul 4, 2022
JavaScript

fefit / visdom

A fast and easy to use library for html parsing & node selecting & node mutation, suitable for web scraping and html confusion.

jquery rust css-selector dom-manipulation scrape confuse html-parse

Updated Apr 26, 2022
Rust

warifp / Shopee-Scrape

Shopee Scrape is a tool that functions to collect data - the data needed, such as finding data from photos, prices, names, store locations and others.

php scraper marketplace curl php-library indonesia scrape scraped-data scraper-engine curl-functions curl-library scrape-websites curlphp shopee scrape-images shopee-api

Updated Aug 24, 2021
PHP

Pringleman83 / SportsBook

A sports data scraping and analysis tool

python json data analysis league prediction soccer comparison menu football-data scrape compare football sport predict

Updated Jul 12, 2019
Python

Swader / diffbot-php-client

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

nlp bot php machine-learning scraper ai scraping crawling artificial-intelligence crawl scrape scraped-data diffbot

Updated Jul 4, 2018
PHP

brenns10 / tswift

MetroLyrics API for Python

python lyrics taylor-swift scrape

Updated Oct 13, 2021
Python

vincetran96 / Scrape-Finance-Data-v2

A standalone package to scrape financial data from listed Vietnamese companies via Vietstock

python docker redis finance data scrape

Updated Sep 27, 2021
Python

Improve this page

Add a description, image, and links to the scrape topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the scrape topic, visit your repo's landing page and select "manage topics."