warc

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Collect and revisit web pages.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

Serverless Web Archive Replay directly in the browser

Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)

WarcDB: Web crawl data as SQLite databases.

🐋 Web Archiving Integration Layer: One-Click User Instigated Preservation

Streaming WARC/ARC library for fast web archive IO

Bitextor generates translation memories from multilingual websites

News crawling with StormCrawler - stores content as WARC

Chrome extension to "Create WARC files from any webpage"

CoCrawler is a versatile web crawler built using modern tools and concurrency.

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

🐋 One-Click User Instigated Preservation

🗄️ A simple CLI for converting WARC to Parquet.

Parse And Create Web ARChive (WARC) files with node.js

Offline-first web browser