5.9B
5.9B
∑collection
�eye
5.9B
The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.
Topic: webwidecrawl
2.6B
2.6B
∑collection
�eye
2.6B
Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.
Topics: web crawl, Alexa
2.4B
2.4B
∑collection
�eye
2.4B
Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
1.3B
1.3B
∑collection
�eye
1.3B
Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
1.3B
1.3B
∑collection
�eye
1.3B
Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
553M
553M
∑collection
�eye
553M
Archive-It is a subscription web archiving service of the Internet Archive that helps organizations harvest, build, and preserve collections of digital content. Partners create domain specific collections of web captures that can be searched on Archive It. Content is hosted and stored at the Internet Archive data centers. Archive-It works with more than 400 partner organizations in 48 U.S. states and 16 countries worldwide including: College and University Libraries State Archives, Libraries,...
Topic: Colleges, Universities, Libraries, Archives, NGOs, Museums
477.5M
478M
∑collection
�eye
477.5M
Survey crawl of domains started April 2013. This data is currently not publicly accessible.
467.3M
467M
Nov 4, 2011
11/11
by
Internet Archive
∑collection
�eye
467.3M
Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.
Topic: webcrawl
398.9M
399M
Apr 8, 2011
04/11
by
Internet Archive
∑collection
�eye
398.9M
National library harvesting.
Topic: ccs
398.3M
398M
∑collection
�eye
398.3M
miscellaneous data
Topic: brad tofel
359.5M
360M
∑collection
�eye
359.5M
Web wide crawl with initial seedlist and crawler configuration from April 2013.
344.4M
344M
∑collection
�eye
344.4M
Wayback indexes. This data is currently not publicly accessible.
312.7M
313M
∑collection
�eye
312.7M
A daily collection of thousands of the most popular web sites according to Alexa.com's top sites rankings.
Topics: daily, popular sites, Alexa
311.9M
312M
∑collection
�eye
311.9M
Archive-It is the leading web archiving service for collecting and accessing cultural heritage on the web and is a service of Internet Archive used by libraries, archives, governments, non-profits, and other organizations to build collections of web materials.
Topic: TK
304.9M
305M
∑collection
�eye
304.9M
These crawls are part of an effort to archive pages as they are created and archive the pages that they refer to. That way, as the pages that are referenced are changed or taken from the web, a link to the version that was live when the page was written will be preserved. Then the Internet Archive hopes that references to these archived pages will be put in place of a link that would be otherwise be broken, or a companion link to allow people to see what was originally intended by a page's...
302.9M
303M
∑collection
�eye
302.9M
this data is currently not publicly accessible.
277.1M
277M
∑collection
�eye
277.1M
Survey crawl of domains started December 2014. This data is currently not publicly accessible.
233.1M
233M
∑collection
�eye
233.1M
Web wide crawl with initial seedlist and crawler configuration from June 2014.
223.1M
223M
∑collection
�eye
223.1M
Web wide crawl with initial seedlist and crawler configuration from August 2013.
217.2M
217M
∑collection
�eye
217.2M
this data is currently not publicly accessible.
214.1M
214M
∑collection
�eye
214.1M
Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
206.8M
207M
∑collection
�eye
206.8M
Collections of Wiki data
Topics: crawls, data, wiki
203.7M
204M
∑collection
�eye
203.7M
Crawl of outlinks from wikipedia.org. These files are currently not publicly accessible. from Wikipedia: Wikipedia is a multilingual, web-based, free-content encyclopedia project operated by the Wikimedia Foundation and based on an openly editable model. The name "Wikipedia" is a portmanteau of the words wiki (a technology for creating collaborative websites, from the Hawaiian word wiki, meaning "quick") and encyclopedia. Wikipedia's articles provide links to guide the user...
201M
201M
∑collection
�eye
201M
Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history. History is littered with hundreds of conflicts over the future of a community, group, location or...
194.9M
195M
∑collection
�eye
194.9M
Web wide crawl with initial seedlist and crawler configuration from April 2012.
193.4M
193M
∑collection
�eye
193.4M
Web wide crawl with initial seedlist and crawler configuration from January 2015.
162.2M
162M
∑collection
�eye
162.2M
This is a collection of web page captures from links added to, or changed on, Wikipedia pages. The idea is to bring a reliability to Wikipedia outlinks so that if the pages referenced by Wikipedia articles are changed, or go away, a reader can permanently find what was originally referred to. This is part of the Internet Archive's attempt to rid the web of broken links.
Topics: Wikipedia, Wikimedia
146.9M
147M
∑collection
�eye
146.9M
Survey crawl of domains. This data is currently not publicly accessible.
143.1M
143M
∑collection
�eye
143.1M
Survey crawl of domains started May 2014. This data is currently not publicly accessible.
142.3M
142M
∑collection
�eye
142.3M
Web wide crawl with initial seedlist and crawler configuration from October 2010
139.2M
139M
∑collection
�eye
139.2M
Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
138.9M
139M
∑collection
�eye
138.9M
Web wide crawl with initial seedlist and crawler configuration from September 2012.
132.7M
133M
∑collection
�eye
132.7M
Data crawled by Sloan Foundation on behalf of Internet Archive
125M
125M
∑collection
�eye
125M
Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
124.1M
124M
∑collection
�eye
124.1M
Survey crawl of domains. This data is currently not publicly accessible.
119.8M
120M
∑collection
�eye
119.8M
A daily collection of hundreds of the world's top news sites.
Topics: daily, news
117.9M
118M
∑collection
�eye
117.9M
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites). To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel...
Topics: archiveteam, archivebot, webcrawl, robot, love
117.6M
118M
∑collection
�eye
117.6M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
112.2M
112M
∑collection
�eye
112.2M
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
106.8M
107M
∑collection
�eye
106.8M
Web wide crawl with initial seedlist and crawler configuration from February 2014.
105.5M
106M
∑collection
�eye
105.5M
101.2M
101M
∑collection
�eye
101.2M
this data is currently not publicly accessible.
97M
97M
∑collection
�eye
97M
this data is currently not publicly accessible.
94.1M
94M
∑collection
�eye
94.1M
this data is currently not publicly accessible.
89.8M
90M
∑collection
�eye
89.8M
This is a collection of pages and embedded objects from WordPress blogs and the external pages they link to. Captures of these pages are made on a continuous basis seeded from a feed of new or changed pages hosted by Wordpress.com or by Wordpress pages hosted by sites running a properly configured Jetpack wordpress plugin.
Topics: Wordpress.com, blogs, jetpack
88.2M
88M
∑collection
�eye
88.2M
Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
88.2M
88M
∑collection
�eye
88.2M
Crawls performed by Internet Archive on behalf of the National Library of Australia. This data is currently not publicly accessible.
86.2M
86M
∑collection
�eye
86.2M
Crawl EG from Alexa Internet. This data is currently not publicly accessible.
81.2M
81M
∑collection
�eye
81.2M
81.1M
81M
∑collection
�eye
81.1M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
80.1M
80M
∑collection
�eye
80.1M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
75.9M
76M
∑collection
�eye
75.9M
Data collected by Internet Archive on behalf of the National Library of Spain. This data is currently not publicly accessible.
65.5M
66M
∑collection
�eye
65.5M
this data is currently not publicly accessible.
63.7M
64M
∑collection
�eye
63.7M
this data is currently not publicly accessible.
59.6M
60M
∑collection
�eye
59.6M
this data is currently not publicly accessible.
55.8M
56M
∑collection
�eye
55.8M
Crawls of the french domain space performed by Internet Archive on behalf of Bibliotheque Nationale de France. This data is currently not publicly accessible.
50.7M
51M
∑collection
�eye
50.7M
this data is currently not publicly accessible.
48.3M
48M
∑collection
�eye
48.3M
Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
47.4M
47M
∑collection
�eye
47.4M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
46.1M
46M
∑collection
�eye
46.1M
Crawl EI from Alexa Internet. This data is currently not publicly accessible.
45.9M
46M
∑collection
�eye
45.9M
Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
44.5M
45M
∑collection
�eye
44.5M
this data is currently not publicly accessible.
44.2M
44M
∑collection
�eye
44.2M
Crawls of International News Sites
43.8M
44M
∑collection
�eye
43.8M
Crawl DX from Alexa Internet. This data is currently not publicly accessible.
43M
43M
∑collection
�eye
43M
this data is currently not publicly accessible.
42.9M
43M
∑collection
�eye
42.9M
This data is currently not publicly accessible.
41.7M
42M
∑collection
�eye
41.7M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
41.5M
41M
∑collection
�eye
41.5M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
41M
41M
∑collection
�eye
41M
Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
40.3M
40M
∑collection
�eye
40.3M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
Topic: crawldata
40.2M
40M
∑collection
�eye
40.2M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
39.7M
40M
∑collection
�eye
39.7M
Internet Archive Global EventsArchive-It Partner Since: Feb, 2006Organization Type: Other InstitutionsOrganization URL:http://www.archive-it.org
39.5M
40M
∑collection
�eye
39.5M
Collection of web items from the 20th century.
Topics: web, 20th century
38.5M
39M
∑collection
�eye
38.5M
This collection contains collaborative Election crawls performed by IA.
Topics: elections, web
38.5M
39M
∑collection
�eye
38.5M
This crawl was performed in Summer & Fall of 2012 to archive the US Federal Elections.
Topics: US, federal, elections, web, 2012