Celebrating 1 Trillion Web Pages Archived

https://blog.archive.org/trillion/

95•pabs3•2h ago

Comments

typpilol•1h ago

I thought this was going to be a technical article but there was nothing in it

ehsanu1•59m ago

Seeing some stats would be fun. I wonder what the amount of data is here. And the distribution would be interesting too, especially since some pages are archived at multiple points in time, and pages have been getting heavier these days.

ChrisArchitect•58m ago

Related blog post inviting stories:

https://blog.archive.org/2025/09/23/celebrating-1-trillion-w...

arjie•45m ago

Something I wish we could have is some kind of peer mirror of archive.org. The main IA web application gets angry pretty quickly if you're trying to click through a few different dates. If there were some kind of way to slowly mirror (torrent-style) and offer pages as a peer from archive.org that would be neat. It would be cool to show up as an alternative source for the data and the archive.org app could fetch it out of there on a user's choice and validate the checksum if required.

In the end, I've ended up just keeping my own ArchiveBox and it's an all right experience. In the end, it's only useful for things I know I wanted to archive. For almost everything I go to the IA - which has so much.

zapataband2•29m ago

Is there such thing as "versioned" torrents? Assuming you have the right PGP key you could mix bittorrent and packaging systems to get an update-able distribution

hsbauauvhabzb•28m ago

A Torrent would probably suffocate under the small file distribution. I’m not sure how the romset torrents work but I thought they were versioned.

But torrent is probably the wrong tech. I’m sure there would be many players willing to host a few TB or more each, which could be fronted via something so it’s transparent to the user.

But a better option might be a subscription model, anything else will be slammed by crawlers.

pabs3•15m ago

Couple of BEPs related to updating torrents:

https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html

pabs3•14m ago

If anyone wants to help feed in more stuff, ArchiveTeam is a related volunteer group that sends data to IA:

https://archiveteam.org/

jonah-archive•14m ago

Hi, I run the datacenter/infrastructure team at the Internet Archive! We would love to see you at our various events this fall but if paying for the ticket is difficult for you, please email me (in bio) and we'll get you in (if possible).

awesomeMilou•6m ago

What events are we talking about here?

zghst•6m ago

A great milestone for internet history!

Why the clock is ticking on the Fed-fuelled AI bubble

Sora Is an Unholy Abomination

Terence Tao on Crowdsourced Effort

More Speculations on Arenas in C++

A Discord Channel Became a Passive Revenue Stream

AI generated 3D textures for artists

'Human intelligence problem': Labor senator slams Deloitte's AI bungle

Understanding Object-Oriented Programming [an open-source book]

Nvidia Partner Hon Hai's Sales Rise 11% in Boost for AI Outlook

Show HN: PageIndex for Reasoning-Based RAG

Gem.Coop – Community-Run Alternative to Rubygems.org, Led by Former Maintainers

The Gem Cooperative

First Brands' Fallout Exposes Risks of Lending Private Debt

Morningstar DBRS points to threat of rising private credit defaults in H2

Show HN: Ready-to-explore, forkable spaces to understand topics

An Open Letter to Llms.txt

The End of tt-rss.org

World Metro Network Connectivity Maps [pdf]

Gem.coop

AI Investors Are Chasing a Big Prize. Here's What Can Go Wrong

The $40k a year school where AI shapes every lesson, without teachers

AI personal assistant, what do you use?

MacBooks Don't Scratch

Seeing Through the Reality of Meta's Smart Glasses

Why the US and Venezuela Are on the Brink of War [video]

Open Building Population Layer for Canada

Oily Pine – Chapter 1

CHERI with a Linux on Top

U.S. tech giants hit pause on India datacenter deals under weight of uncertainty

Errors in Palladium's "How GDP Hides Industrial Decline