It's not crawling for training that is the issue...and it's an over simplification stating that AI companies are "training" on someone's data.
When systems like Claude and ChatGPT fetch Wikimedia content to answer user queries in real time, they're effectively using Wikimedia as an API – with zero compensation, zero attribution, and zero of the typical API management that would come with such usage. Each time a user asks these AI tools a question, they may trigger fresh calls to Wikimedia servers, creating a persistent, on-demand load rather than a one-time scraping event.
The distinction is crucial. Traditional search engines like Google crawl content, index it, and then send users back to the original site. These AI systems instead extract the value without routing any traffic back, breaking the implicit value exchange that has sustained the web ecosystem.
Wikimedia's focus on technical markers of "bot behavior" – like not interpreting JavaScript or accessing uncommon pages – shows they're still diagnosing this as a traditional crawler problem rather than recognizing the fundamental economic imbalance. They're essentially subsidizing commercial AI products with volunteer-created content and donor-funded infrastructure.
The solution has been available all along. HTTP 402 "Payment Required" was built into the web's foundation for exactly this scenario. Combined with the Lightning Network's micropayment capabilities and the L402 protocol implementation, Wikimedia could:
- Keep content free for human users
- Charge AI services per request (even fractions of pennies would add up)
- Generate sustainable infrastructure funding from commercial usage
- Maintain their open knowledge mission while ending the effective subsidy
Instead of leading the way toward a sustainable model for knowledge infrastructure in the AI age, Wikimedia is writing blog posts about traffic patterns. If your content is being used as an API, the solution is to become an API – with all the management, pricing, and terms that entails. Otherwise, they'll continue watching their donor resources drain away to support commercial AI inference costs.
I suspect several factors contribute to this resistance:
Ideological attachment to "free" as binary rather than nuanced: Many organizations have built their identity around offering "free" content, creating a false dichotomy where any monetization feels like betrayal of core values. They miss that selective monetization (humans free, automated commercial use paid) could actually strengthen their core mission.
Technical amnesia: The web's architects built payment functionality into HTTP from the beginning, but without a native digital cash system, it remained dormant. Now that Bitcoin and Lightning provide the missing piece, there's institutional amnesia about this intended functionality.
Complexity aversion: Implementing new payment systems feels like adding complexity, when in reality it simplifies the entire ecosystem by aligning incentives naturally rather than through increasingly byzantine rate-limiting and bot-detection schemes.
The comfort of complaint: There's a certain organizational comfort in having identifiable "villains" (bots, crawlers, etc.) rather than embracing solutions that might require internal change. Blog posts lamenting crawler impacts are easier than implementing new systems.
False democratization concerns: Some worry that payment systems would limit access to those with means, missing that micropayments precisely enable democratization by allowing anyone to pay exactly for what they use without arbitrary gatekeeping.
But so does unrestricted AI use. I guess the nice things era is over.
> Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%. This increase is not coming from human readers, but largely from automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models.
1. Bandwidth consumption - that's on scrapers downloading multimedia files
2. CPU resource exhaustion - AI scrapers don't take contextual clues into account. They just blindly follow each and every link they can find, which means that they hit a lot of pages that aren't cached but re-generated for each call. That's stuff like the article history but especially the version delta pages. These are very expensive to generate and are so rarely called that it doesn't make sense to cache them.
> I suggest Wikimedia to distribute Wikimedia Commons content using tape drive. The largest tape drive (IBM 3592) can store 50 TB content. The total size of Wikimedia Commons is 610.4 TB. So it needs less than 15 tapes to store the entire site. You can lend the tapes to any company want your content, if they promise to return a in period of time.
The thing is also that corporate scrapers are in comparison even the good guys, they respect robots.txt, have properly set user agents etc. Others might do neither and from residential IPs.
The issue isn't even new but any proper solution attempts have been postponed because CDNs seem such an easy solution.
Every customer would prefer a firehose content delta over having to scrape for diffs.
They obviously have the capital to provide this, and still grow their funds for eternity without ever needing a single dollar in external revenue.
Why don't they?
customers is a strong word, especially when you're saying they should be providing a new service useful, more or less exclusively, to AI startups and megacorps
Maybe a similar system needs to be set up so that bot requests need to present their latest cache or hash ID of the requested content before a full request can be granted. This way, if the local cache is recent, it doesn't burden the server with requests for content they've already seen, and they can otherwise serve their users information based on the version they have stored locally.
In my experience, the problematic crawlers choose not to implement this feature.
https://feder001.com/exploring-wikipedia-as-a-database-part-...
At least for local AIs it might not be a terrible idea. Basically a distributed cache of the most common sources our bots might pull from. That would mean only a few fetches from each website per day, and then the rest of the bandwidth load can be shared amongst the bots.
Probably lots of privacy issues to work around with such an implementation though.
(91 points, 30 days ago, 101 comments) https://news.ycombinator.com/item?id=43555898
(49 points, 29 days ago, 45 comments) https://news.ycombinator.com/item?id=43562005
I fear the end state of this game is the death of the anonymous internet.
Problem is, as you've discovered, it can have the cost that anti-fingerprinting browsers can't do the required work.
Even if, as you say, crawlers will hit the PoW thousands of times more, the only way to make it a barrier is if the cost is higher than the profit to be gained. Otherwise it's merely an expense to be passed on to the customer.
If that is the case, I would think that it is a little bit concerning that the model of Wikipedia is based on having most resources not accessed.
Otherwise, if my understanding is wrong, it would mean that AI company are constantly scraping the same content for change like a search engine would do, but it does little sense to me as I easily guess that models are only trained once every few months at most.
And also I don't understand how they were not already encountering this problem with the existing constant crawling of search engines...
ianso•13h ago
So it's not like you need to crawl the sites to get content for training your models...
qudat•13h ago
DarkWiiPlayer•13h ago
Sounds like the problem is not the crawling itself but downloading multimedia files.
The article also explains that these requests are much more likely to request resources that aren't cached, so they generate more expensive traffic.
StableAlkyne•13h ago
It's not clear which files you need, and the site itself is (or at least, was when I tried) "shipped" as some gigantic SQL scripts to rebuild the database with enough lines that the SQL servers I tried gave up reading them, requiring another script to split it up into chunks.
Then when you finally do have the database, you don't have a local copy of Wikipedia. You're missing several more files, for example category information is in a separate dump. Also you need wiki software to use the dump and host the site. After a weekend of fucking around with SQL, this is the point where I gave up and just curled the 200 or so pages I was interested in.
I'm pretty sure they want you to "just" download the database dump and go to town, but it's such a pain in the ass that I can see why someone else would just crawl it.
jsheard•13h ago
More recently they starting putting the data up on Kaggle in a format which is supposed to be easier to ingest.
https://enterprise.wikimedia.com/blog/kaggle-dataset/
StableAlkyne•12h ago
PeterStuer•12h ago
mjevans•11h ago
Also make a cname from bots.wikipedia.org to that site.
GuinansEyebrows•10h ago
neets•10h ago
https://dumps.wikimedia.org/kiwix/zim/wikipedia/
Philpax•9h ago
1. Go to https://dumps.wikimedia.org/enwiki/latest/ (or a date of your choice in /enwiki)
2. Download https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page... and https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.... The first file is a bz2-multistream-compressed dump of a XML containing all of English Wikipedia's text, while the second file is an index to make it easier to find specific articles.
3. You can either:
4. Once you have the XML, getting at the actual text isn't too difficult; you should use a streaming XML decoder to avoid as much allocation as possible when processing this much data.The XML contains pages like this:
so all you need to do is get at the `text`.ks2048•8h ago
I know there are now a couple pretty-good wikitext parsers, but for years, it was a bigger problem. The only "official" one was the huge php app itself.
Philpax•8h ago
I wrote another Rust library [1] that wraps around `parse-wiki-text-2` that offers a simplified AST that takes care of matching tags for you. It's designed to be bound to WASM [2], which is how I'm pretty reliably parsing Wikitext for my web application. (The existing JS libraries aren't fantastic, if I'm being honest.)
[0]: https://github.com/soerenmeier/parse-wiki-text-2
[1]: https://github.com/philpax/wikitext_simplified
[2]: https://github.com/genresinspace/genresinspace.github.io/blo...
cubefox•13h ago
mistrial9•12h ago
bombcar•11h ago
A torrent of all images updated once a year would probably do quite well.
edoceo•11h ago
bombcar•9h ago
indrora•7h ago
For those unfamiliar: The images that are marked NonFree must be smaller than 1Megapixel. 1155 X 866. In practice, 1024x768 is around the maximum size.
mtmail•12h ago
hombre_fatal•12h ago
Crawling is more general + you get to consume it in its reconstituted form instead of deriving it yourself.
Hooking up a data dump for special-cased websites is much more complicated than letting LLM bots do a generalized on-demand web search.
Just think of how that logic would work. LLM wants to do a web search to answer your question. Some Wikimedia site is the top candidate. Instead of just going to the site, it uses this special code path that knows how to use https://{site}/{path} to figure out where {path} is in {site}'s data dump.
black_puppydog•11h ago
Ekaros•11h ago
bombcar•11h ago