Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.
It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!
Sure, you could technically redesign to handle the bot traffic, but if the bot traffic is just taking the data and reducing any need for humans to visit the site, why is he putting in the effort to maintain the site?
The article says (speculates?) that malicious users are trying to get privileged access for an edge in prediction market betting. From the article:
> If you could see The Numbers data before everyone else, every single week, you would have a significant edge over all the other traders - learning the answers slightly ahead of publication would allow you to front-run the trades.
Deplorable behavior indeed
I'm not defending AI bots overwhelming websites or hackers motivated by Polymarket, etc., but I don't really believe a 30 year old website with "approximately 160,000 source files serving around 2 million pages" is a good litmus test for the state of the online world. What's worse is the absurdity that this basic site offering niche data would be targeted because Polymarket depends on it for some of their bets, something that most websites don't have to deal with. Frankly, that seems like a much more interesting angle to explore than "this old website that should have been re-written multiple times over the last three decades now doesn't have a choice but to be re-written."
Oddly, it seems like the solution, at least in this specific case, is relatively simple (and ironic): pay for an LLM to re-write this site in a modern, scalable, secure way and have that LLM monitor the site to ensure it is behaving correctly. Put this thing in a modern platform behind a proper cache (i.e., throw the whole thing up in Cloudflare) and many of these problems just don't exist anymore. If this site is as basic as it seems, this could be a weekend project.
It's certainly an interesting story, and I don't blame the owner of The Numbers for handling his site the way he has, but framing this as "AI is destroying our beloved internet" just seems obtuse, at least through this lens.
Or at least somewhere between that and full on protection racket.
LLMs come on the scene, hammer the site until it goes offline or racks up bills that threaten bankruptcy, and the solution then is to pay for an LLM to fix it and monitor it.
That's a real nice site ya got there, it would be a shame if massive datacenters going up around the world were to start hammering it from tens of thousands of IP addresses....
The intelligence community came to the conclusion (after research and experiments) that things like "prediction markets" were a bad idea in 1996.
Far worse now than then, with the net and AI of 2026 and the insane number of people now online that want to make an easy buck.
Don't get me wrong; I am guessing some of us on here would do well on those places. But they should not exist.
Despite the download link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
dumberquestions•1h ago