my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
https://patronview.com/news/99-percent-of-my-website-traffic...
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
2 access.log [1]
40 botpoop.log [2]
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
https://developers.cloudflare.com/bots/additional-configurat...
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
Quarter Crawl GPT Claude CC G-Ext Byte Bing Google
2023 Q1 2023-06 0.00% 0.00% 0.16% 0.00% 0.06% 0.47% 0.39%
2023 Q2 2023-14 0.00% 0.00% 0.18% 0.00% 0.06% 0.45% 0.38%
2023 Q3 none — — — — — — —
2023 Q4 2023-40 2.21% 0.00% 2.12% 0.04% 0.11% 0.39% 0.27%
2024 Q1 2024-10 0.53% 0.05% 0.31% 0.09% 0.18% 0.34% 0.31%
2024 Q2 2024-18 0.55% 0.09% 0.32% 0.11% 0.24% 0.32% 0.31%
2024 Q3 2024-30 0.68% 0.22% 0.36% 0.20% 0.38% 0.24% 0.33%
2024 Q4 2024-42 1.10% 0.50% 0.44% 0.32% 0.50% 0.25% 0.40%
2025 Q1 2025-05 1.14% 0.66% 0.54% 0.42% 0.66% 0.25% 0.44%
2025 Q2 2025-18 1.37% 0.93% 0.63% 0.70% 0.92% 0.29% 0.19%
2025 Q3 2025-30 1.42% 1.07% 0.74% 0.62% 1.01% 0.31% 0.27%
2025 Q4 2025-43 1.92% 1.51% 1.23% 1.15% 1.52% 0.27% 0.19%
2026 Q1 2026-04 2.13% 1.76% 1.68% 1.58% 1.77% 0.22% 0.15%
2026 Q2 2026-17 2.80% 2.38% 2.26% 2.13% 2.50% 0.22% 0.14%
2026 Q3 2026-30 3.45% 3.01% 2.89% 2.71% 3.16% 0.21% 0.14%
It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and the standard client solver.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
Don't let the perfect be the enemy of the good enough. For now.
Targeted attacks may not be repelled at all. That's not the point.
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
lazerg•1h ago
Yiin•53m ago
pixl97•49m ago
conartist6•39m ago
chrisandchris•47m ago
(i skimmed the whole post, there are none)
rhdunn•42m ago
Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.
askl•46m ago
runjake•40m ago
(This isn't a condemnation. AI can often do a better job of representing thoughts than humans.)
Examples:
- The article organization
- The general language flow
- The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats).
- The images are almost certainly AI generated. They look AI generated.
conartist6•37m ago
Using it slowly sucks the uniqueness out of you.
throwaway219450•30m ago
> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.
I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The <adjective> <noun phrase>”.
behole•45m ago
bookofjoe•39m ago
Best thing I've read on HN so far this year.
mysterydip•39m ago
pixl97•36m ago
So this is how AI wins, humans kill each other off because we might be bots and the bots inherit the earth.