However, I've also seen reports that after getting blocked one way or another, they start crawling with browser user-agents from residential IPs. But it might also be someone else misrepresenting their crawlers as OpenAI/Amazon/Facebook/whatever to begin with.
While we were rate limiting bots based on UA, we ended up also having to apply wider rules because traffic started spiking from other places.
I can't say if it's the traffic shifting, but there's definitely a big amount of automated traffic not identifying itself properly.
If you look at all your web properties, look at historic traffic to calculate <hits per IP> in <time period>. Then look at the new data and see how it's shifting. You should be able to identify the real traffic and the automated very quickly.
All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do not respect robots.txt
"It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/
> If you think these [AI] crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.
Sourcehut (the site described) used Anubis before swithing "to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better)." https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/
As I said, it might be someone else entirely using OpenAI/Amazon/Meta/etc user agents to hide their real identity while ignoring robots.txt. What's to stop them? People blame those companies anyway.
If this isn't the case, then the bot detection systems big sites are using must be pretty bad, because I do almost all of my browsing on a desktop originating from residential ASN IP address, and I routinely run up against CAPTCHAs. E.g., any Stack Exchange site on first visit, and even Amazon. What reason would there be for this, unless these crawlers are laundering their traffic through residential IPs?
While our platform is primarily designed for live, logged-in users, it also works well for bot detection and blocking.
We anonymize IP addresses by replacing the last octet with an asterisk, effectively grouping the same subnet under a single account. You can then use the built-in rule engine to automatically generate blacklists based on specific conditions, such as excessive 500 or 404 errors, brute-force login attempts, or traffic from data center IPs.
Finally, you can integrate tirreno blacklist API into your application logic to redirect unwanted traffic to an error page.
Bonus: a dashboard [2] is available to help you monitor activity and fine-tune the blacklist to avoid blocking legitimate users.
[1] https://github.com/tirrenotechnologies/tirreno
[2] https://play.tirreno.com/login (admin/tirreno)
So as a user, not only do I have to suffer your blockwall's false positives based on "data center IPs" (ie most things that aren't naively browsing from the information-leaking address of a last-mile connection like some cyber-bumpkin). But if I do manage to find something that isn't a priori blocked (or manage to click through 87 squares of traffic lights), I still then get lumped in with completely unrelated address-neighbors to assuage your conscience that you're not building a user surveillance system based on nonconsentually processing personal information.
Just please make sure you have enough of a feedback process that your customers can see that they are losing real customers with real dollars.
As in my example, if the IP is from a data center and creates a lot of 404 errors, send it to a manual review queue or to automatic blocking (not recommended).
Personally, I prefer to manually review even bot activity, and tirreno, even if it's not directly designed for bot management, works great for this, especially in cases when bad bots are hidden behind legitimate bot UA's.
It’s so annoying.
If you ban by the /24, that really feels like you're coming back to the previous approach, just with extra steps.
However, if someone prefers to take action against non-logged in users based solely on IP, that's their own choice.
Overly simplistic solutions like this absolutely will actively cost you real customers and real revenue.
There is a extra cost only if you choose to block users automatically, regardless of the tool used.
tirreno is designed to work with logged-in users, so all actions are tied to usernames rather than IP addresses. From this perspective, we strongly avoid [1] making any decisions or taking actions based solely on IP address information.
I know you're processing them dynamically as they come in and break the rules. But if you wanted to supplement the list, might be worth sourcing the ones from https://github.com/ai-robots-txt/ai.robots.txt at some frequency.
Everyone else is visiting your site for entirely self-serving purposes, too.
I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site.
If you publish data to the public, expect the public to access it. If you don’t want the public (this includes AI startups) to access it, don’t publish it.
Your website is not being misused when the data is being downloaded to train AI. That’s literally what public data is for.
I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.
Which IMO shows that is already possible to do this and and it has been for a while already.
I’m pretty open‐minded about AI, and have no visceral objection to AI scraping in theory. However… the rise of AI scrapers is the first time in twenty years of running tiny websites that my servers—all my servers, consistently—hit CPU and bandwidth usage levels far above baseline, and even brush against my hosts’ monthly limits. When I check server logs, the traffic is largely bots repeatedly trawling infinite variations of version control history pages. Googlebot never crawled these enough to become a problem; Internet Archive never crawled these enough to become a problem. But the current scrapers are, and literally everyone I talk to who hosts their own VCS webservers is having the same problem right now. Many are choosing to reduce the problem by injecting Cloudflare or Anubis, which I as a user hate, and refuse to do myself—but what alternative can I propose to them? People running these servers are just users of Gitea and CGit and whatnot, they’re not capable of rewriting the underlying tools to reduce the CPU usage of diff generation.
Surprised this isn’t a feature in Gitea already (even though it is better done in the reverse proxy web server).
Assuming that there are simple universal solutions, eg based on IP or UA, suggests that you have been lucky enough not to deal with many varieties of this misbehaviour.
Google and traditional crawlers are fine. This new breed of AI crawlers have no scruples, no rate limits, and will just load pages and follow links as fast as they can, very likely overwhelming smaller web servers in the process.
I too have blocked many of them to avoid them filling my logs with nonexistent page accesses, for example.
I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.
Google and the publisher share the value derived from that scraping event (somewhat equitably) when a user clicks a link in Google's index. OpenAI, Anthropic, and co extract all of the value, and incur costs to the publisher, from their scraping events.
How else?
And to clarify,
It's a part of the UI or something and only a human should be pressing it, and there's no other way to access that API or something?
AI agents exist now, there is virtually no way to distinguish between real user and bot if they mimic human patterns.
Of these, I certainly wouldn’t ban Google, and probably not the others, if I wanted others to see it and talk about it.
Even if your content were being scraped for some rando’s AI bot, why have a public site if you don’t expect your site to be used?
Turning the lights off on the motel sign when you want people to find it is not a good way to invite people in.
The current round of AI nonsense also very poor. Again had to send legal notes to investor relations and PR depts in at least one well-known case, as well as all the technical measures, to restore some sort of decorum.
Google also runs an AI scraper, which might be what you saw represented there?
Did Google never ever scrape individual commits from Gitea?
That is not at all what is happening.
What's described in the article did my personal definition pretty well.
Especially when this happens ?
Google is using AI to censor independent websites like mine
https://news.ycombinator.com/item?id=44124820
Sure sounds like we've reached the point where it's more of a liability !
My fave method is still just to have bait info in robots.txt that gzip bombs and autoblocks all further requests from them. Was real easy to configure in Caddy and tends to catch the worst offenders.
Not excusing the bot behaviours but if a few bots blindly take down your site, then an intentionally malicious offender would have a field day.
Maybe I'm just a different generation than the folks writing these blog posts, but I really don't understand the fixation on such low resource usage.
It's like watching a grandparent freak out over not turning off an LED light or seeing them drive 15 miles to save 5c/gallon on gas.
20 requests per second is just... Nothing.
Even if you're dynamically generating them all (and seriously... Why? Time would have been so much better spent fixing that with some caching than this effort) it's just not much demand.
I get the "fuck the bots" style posts are popular in the Zeitgeist at the moment, but this is hardly novel.
There are a lot more productive ways to handle this that waste a lot less of your time.
Never had any actual performance issue, but I can see why a site that expects generally a very low traffic rate might freak out. Could they better optimize their sites? Probably, I know ours sucks big time. But in the era of autoscaling workloads on someone else's computer, a misconfigured site could rack up a big ass bill.
No, they just do it. So that can scrape data, which at this point in time for AI which has hit the cap on what it can consume knowledge wise, scrapes it because live updates and new information is most valuable to them.
So they will find tricky, evil ways to hammer resources that we as site operators own; even minimally to use site data to their profit, their success, their benefits while blatantly saying 'screw you' as they ignore robots.txt or pretend to be legitimate users.
There's a digital battle field going on. Clients are coming in as real users using IP lists like from https://infatica.io/
A writeup posted to HN about it
https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-...
A system and site operator has every right to build the tools they want to protect their systems, data, and have a user experience that benefits their audiences.
Your points are valid and make sense, but; it's not about that. It's about valuing authentic works, intellectual properties, and some dweeb that wants to steal it doesn't get to just run their bots against resources at others detriments, and their benefits.
They do ask: they make an HTTP request. How the server responds to that request is up to the owner. As in the article, the owner can decide to respond to that request however he likes.
I think that a big part of the issue is that software is not well-written. If you think about it, even the bots constantly requesting tarballs for git commits doesn’t have to destroy the experience of using the system for logged-in users. One can easily imagine software which prioritises handling requests for authorised users ahead of those for anonymous ones. One can easily image software which rejects incoming anonymous requests when it is saturated. But that’s hard to write, and our current networks, operating systems, languages and frameworks make that more difficult than it has to be.
It's kind of like me asking to borrow your car to go to work and then I take your car and ship it overseas.
This isn't about mere dozen requests. It gets pretty bad. It also slows down his life.
Another situation is an expensive resource. This might be bandwidth hogs, CPU heavy, or higher licensing per CPU's in databases. Some people's sites or services dont scale well or hit their budget limits fast.
In a high-security setup, those boxes usually have limited performance. It comes from the runtime checks, context switches, or embedded/soft processors. If no timing channels, one might have to disable shared caches, too.
Those systems run slow enough that whatever is in front usually needs to throttle the traffic. We'd want no wasted traffic given their cost ranges from $2,000 / chip (FPGA) to six digits a system (eg XTS-500 w/ STOP OS). One could say the same if it was a custom or open-source chip, like Plasma MIPS.
Many people might be in the poor category. A significant number are in the low-scalability category. The others are rare but significant.
2. Not all requests are created equal. 20 requests a second for the same static HTML file? No problem. But if you have, say, a release page for an open source project with binary download links for all past versions for multiple platforms, each one being a multi megabyte blob, and a scraper starts hitting these links, you will run into bandwidth problems very quickly, unless you live in a utopia where bandwidth is free.
3. You are underestimating the difficulty of caching dynamic pages. Cache invalidation is hard, they say. One notably problematic example is Git blames. So far I am not aware of any existing solution for caching blames, and jury rigging your own will likely not be any easier than the “solution” explored in the TFA.
It's called peering agreements and they are very common. There's a reason social media and sites like YouTube, Twitch, TikTok don't immediately go out of business. The bandwidth is free for most users.
All of this is (and should) cached on a cdn. You can go 1000 QPS on this in that config.
Unless you're running mediawiki.
Are there easy settings I should be messing with for that?
Sure, you could require any number of (user hostile) interactions (logins, captchas, etc) to do expensive operations like that, but now the usability is compromised which sucks.
I'm not against the idea like others here seem to be, I'm more curious about implementing it without harming good actors.
Disallow: /private/beware.zip
But in my experience it isn't the robots.txt violations being so flagrant (half the requests are probably humans who were curious what you're hiding, and most bots written specifically for LLMs don't even check the robots.txt). The real abuse is the crawler that hits an expensive and frequently-changing URL more often than reasonable, and the card-testers hitting payment endpoints, sometimes with excessive chargebacks. And port-scanners, but those are a minor annoyance if your network setup is decent. And email spoofers who bring your server's reputation down if you don't set things up correctly early on and whenever changing hosts.
Do you guys really use these things on real public-facing websites?
Maybe it's time to once again block entire ASN's of ISP's that tolerate being used as residential proxies, and this shit might end.
https://www.trendmicro.com/vinfo/us/security/news/vulnerabil...
Security is hard and there are no easy solutions. People often do not know they have been hacked or even know if a computer on their network has been hacked. Also, it is often not easy to determine if traffic is legitimate, illegal, malicious, or abusive.
it does not work that way. the open internet goal is to be open. "AI" is generally meant to be intelligent, indistinguishable to humans; we created the problem for ourselves, dismissing so many (back then, and now) theoretical problems.
what i fear are increased operating costs for the end users to "prove they are human". what i hope happens is cheaper infra (bandwidth ect) to be able to support the increased demand. and, a big hope that "bots" which are respectful [but this too wont hold up, people are not 100% respectful, why will their bots be?]
so we will be stuck with more captcha related crap, more websites that must use javahelll, even a "trusted browser" (walling out any new browsers, and things like curl), maybe websites that want your ID too (the "think of the children!" folks would be pleased), and ways to "trick" the bots (and not the real users) to ban the bots.... yay
The port list needs to be updated as malware changes it's targets fairly regularly.
It's a small thing, but it's another layer.
# Define a rate limit of 1 request per second every 1m
limit_req_zone $binary_remote_addr zone=krei:10m rate=5r/s;Is it so others can submit issues and pull requests? Do others actually bother to create accounts at your personal git forge so they can contribute?
I could just have a directory in another server, but I like being able to link to a specific file/line/commit, and I prefer viewing the code in the browser in some situations. Forgejo takes the same effort to selfhost as gitweb on my setup, so why not?
I have several remotes for my projects: one on a machine I control, another on GitHub, so I can send links to people etc.
Would it be so crazy to just let it happen? When ordinary users are suffering as much or more than the "criminal" targets, maybe it's not worth it. I understand there are cases where bandwidth usage would become too much, but there's gotta be a better way. Maybe let some bots through so that fewer humans are blocked.
Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself.
I wonder where you ever got the idea that it does.
I solved it by marking that directory read-only. Zip downloads, obviously, won't work. If someone really wants one, they can check out the repository and make it theirself.
If I really cared, of course I'd fix the bug or make sure there's a way to disable the feature properly or only enable it for logged-in users.
Also I server-side redirect certain user-agents to https://git.immibis.com/gptblock.html . This isn't because they waste resources any more but just because I don't like them, what they're doing is worthless anyway, and because I can. If they really want the data of the Git repository they can clone the Git repository instead of scraping it in a stupid way. That was always allowed.
8 requests per second isn't that much unless each request triggers intensive processing and indeed it wasn't a load on my little VPS other than the disk space bug. I blocked them because they're stupid, not because they're a DoS.
Some of us are happy to train AI's but want to block overload. For instance, I'm glad they're scraping pages about the Gospel and Biblical theology. It might help to put anything large that you dont want scraped into specifi directories. Then, upon detecting a bot, block the IP from accessing those.
In my case, I also have a baseline strategy to deal with a large number of requests. That's text only, HTML/CSS presentation, other stuff externally hosted, and BunnyCDN with Perma-Cache ($10/mo + 1 penny / GB). The BunnyCDN requests go to $5/mo. VM's on Digital Ocean. I didnt even notice AI scrapers at first since (a) they didn't affect performance and (b) a month of them changed my balance from $30 to $29.99.
(Note to DO and Bunny team members that may be here: Thanks for your excellent services.)
https://utcc.utoronto.ca/~cks/space/?search=spider
https://utcc.utoronto.ca/~cks/space/?search=crawlers
It's interesting how in 2005 he was advocating for playing nicely, because yes, search engines were hypothetically driving traffic to sites they crawled. And there don't seem to be a lot of complaints about performance hits. Though typically, his comments and observations are restricted to his blog's corner of his department.
For example maybe everything 404s unless you start with a specific unlisted url
I'm thinking more if it's people coming from email or QR code or link from other sites that keep bots out. It can also be more about sequence of accessing pages. The idea is having behavior unlock the site. Because it should be pretty easy to cause people and bots to behave differently.
For example, you could set it up such that access to gitea requires simply accessing the documentation site first.
Things that are needed in Google are documentation or landing sites. Landing on one of those could perform the "knock" to open the domain holding the git. Nobody using Google would notice.
I agree with this. The repo page should just have static HTML pages with information for people new to the project. Dynamic information and especially expensive computations such as git blame should be restricted to logged in users with an account. This would make it easy to ban anyone who was abusing the system or even implement CPU quotas without affecting ordinary users arriving from Google.
No scraping, if I want you to read it I'll send it to you.
sudo fail2ban-client status | sed -n '/Jail list:/{s/^.*://; s/,//g; p}' | xargs -n1 sudo fail2ban-client status | awk '/jail:/{i=$5}; /Total failed:/{jail[i]=$5}; END{for(i in jail) printf("%s: %s\n", i, jail[i])}' | column -t
sshd-ddos: 0
postfix: 583
dovecot: 9690
postfix-sasl: 4227
nginx-botsearch: 1421
nginx-http-auth: 0
postfix-botnet: 5425
sshd: 202157
I'm not banning it out of curiosity, to see if there's a limit to how deep it digs.
1. User-Agent pattern for various known bots. These gets sent to a honeypot (currently implementing zip bombs).
2. Arithmetic captcha page (called "captxa") protecting most of my /cgit tree, which is the primary target for scrapers. Solve a formula, get a persistent cookie that grants you access.
3. Static IP-level bans. The bulk of the unwanted traffic has been from Alibaba Cloud LLC addresses, so I got a list of their blocks and made rules against them all.
4. Log-scan-and-ban system: now mainly just targeting excessive activity over a short period.
About 2: I define the same cookie value given to everyone right inside the Apache httpd.conf. This is passed via en environment variable to the captcha page, a CGI script. When the captcha is solved, and the client has that cookie, it will be the Apache rules themselves which check it. The RewriteRule which redirects to the captcha page is guarded by a RewriteCond which checks the value of %{HTTP_COOKIE} for the presence of the required cookie.
I use the Apache <Macro ...> feature to define the anti-bot rules. Then the rules can be included in any VirtualHost block easily by invoking the macro.
The skeleton of the system looks like this:
Define captxa_secret SecretGoesHere
<Macro nobots>
# Push captxa secret into environment, for the CGI script.
SetEnv CAPTXA_SECRET ${captxa_secret}
# These agents get sent to /honeypot
RewriteCond %{HTTP_USER_AGENT} .*(ezoom|spider|crawler|scan|yandex|coccoc|github|python|amazonbot[Pp]etalbot|ahrefsbot|semrush|anthropic|Facebot|meta|openai|GPT|OPR/|Edg/).* [NC]
RewriteCond %{REQUEST_URI} !honeypot
RewriteRule ^.*$ /honeypot [R,L]
# Clients which have the cookie with the captxa_secret get a pass
RewriteCond %{HTTP_COOKIE} "^(.*; ?)?captxa=${captxa_secret}(;.*)?$"
RewriteRule ^.*$ - [L]
# Rules for no-cookie clients:
# Root level /cgit site: ok for all
RewriteRule ^/cgit/$ - [L]
# First level of /cgit: the project pages: OK
RewriteRule ^/cgit/([^/]+)/?$ - [L]
# Certain project sub-pages like about, spanshot, README: Pass for hassle-free landing.
RewriteRule ^/cgit/([^/]+)/(about|snapshot|refs|tree/RELNOTES|tree/README|tree/README\.md)/?$ - [L]
# Some distros fetch TXR tarball snapshots from CGIT; make it easy for them.
RewriteRule ^/cgit/txr/snapshot/txr-([0-9]+)\.tar\. - [L]
# Everyone else off to captxa script: which will install the cookie
# for those clients who solve it. $0 is the entire URL;
# it gets passed to the script as a parameter, so the
# script can redirect to that URL upon a successful solve.
RewriteRule ^/cgit/?(.*)$ /captxa/?quiz=$0? [R,L,NE,QSA]
</Macro>
RewriteEngine on
Use nobots # include above macro
loloquwowndueo•1d ago
rovr138•1d ago
nulbyte•20h ago
xena•17h ago