LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Show it some examples from your access logs. Enable logging of ssl curves, ciphers, etc...
eatonphil•9m ago
I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.
Bender•27m ago
eatonphil•9m ago