The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.
Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.
Bender•1h ago
eatonphil•56m ago
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
Bender•45m ago
eatonphil•36m ago
Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.
I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.
All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.
edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.
edit 2:
> tiny percentage of bots will still get through but the load becomes much more manageable.
The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.
Bender•27m ago
I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.
Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]
[1] - https://altchan.org/sh/33
P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.
ljlolel•45m ago
i heard hcaptcha is good
j45•30m ago
sixtyj•18m ago
But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).
Bender•10m ago
Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.
I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...