Sometimes it can load it even when it has the "checking if you're a malicious bot". I'm wondering how that works. Isn't AI scraper traffic literally the thing that page is designed to stop?
Do they let Google, OpenAI etc. user agents through? Or do those companies just have a bunch of guys in a warehouse somewhere with Chrome open, forwarding the AI's HTTP requests?