It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).

There is no detection method that will prevent AI from accessing systems without also blocking humans. The only thing we can do at this point is throttling.

I'm becoming more and more convinced that a big source of outrage on the internet is caused by people assuming that all other people are a homogenous blob.

It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.

Just simply a separate Chromium binary (not your usual browser).

> the only solution is to drive your regular browser with all your sessions/cookies via an extension

This is exactly what I'm doing, but mocking/randomizing all the sessions/cookies/params (like resolution, OS, WebGL , etc.) in a separate Chromium binary. It's popular these days, but imo using your normal browser for agentic stuff is a very bad idea. These models do dumb stuff all the time.

no it doesn't fail. Agents are very good at comparing traffic characteristics from a real browser and a headless/automation browser and getting it to behave in the same manner. It's a cat and mouse game for sites stopping unauthorized access but right now llm agents are ahead.

For testing proxies should be used to avoid IP ban issues but given enough time modern agents can figure out how to bypass most of the modern scraping/automation prevention mechanisms.

I have built and used a lot of different automations and web scraping implementations for my business and it's never got permanently stuck yet, some take a bit longer, some shorter, but all within a reasonable time with little external help they have succeeded in their tasks.

Yep, it's super similar to this! I think his is a bit overengineered, but tbh I haven't built Firefox plugins in forever so that might be the way you have to do stuff on FF. All you need is the websocket plugin to have "full access" to all visited webpages, and then your driver just hooks into the DOM like normal (with the extra ws layer—and even the websocket layer can be simplified away I think).

It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).

Are you saying they're blocking the `User-Agent`? Because that's trivial to bypass. I'm not exactly sure what you mean by "Chromium" because that's a whole class of browsers (Edge, Brave, Chrome, etc. are all forked "Chromium" browsers).

How can it do this? Wouldn't you see a hundred fetch requests in your browser network tab?

That doesn’t track with reality as far as I can tell.

> Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised.

Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.

I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.

[0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.

Or the captchas will evolve.