mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.

The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

There is a large community of sites feeding poison to scrapers.

The poison gets better every day, and the community gets larger.

For example, here is the outpost on Reddit: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.

I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.