Set your site to 401 unauthorized with a basic challenge if an auth header isn’t sent, and set the auth description to “Enter anything to proceed. All human access is authorized. Unauthorized non-human access is prohibited.”. Crawlers can’t parse the instructions and will deadstop on them, while people will shrug and enter any password, which will work.

Anubis is also viable and popular, but it lacks the legal threat to AI of being able to file a federal hacking claim against a scraper’s unauthorized intrusion if they code their scraper to transmit an empty/invalid/valid authentication header.

They're more of a speed bump. Deter drive-by scraping and whatever they call script kiddies these days.

Haven’t for a while. There’s a bunch of open source projects that provide a Captcha and Cloudflare bypass proxy where you just point your scraping through the proxy and it takes care of the challenges. It’s rather trivial to handle nowadays.

A bunch of the torrent trackers are now behind Cloudflare so the pirate community has been maintaining many of these projects in order to enable their autodownloaders like Sonarr, Radarr, Lidarr, etc.

There is a product category around this, e.g.:

https://docs.browserbase.com/features/stealth-mode#captcha-s...

Personally, I see this as ToS violation-as-a-service, but I guess ethics left the AI stack space a while ago.

Without authors being paid to create new content AI is nothing/it's irrelevant!

Either some startup needs to come up with compensation for authors or the big players need to set up a system that still gets authors paid as Im guessing in five to ten years we are not visiting websites. Our soon to be AI friend (Facetime the "friend," or just talk or text it) seen on our lockscreens or in a hologram is visiting all sites to create visuals of the info and displaying/discussing it with us immediately upon request.

You may be right that resistance is futile. But I don't think adaptation is. And there's no reason our adaptation can't be rooted in resistance.

For example, maybe smaller local forums make a comeback and their communities decide to hide all threads behind auth. (I don't necessarily see that happening, just an example.)

And honestly given the stronger feelings people are developing against what I'll just call "creative use of generative AI", I'm starting to think maybe resistance isn't futile... Poisoning original digital art so it's less useful for image generators...social shaming of AI generated music being laundered on platforms...those things do feel like meaningful steps towards resistance.

Printed media, radio and television are still very much alive and well, just not as big as they once were. We've past peak Web but it's not going anywhere anytime soon.

We do need new protocols though.

As a social network the web is collapsing but it is entirely possible that a new kind of internet could emerge in its place. After all, routing around damage is part of the essence of the Internet.

Only that there is no way back. Those physical magazines would contain the same AI generated content.

It is not the medium that is dying, rather it is the content that is dying. People are saying the web is dying because there's no longer financial incentive for actual humans to create content (there may be other incentives, and those types of forms may flourish). That goes equally for other media as well: print, digital, it doesn't matter.

One of the reasons people prefer AI to visiting the source websites is because the source websites have so often made it such an unpleasant experience, making you jump through hoops and navigate a maze of dark patterns. Meanwhile, AI gives you what you want without all those roadblocks.

This is like Napster vs iTunes all over again. People started paying for media online as soon as it became convenient to do so. You make things inconvenient for people, you’ll lose out to whatever the more convenient option is.

This might get worse.

The free open web was economically dependent on reaching large audiences by search and social, and supported by advertising. Without that scale, you're looking at more paywalls and membership programs. The smaller the niche, the more expensive and tighter the program needs to be.

A lot of that gating went into effect after social media start suppressing links.

If you believe in making quality products available to mass audiences—information wants to be free and all that—this is a problem.

oh okay so it's fine for a multinational conglomerate to steal flagrantly and profit at the various stages of theft they themselves enable.

Excessive pragmatism leads to utilitarianism which is anti-human.

Are you saying that now, because of all the AI generated content online, you're suddenly willing to pay for sites like Wired, WSJ, FT, NatGeo, NYT, or The Economist? These have been around for ages. Why now, and not before?

The lack of quality is what gets to me. I've used AI tools in many aspects of my life to great benefit. Yet, nowadays scrolling through Reddit, X, or even video based platforms are a deluge of drivel. It was bad enough that I was spending too much time on my phone instead of interacting with other people but now even the content I'm interacting with isn't human!