it could be the "grounding with google search" feature when using Gemini models

Google does crawl with a different crawler. Makes no sense other than clearly two different teams with different goals.

The assumption is that "Google" is a good player, not a malicious one. Same idea as robots.txt. It's only there to configure the "good guys" behavior not to prevent malicious actors.

So I'm assuming Cloudflare is basically asking Google to split its crawler's UA and distinguish between search and AI overview and respect something akin ot robots.txt

cloudflare wants to sell a "AI bots have to pay to access your site" service, so they need to argue it out with google one way or another

the reality of the tech is irrelevant

Google does not use a separate crawler, but they do use the robots.txt semantics to control what's allowed in the AI "index" separately from the search index. From the docs:

> Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.

https://developers.google.com/search/docs/crawling-indexing/...

Google has made it very difficult to completely block their AI crawling by also using the standard googlebot search crawlers to feed data into their AI overviews and other AI features within Google search. Google says there is a workaround but it also blocks your site from fully indexing in Google search. This also also all covered in the article though.

> What does an AI crawler do different from a search engine (indexing) crawler?

Many people don't want the extra bot traffic hitting their site that comes from AI, especially when AI chat & AI overviews in Google provide such a small amount of traffic in return and that traffic pretty much always has horrendous conversion rates (personally seen across multiple industries).

Do you have any more doubts who run the US and makes laws there?

Tech regulation, or lack thereof, tends to be "biggest pile of money wins", but in this case there's already large anti-Google and anti-AI constituencies which CF may be able to mobilize. Especially in the EU.

"Always has been"

What has been democratic about how the internet has evolved over the last 2 decades? Because as far as I can see, the internet has undergone a massive centralization into the hands of a few players with practically no regulation. Especially Google, which can make decisions such as adding AI Overviews to search results leading to millions of websites seeing a ~25% drop in organic traffic in the last few months.

Maybe they're vibe-coding the scrapers.

I don't understand it either. I track requests and AI crawlers are there but not as abusive as people claim. Most annoying requests are from hackers who are trying to find my ".git" directory. But I highly doubt these guys will respect any rules anyway.

I have a symbol server hosting a few thousand PDBs for a foss package.

Amazonbot is every day trying to scan every single PDB directory listed. For no real reason. This is something causing 10k+ requests each day when legitimate traffic sits at maybe 50 requests a day.

If you run a website, you'll realize it's very difficult to get human traffic. Worse, trying to understand what those eyeballs are doing is a swamp; there are legitimate privacy concerns for example. Maybe all you care about is if your articles about sewing machines are getting more traction than your articles about computing Pi, but you can't get that without navigating all the legal complications of your analytics platform of choice, who wants to make sure you suffer for not letting them collect private information on your visitors to sell to third parties and to dump ads onto your visitors. Were it not for the bots, you would be fine just by running grep on your access logs. But no, bot traffic leaves noise everywhere; and for small websites that noise is more than enough to bury the signal and to be most of the traffic bill.

> I don't understand how AI scrapers make up such a large percentage of traffic to websites, as people claim it does.

I think a lot of people confuse scraping for training with on-demand scraping for "agentic use" / "deep research", etc. Today I was testing the new GLM-experimental model, on their demo site. It had "web search", so I enabled that and asked it for something I have recently researched myself for work. It gave me a good overall list of agentic frameworks, after some google searching and "crawling" ~6 sites it found.

As a second message I asked for a list of repo links, how many stars each repo has, and general repo activity. It went on and "crawled" each of the 10 repos on github, couldn't read the stars, but then searched and found a site that reports that, and it "crawled" that site 10 times for each framework.

All in all, my 2 message chat session performed ~ 5-6 searches and 20-30 page "crawls". Imagine what they do when traffic increases. Now multiply that for every "deep research" provider (perplexity, goog, oai, anthropic, etc etc). Now think how many "vibe-coded" projects like this exist. And how many are poorly coded and re-crawl each link every time...

Your speculation assumes a low page count.

Pages are dynamic, they change often: if a page is worth scraping once, it is worth scraping again and again and again to keep up to date with any changes.

Post a URL to a page on your website on something like Mastodon and tail your logs.

On multiple client sites that have > 1 million unique real visitors per month, we are seeing some days where ~25-30% of requests is from AI crawlers. Thankfully we block almost all of this traffic. But it is a huge pain because it add an addition load to your server and messes up your analytics data for what is a terrible return - traffic from AI sources has a horrendous conversion rate, even worse than social media traffic conversion rates.

Ah ok, yet the megacorps continue to reap profits off stealing labor and work from the peasants.

I propose we just remove the charades and require us peasants pay these megacorps 20% of our annual income yearly at tax time.

The thing that he doesn't mention is that as soon as they do something legislatively and announce routes there, etc.....well Google just won't crawl those sites. It turns into a game of whether you would like 0 traffic from Google, or allow them to use your content both for search results and AI summaries.

Google is the bringer of traffic and if you want it, then you play by their rules. I don't like that the web is in that position, but here we are.

After 50 years of effectively zero activity, we had some glimmers of anti-trust enforcement under the Biden admin. But then eggs were expensive in the summer of 2024, so we decided it was actually no problem for these half-dozen companies to control our speech and economy, and here we are.

It used to be just Ma Bell

> we shouldn't have gifted all of our digital infrastructure to just 2 companies

We didn't. Just as we didn't gift all our chocolate-making infrastructure to Hershey's and Cadbury's.

There is nothing stopping other CDN/DNS providers from implementing similar services and tools to what Cloudflare offers. Part of the reason CF has become so popular is because so many of their competitors don't offer nearly the same convenience for routine tasks & protection.

It's a cost thing. It costs more to render a website than it does to consume it. When you have some bot traffic mixed in with human traffic, that is fine.

When you have egregious bot traffic, say 10k requests per minute sustained load, it becomes a real problem for webmasters.

The whole point of publication is so that humans can read it. Robots not so much, especially if they're not paying customers. This is the distinction between how the web works technically and how it works socioeconomically.

This is the next iteration of things like the news snippet case. Publishers are not happy that Google crawls their content (at their expense) and then republishes it on their own site, while serving ads around it and getting user data, without cutting in the publisher who originally made it. And, for what little it's worth, owns the copyright.

HTML was not made for robots to read (a semantic web or an internet of data), it just so happens that crawlers try to index things in meaningful ways. It's an un-ordered blob of unstructured data.

Have you ever been responsible for the performance and security of a publicly accessible web server? I'll accept robots indexing my content if they play nicely. Unfortunately most do not, even from major vendors.

A lot of people don't want AI slop and don't want the companies pushing it to crawl their websites.

That's why we need the Captchaweb. It's like the web, but everything is in captcha text.

Having perplexity or other AI bots go haywire and sending 10s of thousands of requests per minute to your website (despite you having a robots.txt blocking them) is a giant pain in the ass. Not only does your server costs go up, but your analytics and attribution reports start to look messed up because of all the bot traffic.

Because you remain at the top of an extremely shallow perspective. The fact that robots are a part is not relevant, the relevant issue is how some of those robots behave, and what the consequences of that behavior are.

Machine-readable does not mean centralized.

You can use the element picker in uBlock Origins.

I come to the exact opposite conclusion. A few large companies scraping and repurposing original content from publishers kills original content on the internet because it takes out the ability to earn a livelihood from creating original content or running your own ecommerce store that is not tied to a mega-company's platform.

Probably because that's one of the only things you can say to Google these days that will escalate it from an low level support agent on the other side of the world.