It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a legal or medical opinion supported by research.

Are we saying that it's now a problem that we're not getting scraped?

This appears to be a new generation of "SEO", marking itself as a service for getting into AI results?

This is not the future I want to be a part of.

Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever increasing X, to the detriment of users.

Makes me wonder if paid Medium and paid news is input to AI training. Surely, they are.

> 94.8%

This seems consistent with Sturgeon's Law.

We don't have a page rank mechanism. Most likely people are paying or threatening AI companies to boost certain sources as authoritative

Call it web 4.0

This is a weird complaint.

Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia.

There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site?

Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most sites deserve links because they were crawled seems like a statistical fallacy. You could say the same thing about the percent of sites crawled by Google versus ever showing up on first page of results.