Results are certainly thinner than in years past when you could seemingly inspect the entire crawled corpus. You can search for a pretty broad topic and hit the wall pretty fast today. I think they are limiting the depth of queries these days probably owing to search volume and the size of cache they can sustain what with current webdev standards. It was a different story when websites were a few kb to mb 20 years ago even though storage is "cheaper" today.

It is not crawling but indexing is the problem. Google has over years learned the patterns and authority of different articles. It will be hard for others to replicate but not impossible.

What Google should do is offer API based access to these providers but a lot of these providers might no adhere by contracts. So there is that.

Crawling isn't complicated. But ranking? That was Google's reason for existence for a very long time. Remains to be seen if AI companies will be able to replicate that.

> used their own crawlers and indexes

Not yet, but will eventually for sure.

If you're an expert who has worked on Google search or something like that, this would b great time to start a company for this.

Or OAI-SearchBot for "web search" feature to augment queries and GPTBot for training?

I swear I've even read how aggressive GPTBot is. Surely they aren't just googling stuff?

https://platform.openai.com/docs/bots