The reason why LLMs cannot be purely* pre-trained on LLM outputs is because LLM output diversity is low.

This implies two things:

1) There is going be no functional difference between hypothetical Qwen 5.5 and Fable 5.5

2) LLMs are not AGI

I was thinking just today about how, if I could hire my competition's lead designer/programmers, have them design a system for me cleanroom, that I'd effectively know everything there is to know about my competition's limitations. This is trivial to do with LLMs.

"Make me a thing that does X" is going to come out implemented basically the same way most of the time, as a kind of stack-overflow, well commented, internship project. (or, if undirected enough, as a collection of those based on a bog-standard architecture)

Same-y enough that you could train an adversary on it? Or hack it afterward? I am beginning to think so!

I don't see why you couldn't do the same in real projects.

If the design looks too generic, you can just ask for a change, no?

After all if you ask 100 people to draw a horse, it is quite likely it will be a lot more similar to a "here's a stock horse photograph" than a "here's what picasso would draw when asked to draw a horse"