A nice illustration of the homogeneity of LLM responses. Another way to describe this effect would be…

If you ask humans to write 1,000 books, you're asking 1,000 different humans with different experiences and different skills and different moods (etc.) to write those books.

But if you ask LLMs to write 1,000 books, you're probably only talking to 3 or 5 different models, tops. And they've all trained on the same or similar data, and are trained to respond in very similar ways.

The LLMs don't differ much in anything like "life experience" or "skills", and they don't really have anything like a "mood" independent of the prompts you've given them.

When you generate one or two blog posts with LLM they look pretty good. And you will be impressed with that one clever bit it adds that you didn't even ask for. But then you generate 50 of them and they all converge into the same pattern. It's hard to prove that an article is AI generated but they are instantly recognizable.

An aside, I usually take my written blog posts through a pass on Notebooklm to generate a podcast like discussion about it. It used to be a good way to extract some insights I haven't thought of. But after 50 of them, I can predict what the host will "pushback" on and exactly when. Then they magically resolve their differences and agree with whatever the idea was. It's truly impressive when you just consume sporadically. But listen frequently and they converge into one blob.

There used to be a word for this in generative AI: mode collapse. It's not that the model doesn't generate human-like responses, it's that it generates the same 0.0001% of possible human like responses every time. It's almost certainly the instruction tuning which is responsible, maybe some small part of blame could go to the rollout policy (I have no idea how rollout policy works these days).

Maybe the LLMs need some kind of "coverage" metric so they prioritise new paths?

I hate to call this out, but "it’s not this — it’s that". It's an em-dash, a distinctive feature of LLM generated content.

I want to err on the side that the author wrote this piece, but that dash is suggestive.

We likes this "same, complex set of mannerism" when using LLM for programming. If you ask LLM to write a certain function for you, it gives you statistically obvious implementation. But maybe for writing an original book, this feature is not so desirable

I don't know how much of a smoking gun this actually is, the evidence proffered doesn't establish anything - I can see some names there like Havilah Brooks or Celina Briar who are intentionally re-using the same title to create a series, for example. And this doesn't really get into the base rate of generic title re-use among encyclopedias. There isn't much reward for coming up with an imaginative title for kids, they're not very experienced. I'd have no trouble believing publishers come up with very similarly titled books in the kids encyclopedia all the time, they already recycle plots like there is no yesterday in fantasy.

I think the article's point is probably sound to some great extent, but I would believe I owned a book with a title like "100,000 Whys" when I was young. With a dinosaur and a rocket on the front. I loved dinosaurs and rockets, they're even still cool today.

Notably, in programming this is actually a desirable feature for most problems. Even human programmers are taught to produce predictable and obvious code whenever possible. I wonder is ultimately this is an artifact of optimizing the models for code, that they become less creative.