I thought this was a very good read about the many of the issues that are faced without having any ground truth to reason against. It is interesting how many different ways people have developed to work around missing information, and the marginal improvements it makes in some benchmarks.

This is great. We need more research into solving this fundamental problem, yet AI companies prefer to chase benchmarks and pump out value-added products.

The RAG-based mitigation is interesting, but quite limited, as mentioned. It would only work if the user can provide ground truth data, which for code generation is relatively straightforward, but it's much more difficult for most other factual information. We can't directly rely on data from the web, since the sources need to be carefully reviewed by a human first, which is the labor-intensive task that requires human domain experts.

So this approach seems like a band-aid, and wouldn't be generally applicable. I'm not in the AI industry, but from the perspective of a user it seems that the hallucination problem requires a much more foundational solution.

On the other hand, I think it's cute when LLMs hallucinate a Python library that doesn't exist, because it probably means it's worth creating into existence.

My favorite is trying to use it to generate an IAM policy and keys are just hallucinated based on expectations of what the keys would be called and are either wrong or they flat out don't exist if you are dealing with more advanced conditions.