Hi HN!

I’m excited to share this lightweight hallucination detector I built to help identify unreliable or “hallucinated” outputs from LLMs like GPT, Claude, and various local models.

It uses multiple methods — from spotting overconfidence and contradictions to scoring factual density and coherence — to give a hallucination probability score for any generated response.

It’s framework-agnostic, fast (~0.2s per response), and designed for both research and production use. Plus, it’s completely free under the MIT license.

I’d love to hear your thoughts, feedback, and if you find it useful for your projects. Happy to answer questions or discuss how it works under the hood!

Thanks for checking it out!

I'm impressed that you give precision and recall metrics for this -- and amazed that they are non-terrible. I'm amazed because a fully general hallucination detector is obviously a truth oracle -- it can answer any question about anything in the world, by framing the question as a statement and then asking whether that statement is a hallucination.

From among the analyses the tool makes, it makes sense to me that contradictions can be detected, since that doesn't require knowledge of the real world. I'm very interested in how you do this detection ("Logical inconsistencies") in practice. Likewise for "Logical progression".

Two questions:

1. Since "overconfidence" is treated as a red flag, won't applying your tool as a filter cause LLM response precision to drop, often unnecessarily? The safest answer an LLM can give to "When was the Eiffel Tower built?" is surely along the lines of "The Eiffel Tower may or may not have been built at some time in the past."

2. I don't see how this tool can detect the kind of hallucination that (a) involves no contradiction and (b) requires knowledge of the world. These come up often. Examples: Citing plausible-sounding but nonexistent court cases, calling plausible-sounding but nonexistent methods in an API.