We built an open source library for closed book hallucination detection called UQLM (Uncertainty Quantification for Language Models).
It packages over 20 types of scorers from the latest uncertainty quantification literature in one API so that one can calculate a confidence score (between 0 and 1) of an LLMs response without reference to external knowledge. It accepts any LangChain BaseChatModel which wraps every major provider.
UQLM has six families of scorers:
1. Black-box scorers: calculate the consistency across multiple responses to the same prompt
2. White-box scorers: use log-probabilities from the model’s output to estimate confidence
3. LLM-as-a-judge scorers: use a single judge or panel of judges
4. Long-text scorers: work like black box scorers but first decompose the claim into sentences or claims. They also have uncertainty aware decoding which drops low confidence claims / sentences and reconstructs the output.
5. Code generation scorers: calculate functional correctness of a piece of code without running it by using functional equivalence methods
6. Ensemble scorers: calculate a weighted combination of other scorers that can be tuned
A canonical use case for UQLM is to use one or many of the scorers to flag low confidence responses from an LLM in production for human review instead of sending them to the user.
We’d love to hear your feedback about the library’s methods and its use in production.
virenbajaj•1h ago
We built an open source library for closed book hallucination detection called UQLM (Uncertainty Quantification for Language Models).
It packages over 20 types of scorers from the latest uncertainty quantification literature in one API so that one can calculate a confidence score (between 0 and 1) of an LLMs response without reference to external knowledge. It accepts any LangChain BaseChatModel which wraps every major provider.
UQLM has six families of scorers: 1. Black-box scorers: calculate the consistency across multiple responses to the same prompt 2. White-box scorers: use log-probabilities from the model’s output to estimate confidence 3. LLM-as-a-judge scorers: use a single judge or panel of judges 4. Long-text scorers: work like black box scorers but first decompose the claim into sentences or claims. They also have uncertainty aware decoding which drops low confidence claims / sentences and reconstructs the output. 5. Code generation scorers: calculate functional correctness of a piece of code without running it by using functional equivalence methods 6. Ensemble scorers: calculate a weighted combination of other scorers that can be tuned
A canonical use case for UQLM is to use one or many of the scorers to flag low confidence responses from an LLM in production for human review instead of sending them to the user.
We’d love to hear your feedback about the library’s methods and its use in production.