Querying an LLM to output its confidence in its output is a misguided pattern despite being commonly applied by many. LLMs are not good at classification tasks as the author states. They can "do" it, yes. Perhaps better than random sampling can, but random sampling can "do" it as well. Don't get too tied to that example. The idea here is that if you are okay with something getting the answer wrong every so often, LLMs might be your solve, but this is a post about conforming non-deterministic AI into classical systems. Are you okay if your agentic agent picks the red tool instead of the blue tool 1%, 10%, etc of the time? If so, you're never not going to be wrangling, and that's the reality often left unspoken when integrating these tools.
While tangential to this article, I believe its worth stating that when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff.
For some, there will be inevitable dissonance reading this, but consider that these experiments are local examples. Its lack of robustness will become apparent with large scale testing. The data spaces these models have been trained on are unfathomably large in both quantity and depth, but under/over sampling bias will be ever present (just to name one).
Consider the the following thought experiment: You are an applicant for a job submitting your resume with knowledge it will be fed into an LLM. Let's confine your goal into something very simple. Make it say something. Let's oversimplify for the sake of the example and say complete words are tokens. Consider "collocations". [Bated] breath, [batten] down, [diametrically] opposed, [inclement] weather, [hermetically] sealed. Extend this to contexts. [Oligarchy] government, [Chromosome] biology, [Paradigm] technology, [Decimate] to kill. With this in mind, consider how each word of your resume "steers" the model's subsequent response, and consider how the data each model is trained on can subtly influence its response.
Now let's bring it home and tie the thought experiment into confidence scoring in responses. Let's say its reasonable to assume that the results of low accuracy/low confidence models are less commonly found on the internet than higher performing ones. If that can be entertained, extend the argument to confidence responses. Maybe the term "JSON" or any other term used in the model input is associated with high confidences.
Alright, wrapping it up. The end point here is that the model output provided confidence value is not the likelihood of the answer provided in the response but rather the most likely value following the stream of tokens in the combined input and output. The real sampled confidence values exist closer to code, but they are limited to each token. Not series of tokens.
100% this.
Idk about the far-out takes where "AI is an alien lifeform arrived into our present", but the first thing we know about how humans relate to extraterrestrials is: "I want to believe".
Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
Similarly you could not allow re-ordering of operations and similar - so the results are guaranteed to be deterministic (even if still "not correct" compared to infinite precision arithmetic) - but that would also have a big performance cost.
Technically possible, but I think unlikely to happen in practice.
On the higher level, these large models are sequential and there’s nothing to parallelize. The inference is a continuous chain of data dependencies between temporary tensors which makes it impossible to compute different steps in parallel.
On the lower level, each step is a computationally expensive operation on a large tensor/matrix. These tensors are often millions of numbers, the problem is very parallelizable, and the tactics to do that efficiently are well researched because matrix linear algebra is in wide use for decades. However, it’s both complicated and slow to implement fine grained parallelism like “adding together whichever two finish first” on modern GPUs. Just too much synchronization, when total count of active threads is many thousands, too expensive. Instead, operations like matrix multiplications are often assigning 1 thread per output element or fixed count of output elements, and reduction like softmax or vector dot product are using a series of exponentially decreasing reduction steps, i.e. order is deterministic.
However, that order may change with even minor update of any parts of the software, including opaque pieces at the low level like GPU drivers and firmware. Library developers are updating GPU kernels, drivers, firmware and OS kernels collectively implementing scheduler which assigns work to cores, both may affect order of these arithmetic operations.
I don’t think the issue is determinism per se but chaotic predictions that are difficult to rely on.
[1] https://docs.nvidia.com/cuda/cublas/index.html#results-repro...
https://news.ycombinator.com/item?id=37006224
https://news.ycombinator.com/item?id=45200925
The TL;DR is that LLMs are often not deterministic because GPUs compute submatrices in parallel and sum them up in different orders, depending on which finish first. This is maybe a few percent faster than always using the same order, but it absolutely could be made deterministic if people cared enough. CUDA even provides deterministic primitives if desired. Of course also use the same random seed for samplers, but that is trivial.
https://github.com/outlines-dev/outlines
https://github.com/jxnl/instructor
https://github.com/guardrails-ai/guardrails
https://www.askmarvin.ai/docs/text/transformation/
Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).
By they way you don' need to use library to have output schemas: https://platform.openai.com/docs/guides/structured-outputs , https://platform.claude.com/docs/en/build-with-claude/struct...
Calling them structured output is a lie, it's structured validation.
If the LLM persists in generating bad JSON, those are useless.
https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
To produce an application you should have some good unit-tests. So AI produces some unit-tests for us. Then we ask it again, and the unit-tests will be different. Where does that leave us? Can we be confident that the generated unit-tests are in some sense "correct"? How can they be correct if they are different every time we ask??
That’s why you’re using an LLM in the first place.
ironbound•3w ago
dang•3w ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html