Among other reasons, if you turn temperature down to 0, llms stop working. Like they don't give natural language answers to natural language questions any more, they just halt immediately. Temperature gives the model wiggle room to emit something plausible sounding rather than clam up when presented an input that wasn't verbatim in the training data (such as the system prompt).

unfortunately disabling temperature / switching to greedy sampling doesn't necessarily make most LLM inference engines _fully_ deterministic as parallelism and batching can result in floating point error accumulating differently from run to run - it's possible to make them deterministic but does come with a perf hit

some providers _do_ let you set the temperature, including to "zero", but most will not take the perf hit to offer true determinism

In fact, nondeterminism goes all the way down to the training level where every token and weight starts as a random number, a concept I wrote about in this intro to how AI works article [0].

The reason for this is that randomness is crucial to getting emergent data out of a system. Those are unexpected, unpredictable, but often useful results. This is how an LLM can answer a question it's never been asked before.

We have had deterministic databases forever so there would be no AI advance if LLMs were just databases. The AI models of the 1960's tried that very approach, called rules based and it doesn't work. We can never come up with or write down all the rules. The failure of those methods lead to the "AI winter" and no further real progress in AI until the invention of transformers at Google.

[0] https://levelup.gitconnected.com/something-from-nothing-d755...