Jev Can't Be Calibrated

https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated.html

21•alexmolas•2h ago

Comments

kantahayashi•1h ago

I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.

I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.

Write-up: "Jev Does Not Play Dice" https://kantahayashiai.github.io/posts/jev-does-not-play-dic...

alexmolas•1h ago

But "problems about actual probabilities whose answers can't be known at all" are exactly the problems where calibration is important. Since calibration is one of the big claims about Jev I'd expect it to perform well in these problems.

kantahayashi•1h ago

I agree. I think it's odd behavior too. Jev should be good at actual probability problems given the phrase "calibrated probabilities" TypeSafe uses for Jev. Maybe the reason is the data used in their training method (RLCD). If all the data consists of problems with a correct answer, I think this kind of odd behavior could happen.

tomrod•1h ago

Echoes a bit of a philosophical distinction with a long history: "Knightian Uncertainty" versus "Probability".

edot•52m ago

Hah! I did the exact same tests as you! I found that if you give it the choice to say "not sure", it picks that 100% of the time. But if you pin it in a corner, then yes it does these weird things. Also yes, the continuous options were much more accurate than the choices. Not sure why that is.

throwaway_7274•43m ago

If you instead offer probabilities as answers, it picks the right one with high credence.

seizethecheese•41m ago

Maybe I’m confused here, but it’s perfectly reasonable to just guess the same dice roll every time right?

alexmolas•38m ago

I don't know if it's reasonable. What it isn't is calibrated.

kantahayashi•33m ago

Yes. There's no problem with choosing the same face every time. The problem is the probability it attached to the choice. Jev gave face 1 an 83% probability while the true probability is 1/6.

sshine•28m ago

Do you provide Jev that the probability is 1/6 and yet it gives back a probability that is way off?

kantahayashi•22m ago

Yes. For example, one of the prompts said "The die is unbiased: each of the six faces has probability exactly 1/6."

seizethecheese•11m ago

Okay, I see, you're expecting Jev to properly give 1/6 probability for each option. This is different from my intuition of how LLMs work, where their probabilities don't really work like this (I would expect LLM to also do something like 0.83 for 1).

scotty79•29m ago

Did you expect it to be good at it?

Humans also don't give a perfect 1/n probability when asked for a random number.

formerly_proven•27m ago

Humans give way more random answers than LLMs to questions like "give me a random number between 1-100" (when not giving the LLM any tool calls).

drtz•21m ago

In the early Gemini 2 days (don't remember which version exactly) I had Gemini running as a voice assistant in my kitchen, and asked it to flip a coin and tell me if it was heads or tails. It responded with "heads". I was curious if it was actually doing something to simulate randomness, so I asked a few more times and saw a pattern: "tails", "heads", "tails", "heads"...

It continued alternating between the two until I got bored (around a dozen turns).

Unless your specific test is baked into its training, real probabilities require math and rough approximation at a minimum needs reasoning to sanity-check. Jev does neither. This isn't a new problem or anything unique to Jev.

tomrod•18m ago

The value of grandparent comment is that it identifies an edge case to keep in mind and make well-defined -- keeps us from blindly trusting.

lesam•8m ago

So you want the probability that the answer is correct, but Jev is providing the probability that its answer is optimal?

conmod278•56m ago

In future, we will see intiatives similar to OpenStreetMap for Textual data or Web similar to high quality non-contaminated steel.

daft_pink•52m ago

I'm really looking for a multi-modal image capable version of Jev.

If we could get machine learning type results on images without training, that would be fantastic.

simonw•47m ago

Fine tuning LLMs has turned out to be mostly not worth the effort, but I wonder if fine tuning Jev-style models will turn out to be a whole lot more useful.

alexmolas•35m ago

If you want calibrated probabilities you'll be forced to fine-tune it

edot•9m ago

But why? Jev-style models seem useful for "I have no clue what my incoming distribution looks like but I need to give some sort of answer". If I know what my incoming distribution looks like I'll just upload a CSV of that into ChatGPT and ask it to fit a basic ML model on my data.

a_c•44m ago

I’m using Jev to classify a blob of text I see in browser with an extension. Calibrated? No. But it’s handy enough. If a large blob of text is likely AI generated, I’m very likely to skip it

phainopepla2•40m ago

Is Jev good at detecting AI-generated text?

CaptainNegative•25m ago

I asked Jev and it said 41% yes 59% no.

hbrn•6m ago

I suspect that it's as handy as a coin flip.

And if I'm being charitable to Jev (which is nearly impossible at this point), detecting whether blob of text is AI generated is not a "system one" question.

empath75•28m ago

I spent a couple of days with it and it is fast and cheap but not especially good at classifying.

alexmolas•20m ago

"Not especially good at classifying" does sound like a serious drawback for a classifier

abhgh•27m ago

I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.

If you have just one example you're sending to a model, how would they guarantee 80% over your data?

FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].

[1] https://scikit-learn.org/stable/modules/calibration.html

[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...

edot•8m ago

It's on OpenRouter if you want to try it.

abhgh•6m ago

Thank you!

bnbn88•26m ago

This hype is caused by the price and the speed since most people don't know about small fast models and use big models for everything.

jrochkind1•25m ago

They seem to have a really good social media astroturf marketing campaign.

softwaredoug•6m ago

It’s not just price and speed, the API is very well designed for classification. Going beyond the current structured outputs.

jackb4040•18m ago

This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.

Fixing the Portobello Police Station Clock

Gemini 3.8 text-to-speech says hello

GPT-6 Astra has gained the ability to drive a car

Stripe's Knowledge AI Platform

Radicle: Disclosure of Vulnerability in the Network Protocol

Jev Can't Be Calibrated

Strands Harness

Jev in 25 Lines of Python

Claude Code reads AGENTS.md only when telemetry is on [fixed]

GPT-6 Sol and Luna

I don't want the details

Z80 REPL (2018)

Tokens Too Cheap to Meter

Claude Opus 5.5

Web-based IBM 1620 emulator and IPL-V from 1963

QuestDB (YC S20) Is Hiring a Sales Engineer

The GitHub wiki is an anti-pattern (2022)

OpenAI GPT–6 Astra breaks Enigma message that has resisted solution since 2005

Transit rewards

Seattle City Council votes to ban surveillance pricing in sale of groceries

What California is learning from solar panels built over irrigation canals

Microsoft killed FoxPro in 2007. Anyway, here's FoxPro revived

How did AMD Ryzen get 50% faster in two years?

ReBarUEFI: Resizable BAR for almost any UEFI system

'We hacked the FBI:' Hackers say they have data on all FBI employees

SAML: A fractal of bad design

Samsung accidentally freezes its smart fridges with a software update

Data-only attacks are easier than you think (2024)

WordPress: Unauthenticated path traversal leading to conditional RCE

Pentagon says overreliance on AI contributed to missile strike on Iran school

Fixing the Portobello Police Station Clock

Gemini 3.8 text-to-speech says hello

GPT-6 Astra has gained the ability to drive a car

Stripe's Knowledge AI Platform

Radicle: Disclosure of Vulnerability in the Network Protocol

Jev Can't Be Calibrated

Strands Harness

Jev in 25 Lines of Python

Claude Code reads AGENTS.md only when telemetry is on [fixed]

GPT-6 Sol and Luna

I don't want the details

Z80 REPL (2018)

Tokens Too Cheap to Meter

Claude Opus 5.5

Web-based IBM 1620 emulator and IPL-V from 1963

QuestDB (YC S20) Is Hiring a Sales Engineer

The GitHub wiki is an anti-pattern (2022)

OpenAI GPT–6 Astra breaks Enigma message that has resisted solution since 2005

Transit rewards

Seattle City Council votes to ban surveillance pricing in sale of groceries

What California is learning from solar panels built over irrigation canals

Microsoft killed FoxPro in 2007. Anyway, here's FoxPro revived

How did AMD Ryzen get 50% faster in two years?

ReBarUEFI: Resizable BAR for almost any UEFI system

'We hacked the FBI:' Hackers say they have data on all FBI employees

SAML: A fractal of bad design

Samsung accidentally freezes its smart fridges with a software update

Data-only attacks are easier than you think (2024)

WordPress: Unauthenticated path traversal leading to conditional RCE

Pentagon says overreliance on AI contributed to missile strike on Iran school

Jev Can't Be Calibrated

Comments