Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.
why would you use ai for that though? how does that translate to other tasks?
kantahayashi•49m ago
I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.
magicalhippo•12m ago
Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.
saw this yesterday (when someone else posted your post), thanks for sharing such easy to understand cases which show Jev does not produce real world likelihoods for the choices
kantahayashi•1h ago
Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice
Happy to answer questions!
__m•1h ago
kantahayashi•49m ago
magicalhippo•12m ago
[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE