Could and would being two different things, turned out both llama.cpp and vLLM were shitting their pants running the model at the time (love you guys, I know this model was a pita!), particularly around tool calling (vLLM was / is broken seven ways to Sunday), mostly due to the Harmony template introduced by OpenAI (which, coincidentally (?) is almost identically implemented in Gemma 4 and somehwat similar in Muse Glimmer, 9-12 months after the gpt-oss release, so OpenAI was on to something there and likely not just for the OSS release but their bigger and closed siblings too).
Anyway, validating my hypothesis with the vanilla backends proved impossible at the time.
So I did the only rational thing: built an inference harness that fixes the model, then ran probably the most autistic evals in history -- 320,192 questions across 8 seeds, prefilling and decoding over 3.49B tokens, for 1,062 hours of batch size 1 GPU time on a single 3090.
In the words of Carl Sagan, to make an apple pie from scratch, you first have to invent the universe. I spent my nights inventing this one in parking lots between food delivery gigs, so I named it burrito.
All that just to test whether OpenAI shipped a broken model (spoiler: it didn't). Did it work? Here's the hero shots for the final boss of tool calling evals: multi-turn, pass@8 (at least 1 seed of 8) and pass^8 (every seed).
https://raw.githubusercontent.com/iamskeole/burrito-evals/re... > task solve rate on at least one seed
https://raw.githubusercontent.com/iamskeole/burrito-evals/re... > task solve rate on every seed
Sharing everything, MIT:
- harness: https://github.com/iamskeole/burrito-core - evals (incl. full inference traces): https://github.com/iamskeole/burrito-evals - fixed jinja template: https://huggingface.co/openai/gpt-oss-20b/discussions/274/fi...
(Detailed analysis on reasoning zones and "optimal effort" levels can be found in the evals repo)