Hi HN. STROT compiles a decision task — "route these tickets", "score this feedback", "classify this text" — into a standalone pipeline. A large LLM is used once, at compile time, to synthesize the labels, the input projection, and the handlers. At runtime a small local model (144M, CPU) makes every decision: ~60ms/record, $0 marginal, zero LLM calls, bounded typed-JSON output.
The goal was to move the expensive, nondeterministic part (an LLM) entirely to build time and keep runtime frozen, cheap, local, and auditable — the way a compiler moves work off the hot path.
What I think is actually interesting, beyond "small model does inference":
- The compiler measures instead of guessing. It scores candidate labelings on a teacher-labeled sample and keeps the winner; on the identical model that's +18pp on a routing benchmark (92.5% vs 74.8%) and +14pp on text classification (86.7% vs 72.7%) over naive integration.
- It fits a conformal-calibration threshold at compile time, so low-confidence records abstain with a marginal error guarantee rather than guessing.
- A golden-record gate rejects any compile that fails known-answer examples — synthesis is stochastic, and I show that variance in the benchmarks.
What it doesn't do: it can't make a small model understand things it fundamentally can't. On tweet hate-speech it hits the model's ceiling; the fix there is a task-trained model in the swappable slot, not the compiler. All of it — including the losses — is in the repo's benchmarks with reproducible scripts.
There's a browser demo (a 144M model ( which is changeable ) playing Pac-Man, one decision per frame) and a quickstart that runs in a few lines. MIT licensed, pure-stdlib core.
Happy to go deep on the calibration math, the compile/cache design, or where this does and doesn't make sense.
sysguest•59m ago
> Happy to go deep on the calibration math, the compile/cache design, or where this does and doesn't make sense.
yes plz
dpacmittal•48m ago
Was this entirely vibe coded? The readme is very obviously Claude with no human edits at all.
nodja•32m ago
Yes, even the comment you replied to is very noticeably claude authored.
razingeden•27m ago
>On tweet hate-speech it hits the model's ceiling
This sounds like a sus endeavor to redline your LLM with, could you elaborate ?
jMyles•1h ago
I think that "infer once, repeat resulting logic forever" is one of the most important / sustainable / humane ways to imagine how AI acclimates to the universe.
I was working with a large entries on Batch inference, and yes LLMs are quite expensive to process Billions of contracts at latency, so I in my orignial paper was on the expensive thinking once, run the cheap deterministic result forever.
wonnage•41m ago
> If a ghost is close (nearest_ghost_distance small, "
"within 2 steps), choose flee_ghost. Otherwise choose "
"chase_dot to eat the nearest dot."
Dude, this is AI psychosis. Just ask the LLM to write you a program (“compile”) and then you can run it locally forever
ghurtado•22m ago
That is literally what this does
publlus_enigma•26m ago
I love the concept, but tried reading the README on the got, but I am still unclear about precisely what it does or how it does it. Some more detail would be welcome.
solohunter001•1h ago
The goal was to move the expensive, nondeterministic part (an LLM) entirely to build time and keep runtime frozen, cheap, local, and auditable — the way a compiler moves work off the hot path.
What I think is actually interesting, beyond "small model does inference":
- The compiler measures instead of guessing. It scores candidate labelings on a teacher-labeled sample and keeps the winner; on the identical model that's +18pp on a routing benchmark (92.5% vs 74.8%) and +14pp on text classification (86.7% vs 72.7%) over naive integration. - It fits a conformal-calibration threshold at compile time, so low-confidence records abstain with a marginal error guarantee rather than guessing. - A golden-record gate rejects any compile that fails known-answer examples — synthesis is stochastic, and I show that variance in the benchmarks.
What it doesn't do: it can't make a small model understand things it fundamentally can't. On tweet hate-speech it hits the model's ceiling; the fix there is a task-trained model in the swappable slot, not the compiler. All of it — including the losses — is in the repo's benchmarks with reproducible scripts.
There's a browser demo (a 144M model ( which is changeable ) playing Pac-Man, one decision per frame) and a quickstart that runs in a few lines. MIT licensed, pure-stdlib core.
Happy to go deep on the calibration math, the compile/cache design, or where this does and doesn't make sense.
sysguest•59m ago
yes plz
dpacmittal•48m ago
nodja•32m ago
razingeden•27m ago
This sounds like a sus endeavor to redline your LLM with, could you elaborate ?