frontpage.

Hi HN, Rohit here from Tokenless (https://usetokenless.com/), which I’m building alongside co-founders Andrew and Kev. We’re building an API gateway which routes agent traffic dynamically turn-by-turn between different models to save on AI spend.

The cost of AI tokens is top-of-mind for many. Companies like Uber and Salesforce have been complaining about blowing their yearly AI spend faster than expected.

Frontier models are amazing for dev work, but are so expensive. Open-source models are cheap and rapidly improving, closing the gap with frontier models, but aren’t quite there yet.

Tokenless gets you the best of both worlds–routing harder turns to smarter models only when needed, which keeps costs low.

Before Tokenless, I was doing a PhD at Princeton. While using coding/other agents, I constantly agonized over model choice, to make sure my AI spend was going as far as possible on my academic Cursor account.

At the same time, I was doing LLM research, and a small technique I developed while in recovery from NeurIPS submission season seemed to hit SOTA pretty fast. I was surprised that such simple ideas could do routing well.

We’ve been able to develop a version of the router that matches the performance of Claude Fable 5 at half the cost. The blog post on our website explores the technical details on how we did this (https://usetokenless.com/blog/building-tokenless/).

Highlights: - Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this). - Switching models doesn’t destroy the cache if the routing algorithm is aware of when the cache is hot/cold.

To come: - Adding Kimi K3, all other GPT efforts and more to the router

Go ahead and sign up on usetokenless.com and try using Tokenless with your agent, you’ll get $20 of free credit. Here’s a demo on how to use it: https://youtu.be/sjZWriclcls

Tokenless provides frontier-level intelligence for cheaper, so we’d love some feedback on how it feels to use, any corner cases that the router routes incorrectly, and whether you find the routing problem interesting!

So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.

But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.

How do you deal with that?

> So this only switches models if the cache is cold

Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.

Remember that Deepseek can be something like 30x cheaper even cold.

There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.

> the next task is SO HARD that it doesn't make sense to not use frontier

How does tokenless quantify "so hard"?

> 1 turn to crack versus 100 turns for Deepseek to crack

Interesting definition for a "frontier". What is a "turn" here? Token count? Request count? Context-based?

I've found that MiniMax M3 (a smaller model at 295b) will code up better when DeepSeek v4 Pro (1.2t) will not (and vice versa).

How can you be cheaper if you query multiple models at once, compared to me just using a single model, which will always have the correct caching configured?

If we cut off an unconfident/expensive model early in thinking, then we don't have to pay for additional thinking cost at all. For many tasks, this is quite a large saving.

That being said, whatever models we select to race at any point will ALL have their cache updated, so if they're chosen on the next race they'll be a lot cheaper.

Maybe I'm misunderstanding, but this seems to oversimplify the problem.

Most frontier Models won't show you their true reasoning, but only a summary.

There is also no reason you can automatically assume the quality and price of the outcome based on the first few reasoning trace summaries.

If you look at the recent model releases, it becomes clear that the trend is that the largest models provide the most cost-effective solutions, because of the massive turn/tool call/reasoning reduction needed to find the solution, compared to cheaper and smaller models.

Our research seems to indicate you can get a decent signal based on the first few reasoning trace summaries. We analyzed hundreds of billions of tokens to figure this out. Any open-weights model also gives you reasoning, which is very helpful (because they are usually lower-quality candidates you would want to cut off early)

The largest models are very well known for having large reasoning traces. The thing that cracks down on overthinking is reasoning-effort training, which is largely a function of how the model is trained (certain RL objectives/teachers) rather than size of the model. The Kimi K3 technical report has some excellent work on this: https://arxiv.org/pdf/2607.

Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)

Sounds like DigitalOceans Model synthesis (https://www.digitalocean.com/blog/model-synthesis) or AilinOne (https://ailin.guide/) ?

Sounds like this would be expensive on the input token side with some saving on the decode side. For workloads with large documents / context that could be an issue.

AFAIK they solve this through their RAG based knowledgebases to only search and use the most relevant Information.

These approaches, along with OpenRouter Fusion (https://openrouter.ai/blog/announcements/fusion-beats-fronti...) are different from ours in that:

1. we trained custom models to predict LLM performance (this performs much better than asking LLMs directly to predict performance)

2. we cut off model responses early if they are not performing well, saving on costs versus running them to completion

3. finally we serve the unmodified output of the chosen model, instead of synthesizing the outputs into one, which makes us a model router

If it only switches when cache is cold, it seems like it'd lock in one model for the whole session?

What use case are you imagining where cache eviction would happen during it? It might also be the case that a task might seem easy but then it ends up becoming hard for certain parts. For spawning subagents I could see this being useful maybe.

It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.

τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.

Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?

> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track

By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.

Notation design and how it affects thought

KaaS – Knowledge as a Service: an out-of-the-box LLM wiki compiler

Show HN: Jamit

Why Mexico Built an $8,500 Electric Car

Compliance Copilot (UAE)

Show HN: Argpeek – Zero-dependency CLI argument inspector for Python

Some thoughts about Anthropic's new cryptanalysis results

Aisler Lovely Library – curated collection of electronic symbols and footprints

List of most massive black holes

eBay reaches $56M settlement with e-com newsletter writers it terrorized in 2019

Google Gemini Distillation Service [archive link]

Your Wish Is My Command

Ton 618

Show HN: Kudory – Tailor your CV for each job application

Keychron announces first open-source firmware for gaming mice

An agent with a spreadsheet engine beats one without

The last decade of front-end engineering

Every Time I Hire a Linguist, Inference Costs Go Down

AI companies are reportedly shredding books after using them to train AI models

Boss Profits from Your Excuses

NSF partners with universities and industry on pilot 4-year PhD programs

Show HN: Open-sourced, non-AI newtab extension to read your X/Twitter bookmarks

Existing industry processes are blueprints for AI workflows

eBay and ex-executives to pay $55.7M to couple sent cockroaches

Show HN: Energy, carbon and water estimates for AI content, shown as ranges

The rise of fake online shopping platforms that let you pretend to buy things

Emacs Writing Machine

Newsmax, Meta Enter AI Content Partnership

CryptanalysisBench: Can LLMs Do Cryptanalysis?

Launch HN: Tokenless (YC S26) – Automatic model switching to save money

Comments