Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.
We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.
Yeah this is a great observation--we have to collect some more data to understand new models as they come out. However, by having this composable architecture that separates "confidence prediction" and the actual "routing layer," this makes it a lot easier to add new models/restrict the models rather than re-training from scratch.
We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.
Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)
We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).
Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models.
We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made
If we cut off an unconfident/expensive model early in thinking, then we don't have to pay for additional thinking cost at all. For many tasks, this is quite a large saving.
That being said, whatever models we select to race at any point will ALL have their cache updated, so if they're chosen on the next race they'll be a lot cheaper.
Maybe I'm misunderstanding, but this seems to oversimplify the problem.
Most frontier Models won't show you their true reasoning, but only a summary.
There is also no reason you can automatically assume the quality and price of the outcome based on the first few reasoning trace summaries.
If you look at the recent model releases, it becomes clear that the trend is that the largest models provide the most cost-effective solutions, because of the massive turn/tool call/reasoning reduction needed to find the solution, compared to cheaper and smaller models.
Our research seems to indicate you can get a decent signal based on the first few reasoning trace summaries. We analyzed hundreds of billions of tokens to figure this out. Any open-weights model also gives you reasoning, which is very helpful (because they are usually lower-quality candidates you would want to cut off early)
The largest models are very well known for having large reasoning traces. The thing that cracks down on overthinking is reasoning-effort training, which is largely a function of how the model is trained (certain RL objectives/teachers) rather than size of the model. The Kimi K3 technical report has some excellent work on this: https://arxiv.org/pdf/2607.
> So this only switches models if the cache is cold
Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.
Remember that Deepseek can be something like 30x cheaper even cold.
There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.
Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive.
I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie. give model specs for certain delegation roles).
Seems like it might be more advantageous to just adjust reasoning effort to retain cache.
Some agents when you alter the reasoning level, partially or complete wipe the cache. Never assume that changing reasoning is a no-impact change. I do not have the link but a French youtuber did a ton of research on this.
I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For example, there is GPT-5.6-Sol low, med, high, xhigh, max, and lots of "rules of thumb" that people develop on which one to use when.
Highly encourage you to read the blog post (https://usetokenless.com/blog/building-tokenless). Essentially, we estimate the confidence of a specific model failing or succeeding on a specific task using our own foundation models.
A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.
It will inspire more confidence if you say your classifier. Saying "your own foundation model" (charitably) suggests marketing hyperbole. If you're fine tuning a LLM to do this, at best it's going to be worse than a frontier model with suitably tuned skills (hence it should just be something in harness, not a service), and I wouldn't refer to that as "your own foundation model."
An approach that works really well for me in opencode is to use DeepSeek flash for the code exploration sub agent, and a smarter model for the overall task.
Code exploration tends to burn a ton of tokens and do not require frontier knowledge. It then returns a compact overview of the findings to the better model, that can therefore perform its task with less tokens.
The solution that I am working on is designed to leverage cheap LLMs (think MiMo, GLM, etc.) to build cheap context to reduce tokens, but most importantly to keep the context clean.
How would your classifier compare to the frontier model delegating to Deepseek via a MCP tool call itself? Doesn’t it have far more insight on the difficulty of the task then your router could? I’ve been doing this with Claude Code delegating to Codex and an Ollama Qwen model.
Would it not also be more efficient to promot to delegate more tasks where possible to just less capable agents? This is basically what Cursor does (although most certainly in an effort to get more grok/composer usage)
reply