This was the initial anecdotal evidence, but we undertook this small study to collect empirical data:
We added logging between the agentic coding tool (Claude Code and OpenCode) and Anthropic's endpoint, and captured all requests (and the returned usage blocks).
With one caveat (toward the end of the post) we found unambiguously that Claude Code was far more inefficient in terms of its cache strategy and its harness token usage than OpenCode.
https://quesma.com/blog/the-true-cost-of-saying-hi-to-an-ai-...
Tokenflation seems very real: the number of tokens consumed by simple tasks keeps increasing.
I am a bit surprised that you're disappointed that it does exactly what you told it to - people usually have the opposite complaint.
If you're using it interactively and watching what it changes, I'd trigger the tests when you think it's needed. And if you want to go more hands-off, why not add try to encode the same nuance you'd use into the rule?
The entire agent system prompt can be seen here:
https://github.com/earendil-works/pi/blob/main/packages%2Fco...
It works great for long-horizon tasks, and feels like it saves a boatload of tokens.
I'm generally against this context pruning without prompting or details. Sleev is very opaque about how it works and definitely will bust your cache.
I used mitmproxy (setup assisted by Claude, natch) to capture Claude Code's entire initial system prompt and the whole thing was (I just double-checked) 162k of JSON.
This led me to start experimenting with Pi, OpenCode, and Hermes...
Opus 4.8 (1M context)
claude-opus-4-8[1m]
23k/1m tokens (2%)
Estimated usage by category
System prompt: 3.9k tokens (0.4%)
System tools: 13.9k tokens (1.4%)
Custom agents: 235 tokens (0.0%)
Memory files: 28 tokens (0.0%)
Skills: 4.9k tokens (0.5%)
Messages: 8 tokens (0.0%)
Compact buffer: 3k tokens (0.3%)
Free space: 974k (97.4%)
I was simply supporting the article's data - their reported 33k tokens is probably roughly 150-165k.
My $20 sub using gpt 5.6 sol thinking-off lasts for hours using pi.
Doesn't the model need at least a basic system prompt to understand what tools are available?
Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp
That would depend entirely on what your device is. This sounds likely not to be an issue with the harness, but the capabilities of the models you've tried.
I experience almost no tool call failure using my nothing-special harness and DSv4 Flash.
Edit: consumer Claude subs are the 5%. I’d bet most all of CC subs lump in under enterprise.
- API & Enterprise: 75% to 85% of total revenue.
- Business Subscriptions: Roughly 10% to 15%.
- Individual Subscriptions: About 5%.I don't fully agree with the premise that they intentionally increase system prompts, but the enterprise plan usage is going to make that a huge income for Anthropic.
I mean, that's a very weak argument? Isn't a much more plausible explanation that with your tooling you'll have more of a lock-in than with just your model?
I asked both a trivial question (summarize last commit). Opus cost 50 cents, Fable about $1.
That checks out because Fable's twice as much in the API (though I think its emphasis on correctness makes the difference larger for bigger tasks).
But, at $1 per question, I think I will stick to the subscription for now! I was certainly glad GPT-5.6-Sol is included in OpenAI's subscription, and I'm curious if they'll be able to do the same for GPT-6.
All the VC money appears to have run out a few weeks ago.
I haven't tested it on anything bigger but it doesn't seem to do the kind of proactive testing, that they do in bigger harnesses.
Codex at least has a system prompt that tells it not to consider a feature a complete until it has verified it. I'm not sure about Claude Code.
I suppose I could add that one line to the prompt, and it would get me much closer to agi :) I think Fable does this proactively even without a prompt, but I haven't tested that yet.
If Fable in my own harness is significantly cheaper than Claude Code, that would be very appealing. (I could actually afford to use it for most things!) But I think most of the cost comes from the testing it does. So we'll have to see.
Competition is good.
Are we measuring and caring about the right thing?
> When context gets too long, maki compacts history automatically: strips images, thinking blocks, and summarizes older turns.
Don’t the summaries of older turns effectively invalidate the context cache, such that you consume less tokens but more expensive tokens?
I don’t like that Claude became more opaque around February, including the system prompts. 33k feels way too much.
If I let the main agent do the same task sequentially, it was no problem at all. I don't know if it's really just communication and orchestration that makes sub agents so inefficient, or if Anthropic figured that most people using sub agents pay per token on a big corporate account, so this is an easy way to make more money from tokenmaxxers.
Supposed to be hacker news and half the posts are like "this harness steals this" like it cant be avoided.
These API costs are mad.
It pays to be marginally ahead of people stuck on open models.
After reading PUSH_AX's valid comment: ``` This is like saying contractor (A) asked for $33,000 to undertake the work and contractor (B) asked for $7,000 Are we measuring and caring about the right thing? ``` We will update the post to include:
1) A more in-depth task. 2) Qualitative results comparison. 3) As soon as possible, a reproduction of the inputs and outputs.
I wonder if a lot of the 33k is context, like from recent conversations.
If you don't use a subscription, and pay per token instead, you can easily move to another harness.
I read that this is because it wastes time looking through past conversations and other context to figure one what you might want it to do - a less ambiguous prompt would be better.
What do you miss? I ask because I do some heavy work with pi + GLM 5.2 (using opencode Go subscription) and my workflow is plan -> implement.
Sure, but you have to add almost everything, no? It deliberately only comes with read, write, edit, and bash. My point wasn't that you can't add stuff, but that I'd just rather use an harness that's a bit more full featured from the start.
(Pi is a bit like old 3D printing where fettling the printer to work is a central part of the hobby. I'd rather just buy a Prusa.)
I guess the cache would only be invalid if the day changed or the root directory, which would technically happen infrequently enough.
Also I think it’s well known that OpenAI is the much less expensive option (in tokens and $$). For the same $20 you get a lot more mileage.
Curious if folks have strong opinions about the overall UX of OpenCode vs CC…
Especially since compute is such a scarce resource.
and the interesting thing about system prompt wastage is its a cost that scales non linearly with subagent use.
Not sure if intentionally meant as a reference, but it gives "I use Arch btw" vibes.
That said, if your project is "do this well-planned thing on a bunch of things in parallel" then you should absolutely be instructing to have subagents "step down" to less curious models. Their output may well be more cohesive as a result!
If you do it sequentially you only read those files approximately once, and everything hits the same prefix cache
Probably because the general purpose subagents inherit the parent model.
I tell Claude explicitly to use Explore subagents, which use Haiku only, now.
only if you don't specify which model should be used
MallocVoidstar•1h ago
So not only is this article AI-written, but the testing was entirely done by AI, too? I can't see any other reason to use such an old model.
> Our traffic passes through a local LLM gateway that wraps requests in its own envelope, a constant we measured at roughly 6,200 tokens with bare calibration requests
Why do you need to do calibration requests to figure out how your own gateway is affecting requests?
> Its subagent lane did not complete cleanly through our gateway
> We attempted to toggle extended thinking in both harnesses and are declining to publish numbers. Our gateway applies its own thinking policy, neither harness's toggle demonstrably survived the path, and anything we quoted would be noise.
Why is your own gateway screwing with your testing?
systima•1h ago
Cost, mainly. The runs went through a Claude Max subscription rather than metered API billing, and pinning an older stable snapshot kept run-to-run comparisons clean and cheap. The fixed harness payload (system prompt plus tool schemas), so the headline numbers shouldn't change too much.
That said, happy to re-run the matrix on Fable and publish the diff; payload figures should barely move, tool-calling behaviour might.
Gateway:
Meridian (github.com/rynfar/meridian); proxy that bridges the Claude Code SDK to a standard Anthropic endpoint so a Claude Max subscription can drive OpenCode-et-al.
It's the auth route for all agent traffic on the machine, not something built for the benchmark.