The issue was more specific to higher token burn rates, not latency.
I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact.
Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?
If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.
I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.
It mostly works to allow iteration into multiple sub components.
I forget whether it's documented or not, and it is kind of annoying that that's how you find it, but it does tell you. Maybe useful for the future if nothing else.
I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago.
Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface what goes into the "concise summary" afterwards, make it really hard to know if it actually got everything important or not.
In general, codex seems to be moving into the way of hiding as much from the user as possible, and it wouldn't surprise me if they eventually start encrypting the entire session logs just like they did with agent>sub-agent prompts recently. Sad though as it's easily the best harness+model combo available today out of the ones I've tried.
But definitely earlier GPT models suffered greatly as context got large, and the compaction itself in Codex was really crappy. That changed around January.
Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.
Claude had the same issues when they released their newer models.
Before taking a destructive action:
- Make sure the action is clearly within
the user's request.
- Resolve the exact targets with
read-only checks when necessary.
- Do not use `$HOME`, `~`, `/`, a
workspace root, or another broad
directory as the target of a
recursive or destructive command
I've overridden my rm with this, which I threw together for fast-deletes of things like Rust target/ directories, and after seeing the GPT horror story, I taught it to flatly reject deletions directly under `/` and under home directories, with a message printing the path that it's trying to delete.
Not exactly a perfect mitigation, but given that the stated risk was the model mistakenly using the wrong $HOME, it seems like a reasonable safety. I should probably make it use an even scarier rejection notice, though.
I also... have backups.
Being open source was also a useful community lever they could pull when they were trying to catch up with Claude Code.
Will say that 5.6-Sol is a minor bump in my benchmarks in most areas vs 5.5 but a severe regression in a few specific task focused on rearranging trees, addressing merge conflicts, etc. where the model to accomplish the task does not properly adhere to prompts in a way GPT-5 originally managed, not retaining parts of history in the way prompted despite specific instructions not to as that made the final completion easier…
I am of the conservative and cautious opinion that no model should be able to run destructive tasks at all, I have seen every model do things that make me concerned enough to maintain that opinion and know my evals can’t catch everything. But for 5.6-Sol specifically, I’d caution everyone to reevaluate how you run the model, maybe take a few more precautions you tend to forgo.
It is extremely capable as a reviewer and for extensive tasks, though for the later, the safety net I feel is required to be comfortable limits the utility. The code 5.6-Sol provides also still is a bit harder to parse in reviews.
Release strategy wise, feel it’s have been smarter to release only Luna and Sol now, then Terra a few weeks of posttraining later, I simply cannot see a purpose for it in the current form given how well both Luna and Sol scale up and down respectively with reasoning. Two models from a lab at a time is also the limit I feel one can properly assess at a time.
I swear there is a executive convention out there sharing worst practices.
Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine?
The lack of long context is the main reason that I still end up using Anthropic.
The worst is when you need it to hold for example a number of papers in its head, or large and complex materials that it needs full resolution on and your context window ends up being perennially at 16%. You have about five minutes of conversation and it compacts and then you have to wait for it to read that again, get to 16%... and repeat.
372 was not perfect, but it was so much better and a godsend. It turned that 12 to 20% into more like 40%.
Poking around in the repo the whole implementation is an unsupervised LLM fever-dream.
I had a /goal running last night for 9.5 hours straight while I slept. When I woke up in the morning it was fully on task and focused.
Write up a detailed design doc. Build a decent AGENTS.md, and write up a good prompt or /goal.
Long context can be more of a curse than a benefit sometimes anyways.
Another thing that's preventing me from trying Codex. (the other is @ referencing files not auto including them to the context)
1M should be table stakes for frontier models at this point for programming.
Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs.
I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better.
I had one critical rule I was maintaining about searching the codebase using a structural index/graph and not grep. Every time the agent missed it, I asked it how to improve the rules. Eventually, I asked the AI to review that rule file and it rewrote it to be 30% smaller, but, crucially, structured to be more understandable by the LLM.
Another helpful thing was to ask AI to review my rules for things it can load on-demand when it works in that area.
Problem stems from using my own abstractions instead of common frameworks.
For a full stack session, backend + front end + docs about the part we are working on adds to that amount.
I’ve accepted it because it results in code exactly the style I would’ve written. It’s a good tradeoff to cut off the slop.
There are attention mechanisms that help to mitigate this, but you can clearly see in the chart that Tibo posted that the attention mechanisms they are using are still scaling quadratically.
This becomes an optimization problem for token t:
min f[cost, quality loss] = compaction cost[t] + compaction quality loss[t] + token cost[t]
On the other hand, $10 for 1M tokens still seems really high? It's not too hard to blow through that in an hour or two.
For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you go to the /model menu. I didn't like this change, so I undid it locally. Easy peasy.
I was going to explore paying for Codex since OpenAI seems to be a bit more generous with rate limits, but I'm now not sure - for the stuff I do limited context size would be a dealbreaker - basically lots and lots of documentation and guideline references, code review loops, documentation references and web searches, multi-repo exploration, lots of tool calls and so on.
I might just upgrade to Anthropic's more expensive Max subscription or something to get more subsidized tokens. Even with plan files and the plan mode, it's like a slot machine after compacting the context, sometimes steps or other details just evaporate in thin air, the less of that I need to do the better. It's not that it can't work, it just doesn't work reliably enough not to be annoying.
Luckily DeepSeek V4 Pro, GLM 5.2 and Kimi K3 don't seem to have those limits either - though DS is around Sonnet, GLM 5.2 feels a bit above Sonnet and only K3 only really is in the proper Opus ballpark that's good enough for me to work undisturbed, even if it seems to be slower.
I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.
It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.
It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.
My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.
When context compaction introduces a gap, I use the /total-recall skill to pull prior turns back into context and off it goes.
The tool is free for personal use and has a source available local cloud option to sync convo histories across multiple machines.
However at this point it can completely maintain itself. When a new version of Claude or OpenCode is released, it updates itself to work on the latest version. It can also add new implementations for harnesses pretty reliably. It's actually pretty fun to watch it at this point. "Make this work on Hermes agent and message me when you're done" and an hour later or so, I can go play with it in Hermes.
So... at least as of a week ago or so, I don't believe so.
P.S. Love the "bonsai" name btw!
With Anthropic I run out of quota very fast on a Pro plan with long contexts.
Is long context primarily for Max tier or for API usage? Or are there special ways of working with long context on a Pro plan?
Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.
This is my super simple worse-is-better development process. I make up what is needed on the spot.
Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.
That document gets saved, and has the big picture context.
Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.
The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.
Then have a DAG of stacked pull requests designed for subagents to work through.
All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.
Think: A VP of engineering has some super high level goal: "build a new product to beat our competitor in this new market". It then filters down and various levels of reports build more and more detailed yet more narrow pieces of planning/orchestration. They don't make it work by keeping all of the context in their head and micromanaging.
Anthropic's "long context" is mostly marketing bullshit. It semi-reliably holds about the same amount of info, and then starts suffering the same issues.
Multiple (slow) conversations seems to be the efficient path for me the past week or so
it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.
so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.
I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.
that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.
but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.
large context limits is one of the things that make the open model competitors very attractive to me.
npm uninstall -g @openai/codex && curl -fsSL https://claude.ai/install.sh | bash
OpenAI devs, if you're reading, this is how you lost a customer.
It asks you to create new chats (Claude Design) and Claude Code nudges you to start over too, I think `/compact` is optional though?
This is really a killer feature in my opinion. I'm currently working on pi brains extension that is designed to solve the compaction issue I have, which is, I don't know what the AI knows after compaction.
https://gitsense.com/screenshots/pi-inspect-overview.png
The screenshot shot above contains what I call a compaction capsule which contains all the messages/events at the point of compaction. The "Files" section in the right side panel shows me how many files were read/modified since compaction.
After compacting, I can easily tell what Pi use to know and use `/tree` to go back in time.
I still wouldn't compact more than 5 times before starting a new session since compaction messages persists in Pi.
I didn't quite get it tuned up enough to be a daily driver but I'm basically sure it can be hotrodded however you want. It comes out of the box with like four different vendor hostile compact strategies, including compacting into images at the smallest size Claude can read, which the "coding is basically solved" geniuses conveniently leak the resolution heuristic out of their website along with all the other side channels they print for the MSS.
You can also ask it to read your past sessions, find places where it was wrong after compaction, and figure out a strategy to persist the most important summary information.
I have several Hermes threads that have each had >10 compactions with a 200k context limit, and with the right instructions on note-taking, they require at most a "double-check that against our past decision records" to be put back on track after a slip-up.
dannyw•10h ago
mkl•9h ago
The linked tweet is an unofficial reply to Tibo's official info and Tibo makes a correction in a reply.
imgyuri•5h ago
causal•3h ago