There seems to be a need to be some kind of hierarchy to contextualization brought to LLMs and not a flat history stuffed into the context. For comparison, human memory is not just a compressing of linear history of which we rewind the tape when needed. Its more like a Russian doll of bounded contexts which we break out of, when our inner context is no longer sufficient to solve the problem.

I find this topic very interesting because it's something I've run into and mitigated ever since gpt3 was available.

Plenty of long, loooong and complex role plays, world building and tests to see if I could integrate dozens of different local models into a game project or similar.

All of the same issues there apply here for "agents" as well.

Very quickly learn that even current models are like distracted puppies. Larger models seem to be able to brute force their way through some of these problems but I wouldn't call that sustainable.

Are there published numbers or benchmarks that tells model X is brain fried at Y tokens?