At 10M tokens, the scale is vastly larger than any model's context window, so context stuffing isn't an option (aside from the fact that only about half of a large window can be effectively utilised without degradation). The only way to score well is recall that fundamentally works.
Our system (M-1) scored 76.9% at 100K, 75.0% at 1M, and 68.0% at 10M. Previous leaders were Hindsight (73.4%, 73.9%, 64.1%) and Honcho (63.0%, 63.1%, 40.6%), both using Gemini 3 Pro, while we used Flash.
We saw the competitive gap get wider at scale: 3.5 points ahead of Hindsight at 100K, 3.9 at 10M. The gap with Honcho goes from 13.9 to 27.4 points. As the corpus gets bigger, it filters out effective recall vs. brute-forcing / model capability.
M-1 also consumed about 20% fewer tokens per query than the next best system.
About the BEAM benchmark: BEAM tests ten memory abilities including some that other benchmarks don't cover: contradiction resolution, event ordering, and instruction following. The 10M token scale is vaguely equivalent to a year of long daily chats with an LLM.
Of course our system is still far from perfect, with strength in some categories (preference following, instruction following, summarization, abstention) all consistently above 90%, even at 10M token scale.
And our system shows weakness in others: for example, multi-session reasoning: 44.7% at 100K, collapsing to 9.6% at 10M. Although that challenge seems to be a general problem across memory systems at this scale, not M-1 specific. Something we'll continue to work on.
Methodology: We forked Hindsight's open-source benchmarking script, replaced the retrieval layer, and used the runner's prompt structure with minor adjustments for production use. Full methodology, results JSON for all three scales, and the prompt generator are linked in the paper (paper linked below).
Combined with our LongMemEval result (96.4%), M-1 is now the only system to hold SOTA across both major memory benchmarks at every scale, from 115K to 10M tokens.
Research paper: https://exabase.io/research/exabase-achieves-state-of-the-art-on-beam-benchmark
Happy to discuss architecture, the benchmark, scale challenges etc.
johnnymakes•41m ago