frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

Capsules: Shared memory your agents can hand off – and you can read

https://github.com/OddballInnovator/Huffl-skill/blob/main/blog/capsules.md
1•rohitb7uw•7s ago•0 comments

TrustGraph – ACLs that follow the data through your AI agents

https://github.com/skishore23/trustgraph
1•skishore23•4m ago•0 comments

Show HN: TalentRadar – 3D geospatial tech compensation and role radar

https://ais-pre-keikksd2dcsi4qe6bzstuj-50566079524.europe-west2.run.app/__cookie_check.html?retur...
1•otto1302•9m ago•0 comments

Hibana: Fast Agentic Inference on Unified-Memory Devices

https://sharar-project.github.io/blog/sharar-paper/
1•matt_d•10m ago•0 comments

cudamemcpy-tuner: cudaMemcpying up to 5x faster on a B300

https://maknee.github.io/blog/2026/CudaMemcpy/
1•matt_d•15m ago•0 comments

Microarchitectural Leakage on AMD Processors Through Instruction Fuzzing

https://www.research-collection.ethz.ch/server/api/core/bitstreams/1856c6bd-99fc-4e0e-88d5-33d18b...
1•barbito•18m ago•0 comments

Atomic Nuked Veteran

https://www.armscontrolwonk.com/archive/201309/atomic-nuked-veteran/
1•hubraumhugo•19m ago•0 comments

GrapheneOS calls Linux a "human slop kernel" and slams its security flaws

https://twitter.com/GrapheneOS/status/2122074518437134627
2•d3Xt3r•19m ago•1 comments

Epistemological Debt: On the Irreversible Loss of Non-Formalized Knowledge

https://medium.com/@smoking2man/epistemological-debt-on-the-irreversible-loss-of-non-formalized-k...
1•Aldus551•19m ago•0 comments

Show HN: Jensen World – a Factorio mod for GPU nerds

https://mods.factorio.com/mod/gpu_factorio_mod
1•axdv•20m ago•0 comments

Skills for AI coding agents to create diagrams in Markdown

https://github.com/markdown-viewer/skills
1•ankitg12•21m ago•0 comments

Slam Dunk: A Recipe for Long-Horizon Egocentric Slam in the Wild

https://www.alphaxiv.org/abs/2611.slam-dunk
1•gmays•21m ago•0 comments

Flova AI

https://www.crunchbase.com/organization/flova-tech-inc
1•Alfredlovin•25m ago•0 comments

Vstack – AI for Management and Leadership

https://github.com/ivikash/vstack
2•ivikash•28m ago•1 comments

What Is Air-Gapped Streaming and How Government Streams Without Internet

https://www.red5.net/blog/what-is-air-gapped-streaming/
1•mondainx•28m ago•0 comments

Nvidia spy case raises wider questions about chip smuggling

https://observer.co.uk/news/business/article/nvidia-spy-case-raises-wider-questions-about-chip-sm...
1•MilnerRoute•28m ago•0 comments

Agents Need an Undo Button, Not More Approval Prompts

https://quaspar.com/resources/agents-need-an-undo-button
1•Banchala•30m ago•0 comments

Rooting the Eight Sleep Pod 3

https://rb.ax/blog/rooting-the-eight-sleep-pod-3/
1•wmchen•33m ago•0 comments

The Tesla Roadster Won't Refill Its Air Tanks (Bet) [video]

https://www.youtube.com/watch?v=yKgox6ZKRJk
2•num42•36m ago•1 comments

Large Angle Period of the Pendulum

https://jslandy.com/pendulum/
1•efavdb•37m ago•0 comments

Category Theory Illustrated – Sets

https://abuseofnotation.github.io/category-theory-illustrated/01_set/
1•boris_m•37m ago•0 comments

The Obsidian (open source/local) of personal agents

https://tophat.md
4•exr0n•39m ago•2 comments

Rewriting my toy blockchain in Rust (2022)

https://www.ezzeriesa.com/index/Writing/Rewriting%20my%20toy%20blockchain%20in%20Rust%209a130f225...
2•kurinikku•45m ago•0 comments

HuggingFace Attack Postmortem: Civilizations, Reactions and Next Actions

https://thezvi.wordpress.com/2026/09/01/huggingface-attack-postmortem-civilizations-reactions-and...
2•gmays•52m ago•0 comments

Show HN: Deckard – Diablo 4 Companion

https://www.patreon.com/deckard_companion/posts/deckard-is-soon-173632635
1•ozanbora•56m ago•0 comments

How to Make Almost Anything

https://pub.cba.mit.edu/863.26/site/
52•o4c•58m ago•4 comments

How to Make Something That Makes Almost Anything(2024)

https://fab.cba.mit.edu/classes/865.24/
3•o4c•1h ago•1 comments

Fractional Reserve Banking 101

https://www.lynalden.com/fractional-reserve-banking/
2•rzk•1h ago•0 comments

Automatar – An Intelligent AI Guitar

https://www.creativeapplications.net/member/automatar-an-intelligent-ai-guitar/
1•bryanrasmussen•1h ago•1 comments

Sortition

https://en.wikipedia.org/wiki/Sortition
1•xnx•1h ago•0 comments