frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

A Man and His Muse AI Agent Sue Meta

https://www.404media.co/a-man-and-his-muse-ai-agent-sue-meta/
1•Michelangelo11•57s ago•0 comments

I Started a Shell Company

https://thijs.net/posts/shell-company
1•skadamat•3m ago•0 comments

Bee: Remember What Matters

https://bee.computer/
1•PaulHoule•4m ago•0 comments

1984 Hosting suspends all services following serious cyberattack

https://dv.is/frettir/2026/11/20/stodva-thjonustu-timabundid-eftir-alvarlega-tolvuaras-a-innvidi-...
1•circularfoyers•4m ago•0 comments

Anti-fragility, adaptation and its limits

https://medium.com/@smoking2man/anti-fragility-adaptation-and-its-limits-3d67fac6f5aa
2•Aldus551•4m ago•0 comments

Nuclear bomb in space would disable 98 per cent of satellites within a week

https://www.newscientist.com/article/2598378-nuclear-bomb-in-space-would-disable-98-per-cent-of-s...
1•Noaidi•4m ago•1 comments

Building the Agent Host Protocol for sessions that continue across devices

https://github.blog/engineering/engineering-principles/building-the-agent-host-protocol-for-sessi...
1•torutofu•5m ago•0 comments

Could All the Rain in California Trigger Earthquakes? (2023)

https://www.newsweek.com/rain-california-earthquakes-floods-extreme-weather-1772913
1•Noaidi•6m ago•0 comments

Show HN: Hobby MQTT 3.1.1 broker written in Zig

https://github.com/threeaccents/zancudo
1•threeaccents•8m ago•0 comments

Be Kind to AI (?)

https://elsevanderberg.substack.com/p/be-kind-to-ai
1•flail•8m ago•0 comments

Nvidia adds possible hack to list of bad things under investigation

https://www.theregister.com/security/2026/11/19/nvidia-adds-possible-hack-to-list-of-very-bad-thi...
2•Bender•9m ago•0 comments

Flaws in VMware and KVM need attention, if you have time while fixing Java, PHP

https://www.theregister.com/security/2026/11/20/flaws-in-vmware-and-kvm-need-attention-if-you-hav...
1•Bender•10m ago•0 comments

ViReC: The Virtual Register Context Architecture for Efficient Multithreading [pdf]

https://slam.ece.utexas.edu/pubs/icpp25.ViReC.pdf
2•rbanffy•10m ago•0 comments

Another company I've never heard of lost my Data

https://idiallo.com/byte-size/another-company-lost-my-data-and-free-credit-monitoring
2•Brajeshwar•12m ago•0 comments

AI Search Analytics Confirm Higher Ed's LLM Winners Are Large Publics/Nonprofits

https://insights.kolariconsulting.com/p/ai-search-analytics-confirm-higher
1•aesokol•12m ago•0 comments

Uber tests rides for kids as young as 10, with extra pay for drivers

https://techcrunch.com/2026/11/19/uber-tests-rides-for-kids-as-young-as-10-with-extra-pay-for-dri...
2•ibn_khaldun•13m ago•0 comments

Show HN: Free red-flag checker for VPAT accessibility reports

https://acrpacket.com/check
1•rpst•14m ago•0 comments

Realistic lunar ISRU processing facilty simulation: Smelting rocks into metal [video]

https://www.youtube.com/shorts/bUqcwLoCO50
2•shroom1•14m ago•0 comments

FastAPI SaaS starter with Stripe subscriptions, EU VAT and HTMX

https://demo.getpylaunch.com
1•patryk_builds•15m ago•0 comments

Bending Spoons' Acquisition History

https://www.kadoa.com/companies/insights/bending-spoons-acquisitions
2•hubraumhugo•16m ago•0 comments

Agentic engineering with GitLab and your local agent: A walkthrough

https://about.gitlab.com/blog/agentic-engineering-walkthrough/
1•jerrygarcia•17m ago•0 comments

It's the End of the Academic World as We Know It

https://gkallis.substack.com/p/its-the-end-of-the-academic-world
2•the-mitr•20m ago•0 comments

Why small IT teams get stuck between free VPNs and Netskope or Proofpoint

https://arxcloud.io/vs/netskope-proofpoint/
2•justmedric•21m ago•0 comments

Why the U.S. air traffic control system still relies on an 'antique' network

https://www.npr.org/2026/11/19/nx-s1-6045773/faa-air-traffic-control-system-modernization-challenges
4•blinding-streak•22m ago•0 comments

Copyright is dead. Thank you Dario

https://larrysalibra.com/copyright-is-dead-thank-you-dario/
2•larrysalibra•23m ago•1 comments

Jon McCormack Retires from Apple

https://daringfireball.net/2026/11/mccormack_retires_from_apple
1•Brajeshwar•23m ago•0 comments

Introduction to Genie

https://docs.vala.dev/genie/introduction
1•giancarlostoro•24m ago•0 comments

SoduWork – AI agents must show evidence before a task passes

https://medium.com/@csdn/two-days-at-the-top-soduwork-hits-no-1-1f32a3f89d85
4•Aki_ii•25m ago•0 comments

Show HN: Drive around my village in 3D

https://cherwellvalleymotorclub.app
3•johnwards•25m ago•3 comments

Jonathan Swift vs. AI

https://www.thedial.world/articles/news/schools-teachers-artificial-intelligence
2•bookofjoe•26m ago•0 comments