frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

What is the generalized Riemann hypothesis? The claim behind two rival proofs

https://www.nacionale.com/en/articles/what-is-generalized-riemann-hypothesis-grh-explained-alpoge...
1•doener•10m ago•0 comments

Show HN: Paper Trail – Local work records

https://chromewebstore.google.com/detail/paper-trail-—-local-work/glpahjppiibjanjkknphegmlmindi...
1•cflsdf•13m ago•0 comments

Show HN: Food Sound · White Noise ShiYin

https://nonloop.github.io/crunchy/
1•cflsdf•14m ago•0 comments

Portkey Security Incident

3•redrix•18m ago•0 comments

Why Does Windows Use Backslash as Path Separator? (2019)

http://www.os2museum.com/wp/why-does-windows-really-use-backslash-as-path-separator/
3•sammatilda•24m ago•0 comments

Show HN: AbcTab – Native Swift 6 macOS ⌘-Tab Switcher Without Screen Recording

https://solocraft.app/abctab
1•addozhang•24m ago•0 comments

Show HN: Name the World – An Atlas Quiz

https://nametheworld.starshadowx2.workers.dev/
1•starshadowx2•25m ago•0 comments

Pint – Windows NT 4.0 Ported to Raspberry Pi

https://github.com/tenox7/pint
2•majorchord•26m ago•0 comments

Show HN: Find and serve fast LLM config for your GPU

https://github.com/Aagam-Bothara/rigtune
2•Rogansempai•30m ago•0 comments

Show HN: yavcpdf - A keyboard-driven, vibecoded PDF viewer

https://github.com/araaha/yavcpdf
1•Ithinkthatsnice•33m ago•0 comments

Anthropic's IPO bank closed position meant to stabilize stock, removing support

https://www.reuters.com/markets/us/anthropics-ipo-bank-closed-position-meant-stabilize-stock-remo...
3•greatgib•40m ago•0 comments

Show HN: Voxstead – Live captions and bilingual call assistance for macOS

https://github.com/TriHerm/voxstead
2•abishai33•42m ago•0 comments

Show HN: Pstview – read-only PST/OST viewer

https://github.com/publicarray/pstview
3•publicarray•44m ago•0 comments

What if GTA 6 were set in Boulder?

https://www.axios.com/local/boulder/2026/11/20/boulder-grand-theft-auto-6-gta-subaru-parody
1•mooreds•45m ago•0 comments

AI R&D-relevant model capabilities are improving rapidly

https://metr.org/blog/2026-11-20-q4-2026-capability-update/
2•ravenical•47m ago•0 comments

Show HN: Tool for WordPress that researches, writes, publishes SEO articles

https://loopseo.org/
1•loopseo•48m ago•0 comments

Ask HN: What are people doing to improve the security of your organization?

1•fazkan•48m ago•2 comments

Things I learned coding with AI for a year

https://twnstr.com/blogs/software-development-post-ai
2•BobbyJo•52m ago•1 comments

Integrals of Polynomials with Galois Group S_p

https://m-slee.netlify.app/posts/integrals-galois-polynomials
3•richard_chase•53m ago•1 comments

The Normal Person's Guide to the Jensen Huang Scandal

https://www.todayintabs.com/p/the-normal-person-s-guide-to-the-jensen-huang-scandal
1•MBCook•55m ago•0 comments

Takes for Builders in 2027

https://lincolnquirk.substack.com/p/7-takes-for-builders-in-2027
1•lincolnq•56m ago•0 comments

Procrastination Station

https://thenerdnet.com/procrastination-station
1•gsverrisson•56m ago•0 comments

Nobody Wants to Think Anymore

https://notesfromundergroundd.substack.com/p/our-brains-are-gone
2•nwhnwh•57m ago•0 comments

Punishment is on its way out

https://jessicaocean.substack.com/p/punishment-is-on-its-way-out
1•paulpauper•58m ago•0 comments

Search Engine Relationship Charts

https://www.searchengines.net/guides/search-engine-relationship-charts/
2•bhartzer•58m ago•0 comments

Claude Watermarks Your Words, and You Can't Check Them (Yet)

https://www.oreilly.com/radar/claude-watermarks-your-words-and-you-cant-check-them-yet/
2•cyndunlop•1h ago•0 comments

Detecting AI Agents in Next.js with Vouched KYA

https://www.vouched.id/learn/blog/detecting-ai-agents-in-next.js-with-vouched-kya
1•bminnick•1h ago•0 comments

Show HN: Craoli – describe a job in one sentence, it finds real postings

https://craoli.com
2•thanouil1411•1h ago•0 comments

Show HN: Scots-Codes

https://github.com/vanyakosmos/scots-codes
1•aiven•1h ago•0 comments

Android phone and desktop mode on your Mac

https://github.com/lucianmarin/macdroid
4•lucianmarin•1h ago•0 comments