frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

'Companies don't care about you' is a parse error

https://blog.oliverbalfour.com/2026/11/20/companies-dont-care-about-you.html
1•OliverBalfour•4m ago•1 comments

Top White House Cybersecurity Official Steps Down

https://www.nytimes.com/2026/11/19/us/politics/white-house-cyber-director-departs.html
1•shevis•8m ago•1 comments

Serial: The Port That Won't Die

https://hackaday.com/2026/11/18/serial-the-port-that-wont-die/
1•lucaslazarus•9m ago•0 comments

WebTape – Record browser demos to GIF/MP4 from a plain-text script

https://github.com/zhushen12580/webtape
1•zane12580•11m ago•0 comments

Show HN: Tyche, An intelligent interface that clicks, books, and shops on iPhone

https://apps.apple.com/us/app/tyche-intelligent-interface/id6835683787
1•iop030290•14m ago•0 comments

Hyperscale. Play as a Frontier AI Lab and try cash out before society collapses

https://hyperscale.serinstudios.com/
2•ownbaggy•15m ago•0 comments

OpenSSL heap buffer overflow in EVP_EncodeBlock() output sizing

https://cve.now/CVE-2026-121481
2•choult•20m ago•0 comments

Grand Theft Auto Used to Satirize American Life. Now We've Caught Up with It

https://www.nytimes.com/2026/11/19/magazine/grand-theft-auto-six.html
1•elffjs•22m ago•0 comments

Fiction Machines – Léon Bottou [video]

https://www.youtube.com/watch?v=B64cnPA36Dc
1•matt_d•23m ago•0 comments

'Anthropic was supposed to save the AI trade. What went wrong?' – CNBC

https://www.cnbc.com/2026/11/19/anthropic-ipo-ai-stocks-selloff.html
1•MilnerRoute•25m ago•0 comments

Ex MS researcher publishes claimed Windows lock-screen bypass through Narrator

https://runtimewire.com/article/former-microsoft-researcher-publishes-claimed-windows-lock-screen...
2•thunderbong•27m ago•0 comments

Looking like nothing: surviving deep packet inspection with VLESS and REALITY

https://undef.dev/writing/learn/vless-reality/
1•zenincognito•29m ago•0 comments

Vint Cerf on why TCP/IP was never patented

https://seremot.com/p/vint-cerf-on-why-tcpip-was-never
1•emot•30m ago•2 comments

Ihash : the fastest high-quality non-cryptographic hash

https://crates.io/crates/ihash
1•i18n•31m ago•0 comments

The fastest double-to-string algorithm you've never heard of

https://vitaut.net/posts/2026/yy-dtoa/
1•fagnerbrack•35m ago•0 comments

Writing Python like it's Rust (2023)

https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html
1•kaycebasques•35m ago•0 comments

FreeBSD Is What Unix Was Supposed to Be

https://milkcrunch.com/freebsd-unix-philosophy/
2•zenincognito•43m ago•2 comments

Show HN: Zeus – Claude Code for AI voice agents

https://thunderphone.com/blog/introducing-zeus
1•kolchinski•44m ago•0 comments

Palo Alto Networks CEO on the network outages through major companies

https://finance.yahoo.com/technology/article/palo-alto-networks-ceo-just-dropped-a-truth-bomb-on-...
2•thunderbong•45m ago•0 comments

Top BOF Repositories for Red Teamers

https://medium.com/@nukeratframework/top-3-bof-repositories-for-red-teamers-ba67b3c5e2d7
2•JohnTheTitor•47m ago•0 comments

UK government forced to disclose secret Apple backdoor order

https://techcrunch.com/2026/11/19/u-k-government-forced-to-disclose-secret-apple-backdoor-order/
3•Coeur•54m ago•1 comments

Perfecting Terminal Character Width Using Correction Tables

https://www.jeffquast.com/post/perfecting-terminal-character-width-using-correction-tables/
2•birdculture•58m ago•0 comments

Postgres 19 is out, and it's available on Neon and Databricks Lakebase

https://www.databricks.com/blog/postgresql-19-available-lakebase
2•sp_from_db•59m ago•3 comments

Perceptle

https://perceptle.com/
3•soheilpro•1h ago•0 comments

The Game – Help people with math

https://game.poshenloh.com/
2•qainsights•1h ago•0 comments

ChatGPT for SNES

https://www.bxfoundry.com/chatgpt_snes/
3•evan_•1h ago•0 comments

New tools are helping chemists reveal a hidden world of contamination

https://www.science.org/content/article/chemists-are-uncovering-hidden-world-pollution
3•EA-3167•1h ago•0 comments

The Part About Machines (On AI Literary Scandals)

https://thepointmag.com/examined-life/the-part-about-the-machines/
3•anarbadalov•1h ago•0 comments

Using a Natural Language for Programming

https://thingspool.net/shards-of-sanity/page-6.html
3•architsch•1h ago•1 comments

US National Cyber Director to step down

https://www.axios.com/2026/11/19/trump-national-cyber-director-sean-cairncross-leaving
4•qphe95•1h ago•1 comments