frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

Microsoft's ONNX Runtime has disappeared from GitHub

https://github.com/microsoft/onnxruntime
2•nvader•3m ago•0 comments

Nvidia closes lower ahead of earnings as company reels from Huang resignation

https://www.cnbc.com/2026/11/17/nvidia-stock-earnings-jensen-huang-resignation.html
1•aghuang•3m ago•0 comments

Fucked Company

https://en.wikipedia.org/wiki/Fucked_Company
1•UmYeahNo•3m ago•0 comments

Netmap caching: keeping Tailscale up on iffy connections

https://tailscale.com/blog/netmap-caching
1•mfrw•4m ago•0 comments

Everything we got before Grand Theft Auto 6

https://www.polygon.com/what-we-got-before-grand-theft-auto-6/
1•abdullahalharir•11m ago•0 comments

The Validation Gap

https://cwodtke.com/the-validation-gap/
1•mooreds•11m ago•0 comments

Instrumental Convergence

https://en.wikipedia.org/wiki/Instrumental_convergence
1•jonbaer•11m ago•0 comments

Show HN: A non-brainrot, education Roblox game for your kids

https://www.roblox.com/games/129981060143619/Roxy-Island-Math-Reading
2•velex•13m ago•0 comments

Personhood: The Killer Credential?

https://www.vennfactory.com/blog-post/personhood-the-killer-credential
1•mooreds•14m ago•0 comments

OODA Loop

https://en.wikipedia.org/wiki/OODA_loop
1•mooreds•14m ago•0 comments

Making Art with QR Codes, and the Fine Line Between Inspiration and Idea Theft

https://www.gorillasun.de/blog/making-art-with-qr-codes-and-the-fine-line-between-inspiration-and...
1•shminge•17m ago•0 comments

Salesforce and Microsoft Announce Copilotforce

https://www.salesforce.com/news/stories/copilotforce-announcement/?bc=HL
2•qainsights•19m ago•0 comments

I hate GPT-6.2 Astra

https://twitter.com/i/status/2122760447103484410
4•Michelangelo11•24m ago•1 comments

StubHub International Announces Data Breach

https://news.pollstar.com/2026/11/17/stubhub-international-announces-data-breach/
1•zgeor•25m ago•0 comments

How to Tell If It's a Cyberattack or Just an Outage

https://www.wired.com/story/how-to-tell-if-its-a-cyberattack-or-just-an-outage/
1•emot•25m ago•0 comments

Revisiting my Master's thesis with AI – Understanding the Riemann Hypothesis

https://riemannhypothesis.blog/2026/11/revisiting-my-masters-thesis-with-ai/
2•kurinikku•29m ago•0 comments

Nvidia Beats Estimates, Raises Guidance After Huang's Ouster

https://www.wsj.com/tech/ai/nvidia-nvda-3q-2027-earnings-report-stock-6e642303
3•karimf•29m ago•0 comments

Ask HN: I read Blake Masters' notes for Zero to One went viral on HN

2•illidan1000•30m ago•1 comments

Show HN: Vyro Clipping Calculator – what your views actually pay

https://vyroclipping.org/
1•fireflywedding•31m ago•0 comments

Nvidia says it does not intend to reinstate Jensen Huang as CEO

https://www.theverge.com/report/1035465/nvidia-says-it-will-not-reinstate-jensen-huang-as-ceo
7•granto•31m ago•0 comments

Ageing could be switched off by a new gene

1•denisreshetov•31m ago•0 comments

The Accelerator Gap in Cancer Therapy

https://cerncourier.com/the-accelerator-gap-in-cancer-therapy/
1•EA-3167•32m ago•0 comments

ReactiveScript: A language for describing reactive UI

https://reactivescript.dev/
1•thomastill•37m ago•0 comments

Elderly people are buying Teslas for the 'full self-driving.' This is a problem

https://www.fastcompany.com/91641371/elderly-people-are-buying-tesla-for-full-self-driving-this-i...
2•danso•37m ago•0 comments

TDD with AI Agents: Design the Tests First, Let the Agent Implement

https://realpython.com/tdd-ai-agents/
1•rbanffy•40m ago•0 comments

StubHub Data Breach Exposes Customer Details

https://cypro.co.uk/insights/cyber-bulletins/stubhub-data-breach-exposes-customer-details/
1•zgeor•41m ago•0 comments

GPT-6.2 Astra and GPT-6.2 Sol

https://simonwillison.net/2026/Nov/17/gpt62-astra-and-sol/
2•acossta•41m ago•0 comments

Migrating to Python 3.15 – Mathspp

https://mathspp.com/blog/migrating-to-python-315
2•rbanffy•42m ago•0 comments

GTA 6 on Google: four years of searches, in numbers

https://www.seodata.dev/blog/gta-6-search-trends
1•alentodorov•46m ago•0 comments

Show HN: Infoboxing – Guess the Wikipedia article from its infobox

https://www.infoboxing.net/
1•spykr•46m ago•0 comments