frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

A Field Guide to Elephpants

https://afieldguidetoelephpants.net/
1•theanonymousone•1m ago•0 comments

A paste-y journey – finding a pastebin candidate to use

https://savyasacheejha.com/posts/a-paste-y-journey/
1•genghizkhan•4m ago•0 comments

Arduino App Lab

https://docs.arduino.cc/software/app-lab/
1•teleforce•6m ago•0 comments

Anti-Photoshop: A Mega-Combine Without a Technical Specification

https://medium.com/@smoking2man/anti-photoshop-a-mega-combine-without-a-technical-specification-f...
1•Aldus551•8m ago•0 comments

Harsher on Male? Evaluating LLMs on Gender-Asymmetric Moral Framing in Conflicts

https://arxiv.org/abs/2606.14068
1•rzk•9m ago•0 comments

Rejoining EU 'on the table' as UK makes deals on trade and young people

https://www.theguardian.com/politics/2026/nov/21/rejoining-eu-uk-trade-youth-deals-ed-miliband
1•vrganj•9m ago•0 comments

Sphere

https://en.wikipedia.org/wiki/Sphere_(venue)
2•tosh•13m ago•0 comments

ChatGPT shrunk down to run on a Super Nintendo

https://www.theregister.com/offbeat/2026/11/21/chatgpt-shrunk-down-to-run-on-a-super-nintendo/531...
1•nephihaha•15m ago•0 comments

Roast My UI

https://heltweg.org/posts/roast-my-ui/
1•rhazn•16m ago•0 comments

TI Embedded AI

https://www.ti.com/en-us/technologies/edge-ai
1•teleforce•18m ago•0 comments

Lean Should Be Ashamed

1•LTaoist•19m ago•0 comments

Show HN: Tartelette – an RP2040 emulator and breakout simulator

https://pferreir.github.io/tartelette/
1•_pferreir_•23m ago•0 comments

Kvrocks

https://www.percona.com/blog/inside-kvrocks/
4•bashhq_r•23m ago•0 comments

87% of the Anthropic power will come from gas and 11% from coal in 2028

https://betterdatacenterproject.com/2026/11/16/how-ai-is-powered-a-breakdown-of-byop-and-grid-power/
5•happosai•30m ago•1 comments

Is AI plagiarizing human research?

https://twitter.com/Robert_E_Grant_/status/2124338413662847228
2•doener•30m ago•0 comments

Scientists build whole-Earth simulation to help find life on other planets

https://www.washington.edu/news/2026/11/02/whole-earth-simulation-exoplanets-alien-life/
1•giuliomagnifico•30m ago•0 comments

Scientists sequenced a whale's entire genome using drones and whale snot

https://www.npr.org/2026/11/21/nx-s1-6044816/scientists-sequenced-a-whales-entire-genome-using-dr...
2•Brajeshwar•32m ago•0 comments

Derivative flipping: another way of thinking about automatic differentiation

http://blog.sigfpe.com/2026/11/derivative-flipping.html
2•matt_d•33m ago•0 comments

Your personal TV

https://github.com/joonfjp/yoltapp
1•joonfjp•34m ago•1 comments

The day the Riemann hypothesis fell to AI, I took my son for a walk

https://www.theintrinsicperspective.com/p/on-the-day-the-riemann-hypothesis
1•porsager•37m ago•0 comments

Intruder swaps DeepSeek at Prime Intellect. The copy almost never refuses

https://maccus.fr/en/prime-intellect-deepseek-swap/
1•theanonymousone•37m ago•0 comments

Bypassing iOS Safari's 60 FPS Limit

https://rida.dev/blog/bypassing-ios-safari-60-fps-limit
1•HatchedLake721•39m ago•0 comments

How to Be an Expert in a Changing World (2014)

https://paulgraham.com/ecw.html
1•sammatilda•43m ago•0 comments

Volta-Forever Brings CUDA 13 Support to Nvidia V100

https://www.gizmoweek.com/cuda-13-support-nvidia-v100/
1•loog5566•44m ago•0 comments

An app for M1 Macs that plays the sound of a fan as CPU usage goes up (2021)

https://fanfan.rambo.codes
2•sammatilda•44m ago•1 comments

Show HN: Built an Android app that lets you control your TV on local network

https://www.useroom.app
1•AkshayS96•45m ago•0 comments

Power harvesting from living solar panels

https://nph.onlinelibrary.wiley.com/doi/10.1002/ppp3.70295
1•fodmap•54m ago•0 comments

German unions are fighting to defend the 35-hour week

https://www.economist.com/business/2026/09/24/german-unions-are-fighting-to-defend-the-35-hour-week
3•teleforce•57m ago•0 comments

First Concrete Evidence for String Theory

https://www.math.columbia.edu/~woit/wordpress/?p=16009
1•untoxicness•58m ago•0 comments

Comcast: At peak, GTA 6 accounted for nearly a quarter of all internet traffic

https://www.pcgamer.com/games/grand-theft-auto/comcast-at-peak-gta6-accounted-for-nearly-a-quarte...
3•Brajeshwar•1h ago•0 comments