frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

"A milion token context" Big AI says. But the model is accurate for 2-4K tokens

https://unagent.eu/2025/04/22/misleading-promises-of-long-context-llm/
2•kzawpl•1y ago

Comments

kzawpl•1y ago
Over last two years there were claims of better long context capabilities for LLM, but that is often tested on exact text search. New benchmark called NoLiMa shows that long context capability of LLM is still poor, if you want LLM to perform some abstraction and reasoning.
vessenes•1y ago
Meh. NoLima is helpful, in that it shows what we all "feel" working with models -- there's a marked dropoff in accuracy and intelligence as we get past 4-32k of context, depending on the model.

But, it seems unreasonable to be super worried about this -- a year or two ago, models couldn't easily find needles in haystacks of long context. As training and test strategies delivered trainable content, this became a thing that could be done perfectly across millions of tokens of context. There has not been a good way to incentivize models to do anything more but remember locations yet.

We are (mostly) paying the full costs of attending to the entire context in current architectures, and it seems pretty reasonable that we will therefore be able to train those architectures to more fully attend across context if we get the right training data into (ideally) an RL loop.

NoLima is an okay test, but I think the most recent OpenAI tests are significantly better and quite interesting; OpenAI-MRCR and Graphwalks are both super smart ideas about how to programmatically generate data that is easy to evaluate and forces better cross context attention.

From their 4.1 announcement: Graphwalks fills the context window with a directed graph composed of hexadecimal hashes, and then asks the model to perform a breadth-first search (BFS) starting from a random node in the graph. We then ask it to return all nodes at a certain depth.

MRCR asks for direct quotes at semantically identified locations in the text, e.g. poems about tapirs, bears and ballerinas, as well as stories about tapirs, bears and ballerinas are generated, perhaps fifty each. The system is asked "give me the third poem about tapirs". This requires counting, conceptual attention, and also distinguishing between stories and poems.

They only test their own models on MRCR for the benchmark graph, but it's still worth reviewing: the accuracy curves are super interesting. https://openai.com/index/gpt-4-1/

'He used Pornhub as the gun'

https://www.icij.org/news/2026/11/he-used-pornhub-as-the-gun/
1•DeepLogin•26s ago•0 comments

Show HN: Loisium – An Autonomous Ecosystem for Software

https://loisium.dev/
1•clinexaadmin•1m ago•0 comments

Deutsche Bank accidentally leaked Donald Trump's tax statements

https://www.ft.com/content/784cbdc7-c852-4081-b1e5-390779e9fad1
1•dudefeliciano•1m ago•1 comments

NashGate: Multi-agent LLM consensus with staking and liquidation

https://github.com/Daiv1007/nash-gate
1•Daiv1007•3m ago•0 comments

Show HN: Askingprice.io – guess the price of 7 marketplace listings each day

https://askingprice.io
1•bigboingy•3m ago•0 comments

AI Is Finally Good Enough for Travel Planning

https://eamag.me/2026/ai-travel-planning
1•eamag•4m ago•0 comments

Rocket.Chat 9.0 removes LDAP authentication from Community Edition

https://docs.rocket.chat/docs/rocketchat-release-notes
1•ycombluenn•4m ago•0 comments

Opus 5.5 covertly sabotaged a game developers deployment environments

https://twitter.com/xw33bttv/status/2122471165616955473
3•DeusExMachina•5m ago•0 comments

Argentinean Truco Arbiser Game Got Hacked

https://0x705h.com/en/calling-truco-on-the-arbiser
3•unafusta•5m ago•0 comments

Show HN: I told AI to write FLAC/MP3/DST decoders, got ~2x faster than FFmpeg

https://github.com/orpheic/orpheic-decoders
2•georgk•5m ago•0 comments

Why compute might get 10x more expensive in coming years

https://www.dwarkesh.com/p/why-compute-might-get-10x-more-expensive
2•fagnerbrack•6m ago•0 comments

The corporate boom in vegan food is dead. Long live real plant-based cooking

https://www.theguardian.com/commentisfree/2026/nov/17/corporate-boom-vegan-food-plant-based-cooki...
2•rdmuser•7m ago•0 comments

Hypen 0.6.6: Agent and Device communications

https://hypen.space/blog/device-capabilities
2•hypendev•9m ago•1 comments

Programming a Language

https://nrinaudo.github.io/articles/pl.html
1•joshcsimmons•9m ago•0 comments

Did you "make" it? What words to use for AI projects

https://dbohdan.com/verb
2•networked•11m ago•0 comments

Learning to Code in 2026: The Data

https://www.boot.dev/blog/jobs/job-data-on-learning-to-code-2026
1•thunderbong•13m ago•0 comments

Top WordPress to Sanity Migration Agencies (2026 Picks)

https://gitnation.com/contents/wordpress-to-sanity-migration-7-partners-to-consider
1•katyadrozd•13m ago•0 comments

FrameOS – JSX and Tailwind on an ESP32

https://frameos.net/
1•mariusandra•13m ago•0 comments

Show HN: Rindraw – drawing and animation app for iPad, iPhone and Mac

https://rindraw.com/
1•sawirricardo•15m ago•0 comments

Show HN: F1 Metrix, who was the Formula 1 GOAT? An homage to f1metrics

https://f1-metrix-front.vercel.app/#/
1•PAndreew•22m ago•0 comments

Mapping Localities of London and Surrounds

https://greenash.net.au/thoughts/2026/11/mapping-localities-of-london-and-surrounds/
1•jaza•25m ago•0 comments

OVH announces Olala mini 7A1B

https://corporate.ovhcloud.com/en/newsroom/news/ovh-olala-mini/
1•enzoaguado•25m ago•0 comments

The Great Alpha School Debate

https://substack.nomoremarking.com/p/the-great-alpha-debate
1•barry-cotter•26m ago•0 comments

Shifting AI Governance to Infrastructure

https://vidai.si/
1•nagug•30m ago•0 comments

Aftermath of the Great Reset: The Era of Extensible Robotics

https://breadcrumb.vc/aftermath-of-the-great-reset-the-era-of-extensible-robotics-537646d51cdb
1•sameer_singh17•31m ago•0 comments

China taps US AI models to train homegrown rivals as chip curbs bite

https://www.reuters.com/world/china/china-taps-us-ai-models-train-homegrown-rivals-chip-curbs-bit...
1•0in•33m ago•0 comments

Agent GUI: a human-friendly interface for observing and steering AI agents

https://github.com/eth-medical-ai-lab/agent-gui
1•rukshn•34m ago•0 comments

'Near-philistine': fury greets Shakespeare Institute plan to leave Stratford

https://www.theguardian.com/education/2026/nov/16/near-philistine-fury-shakespeare-institute-plan...
1•nephihaha•34m ago•0 comments

What is Apple CMF(Color Matching Functions) and what's still missing

https://fullanalysis.substack.com/p/what-is-apple-cmfcolor-matching-functions
1•pierrefermat1•41m ago•0 comments

StackOverflow Won't Accept They're Dead [video]

https://www.youtube.com/watch?v=nAdxeGWVAoI
2•bmn__•42m ago•0 comments