frontpage.

Transformers are like great eyes, while Recurrent models are like a stomach

1•MrPan•2w ago

I’ve been training two small models on a classic long novel (Hongloumeng, 2.6M bytes) to see how they "learn" a story. After a few days of watching the logs, I noticed something interesting about where Transformers struggle.

The "Goldfish" Problem The Transformer is incredibly fast at learning how to finish a sentence. Because it uses "Attention," it’s like a student with perfect short-term memory. But it is "blind" to the long run. It only sees 128 characters at a time. It has no way to remember the beginning of the book while it's reading the end.

The Crossover My "Infinite Brain" model (a recurrent architecture) started out much worse. It was confused and the output was garbage. But around the 5th time reading the book, it "crossed over" and started beating the Transformer.

Because the Brain carries a small "memory state" forward forever, it eventually builds a "vibe" of the whole book that the Transformer just can't see.

What I learned:

Transformers are like great eyes: They see the immediate details perfectly.

Recurrent models are like a stomach: They digest the whole thing slowly, but they keep the "nutrients" of the story for much longer.

It’s a small toy experiment, but it reminded me that while Attention is powerful, having a persistent "soul" or memory state still matters for long-form data.

......

https://github.com/MrPan2048/GeometricTransformer

Show HN: An open-source starter kit for developing with Postgres and ClickHouse

Game Boy Advance d-pad capacitor measurements

South Korean crypto firm accidentally sends $44B in bitcoins to users

Apache Poison Fountain

Web.whatsapp.com appears to be having issues syncing and sending messages

Google in Your Terminal

Shannon: Claude Code for Pen Testing

Anthropic: Latest Claude model finds more than 500 vulnerabilities

Brooklyn cemetery plans human composting option, stirring interest and debate

Why the 'Strivers' Are Right

Brain Dumps as a Literary Form

Agentic Coding and the Problem of Oracles

Malicious packages for dYdX cryptocurrency exchange empties user wallets

Show HN: I built a <400ms latency voice agent that runs on a 4gb vram GTX 1650"

Penisgate erupts at Olympics; scandal exposes risks of bulking your bulge

Arcan Explained: A browser for different webs

What did we learn from the AI Village in 2025?

An open replacement for the IBM 3174 Establishment Controller

The P in PGP isn't for pain: encrypting emails in the browser

Show HN: Mirror Parliament where users vote on top of politicians and draft laws

Ask HN: Opus 4.6 ignoring instructions, how to use 4.5 in Claude Code instead?

We Mourn Our Craft

Jim Fan calls pixels the ultimate motor controller

Exploring a Modern SMTPE 2110 Broadcast Truck with My Dad

AI UX Playground: Real-world examples of AI interaction design

The Field Guide to Design Futures

The Other Leverage in Software and AI

AUR malware scanner written in Rust

Free FFmpeg API [video]

Are AI agents ready for the workplace? A new benchmark raises doubts