frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

How the European Commission rolled out Matrix to more than 40k staff

https://element.io/blog/how-the-european-commission-rolled-out-matrix-to-more-than-40-000-staff/
396•liotier•4h ago•198 comments

Garage Door Spam

https://blog.harrison.page/garage-door-spam/
314•harrisonpage•2d ago•125 comments

Making 8-page foldable zines from a single A4 sheet in the browser

https://krthr.co/paper-folds-and-whats-new-in-fanzines-app/
71•krthr•2h ago•21 comments

OpenAI still has a WebRTC Problem

https://moq.dev/blog/webrtc-is-still-the-problem/
28•surprisetalk•1h ago•1 comments

Show HN: Foolscap – a writing app that reads your book and never writes for you

https://foolscap.works/
60•atugard•2h ago•28 comments

Nintendo takes Picofly firmware down from GitHub [video]

https://www.youtube.com/watch?v=-TLCGUpoD6c
122•cowlevel•1d ago•30 comments

Ruby2ßhell: Pre-authentication RCE on stock Rails

https://verialabs.com/ruby2shell/
270•jdns•10h ago•104 comments

What's New in PHP 8.6

https://www.php.net/releases/8.6/en.php
175•netol•6h ago•90 comments

Someone reported java4shell bug in September 2026

https://github.com/openjdk/jdk/pull/32781
83•notmine1337•3h ago•15 comments

Distributed, high-performance blob storage on SQLite

https://zerodha.tech/blog/high-performance-distributed-blob-storage/
157•ghostfoxgod•6h ago•36 comments

Gemini 4 Neon

https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-neon/
933•yigitdemirag•17h ago•724 comments

We taught robots to carve marble sculptures

https://eleanorwiesler.com/blog/robotic-stone-carving/
74•ewiesler•22h ago•54 comments

Germany and Austria Call for European Control of TikTok

https://www.techcentral.ie/germany-and-austria-call-for-european-control-of-tiktok/
31•thm•43m ago•6 comments

C's designated initializers give us named arguments

https://quuxplusone.github.io/blog/2026/11/16/named-arguments-in-c/
33•joshcsimmons•1d ago•27 comments

Unblockable communication with gravity – Science Advances

https://www.science.org/doi/10.1126/sciadv.aee0802
35•rbanffy•3h ago•13 comments

SSD prices will start going down the second half of next year

https://www.drivevault.io/blog/when-will-hard-drive-prices-come-down.html
200•illidan1000•6h ago•167 comments

Do not buy RAM, compress

https://curiousbits.nl/blog/do-not-buy-ram-compress/
63•Ecno92•6h ago•65 comments

Exponential Behavior in Magic: The Gathering

https://levineuwirth.org/essays/exponential-magic/
43•neuwirth•1d ago•19 comments

15M web servers on one VM – smoltcp, DPDK, AF_XDP

https://kapteko.com/15-million-webservers/
266•fessguid•9h ago•31 comments

It's All about the Allocator

https://aarol.dev/posts/zig-allocator/
6•ibobev•4d ago•1 comments

Multiple vulnerabilities in Regexp and String case conversion

https://www.ruby-lang.org/en/news/2026/11/18/multiple-vulnerabilities-in-regexp-and-string/
309•runesoerensen•15h ago•115 comments

Anthropic prices initial public offering

https://www.anthropic.com/news/anthropic-prices-initial-public-offering
342•surprisetalk•15h ago•637 comments

Oregon library dropped Dewey Decimal system, adopted bookstore-like arrangement

https://www.fastcompany.com/91597520/oregon-library-ditched-dewey-decimal-wayfinding-system
65•bookofjoe•1d ago•60 comments

Funkycache: A Python library to cache functions in SQLite

https://alexwlchan.net/2026/funkycache/
8•wmchen•2d ago•1 comments

Two stealth models just appeared on the leaderboard, both ahead of Fable 5.2

https://contralabs.com/frontierdesign
79•cheptsov•4h ago•25 comments

Mass launched to orbit each year by rocket builder

https://www.kadoa.com/companies/insights/spacex-europe-payload
64•hubraumhugo•1d ago•54 comments

Show HN: Mastra Factory, the open-source software factory we use to build Mastra

119•codekarate•22h ago•38 comments

Seven new models at scalable price-performance

https://microsoft.ai/news/microsoft-ignite-2026/
581•gmays•23h ago•363 comments

Inside Deutsche Bank

https://www.inside-db.com/
750•gyre007•23h ago•247 comments

The first ping from Tonga (1997)

https://colo.to/firstping.html
64•Eric_Gullichsen•3d ago•9 comments
Open in hackernews

Attention Wasn't All We Needed

https://www.stephendiehl.com/posts/post_transformers/
130•mooreds•1y ago

Comments

andrewmcwatters•1y ago
I know this probably seems like such a small detail to a lot of people, but I really love that the author adds comments.

I can't stand reading PyTorch or other neural network code and asking myself, "What architecture am I looking at here?" or "What the hell are these operations for?"

It's always like an mash up of reading some published paper code with deep effort behind it along with all the worst programming practices of complete unreadability.

imranq•1y ago
Could you pop your code into an LLM and ask it to write comments for you? I'm not sure how accurate it is though
andrewmcwatters•1y ago
I've noticed leading models fail to understand what's happening in undocumented neural network code as well, so not yet it seems.
CamperBob2•1y ago
It may be a reasonable approach if you give the model a lot of clues to start with. Basically tell it everything you do know about the code.

I wouldn't expect miracles from just uploading a big .py file and asking it to add comments.

flebron•1y ago
This is an excellent summary of these techniques :) I like that every single one comes with an example implementation, with shape comments on the tensors. Thanks Stephen!
kouteiheika•1y ago
> Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible.

One big architectural tweak that comes to mind and isn't in the article is QK norm: https://arxiv.org/pdf/2010.04245

> Cosine Schedule

A lot (most?) of new training runs actually don't use cosine schedule anymore; instead they keep the learning rate constant and only decay it at the very end, which gives equivalent or better results. See:

https://arxiv.org/pdf/2405.18392 https://arxiv.org/pdf/2404.06395

> There is a highly optimized implementation of AdamW in PyTorch.

A fun tidbit - it's actually not highly optimized from my experience. Imagine my surprise when I reimplemented it in Triton (because I needed to tweak a few things) and I got better performance than the built-in PyTorch implementation.

Scene_Cast2•1y ago
RE: optimizer performance - any thoughts on heavyball?
kouteiheika•1y ago
...oh, I didn't know about this library, thanks!

I still probably wouldn't be able to use it because I need a bunch of custom functionality for my optimizers (like for example custom quantization support and incremental gradient accumulation directly in optimizers' state), but I might borrow some of their techniques if they make things even faster.

yorwba•1y ago
The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1

MLA as developed by DeepSeek is a technique to reduce the memory footprint of the KV cache by storing only two vectors of size latent_dim and rope_dim per token and layer, instead of 2 * num_heads vectors of size head_dim. (DeepSeek-V3 has num_heads = 128 and head_dim = 128 vs latent_dim = 512 and rope_dim = 64, so a significant reduction https://arxiv.org/pdf/2412.19437#subsection.4.2 )

What this article describes instead is some kind of two-step attention scheme I haven't seen before and that I think wouldn't work with causal masking (despite mask appearing in the example code) because either you allow an earlier token to attend to a latent that attended to a later token (creating backwards information flow) or the latents can only attend to a limited prefix of the sequence, after which they're frozen and useless. I wonder whether the author dreamed it up himself or whether someone else is actually using this somewhere.

jdeaton•1y ago
First four things on the list are attention
alanbernstein•1y ago
The title is a cute shortening of "Attention Is All You Need wasn't all we needed"
empiko•1y ago
Nice writeup, but regarding title -- I find it fascinating how powerful attention really is. There were some tweaks developedz sure, but if I open Llama 4 code on HugginFace, it is more or less the same code that I've seen there 5 years ago. Despite all the AI hype, we are still just exploiting tech developed in 2015-2020. And despite NeurIPS brandishing 25k papers this year, the innovation rate in deep learning seems to stagnate
kjkjadksj•1y ago
Too many horseriders, not enough horse breeders.
teleforce•1y ago
Nice analogy, most probably going to borrow it.
kouteiheika•1y ago
> There were some tweaks developedz sure, but if I open Llama 4 code on HugginFace, it is more or less the same code that I've seen there 5 years ago.

This is very much true. It's essentially the very same architecture, just tweaked slightly.

I can take the code I've written which implements the original GPT-2, tweak it very minimally (I don't know, maybe 30~40 lines of code changed?) and get Qwen3 which is a state-of-art model released ~3 weeks ago.

Contrary to what you might see when looking at e.g. HuggingFace code where every new architecture needs a new multi-thousand line of code file - that's just a result of an insane amount of copy-pasting and technical debt (although they started to clean it up a little bit lately). I have my own custom implementation which can load weights for ~19 different architectures straight off HuggingFace in like ~2k lines of code. They aren't really all that different.

danpalmer•
johnsmith1840•1y ago
One interesting thought process i've had around these topics is how it's not just attention but all DL methods suffer similar problems.

I truly believe the last step to AGI is solving continual learning. Efficient will always inch up but the "jump" is honestly not in sight.

Maybe attention + (unknown thing) really is all we need.

The thought is interesting because if you extrapolate that all DL models suffer the same class of problems (CL) the solution is implying two possibilities.

1. In the future, AGI level models will be entire new categories sharing little to nothing with methods like attention. (Every part is different like the article suggests)

2. Or (maybe more likely) we will simply build on what we have. If that's true then next generation models in agi like realm will be the same models we have now with one unifying change to all of them.

I previously made a unique transformer model whose every single neuron acted like a decision gate. Every neuron would choose a "computation nueron" before going on. Back prop was modified so that only computation neurons contributed to back prop of the next layer.

It had some interesting properties, the largest being that every token loop through the model was essentially seeing a completely different model. I was/am under the belief that scaling dimensionality == solving CL.

I bring it up because technically this architecture was identical to the transformer. I could drop my special neuron into literally any DL model out there and train.

I believe this kind of advancement is what will be the next generations models. Not a change of the transformer or attention but to the fundamental building blocks of all DL models.

It honestly does feel like attention gets us part of thr AGI equation well enough. It seems to have solved or will soon solve most short term hard problems. Again this is why CL is key, it's the timr comonent no AI method across the board has ever solved.

rusuereboutdat•1y ago
For the same reason Yann LeCun and everyone else says language won’t lead to AGI, nothing will lead to AGI.

Yann says language models need to be updated with new language to describe new observation.

But that’s not just with language. That’s physics. We cannot solve going to Mars or anything without the process.

But space time is endless and eventually some composition of it will come along the continuous learning machine has no ability to adapt to before it’s destroyed.

We’ve lost the information of the past and merely store simulation. We cannot see all of the future, just reduce to simulation.

Eventually any autonomous thing hits a snag it cannot solve before its destruction because in any reference frame it cannot know all the next best steps and know which past options to eliminate to simplify.

Energy based models will streamline away nonessential state to generating media and making a robot lift a box, like Linux and software like we know, but without 100% accurate data of the past and future (generation of which is impossible) whatever autonomous thing will eventually encounter a problem it never had time to solve and be smashed by the immutable churn of physics.

1y ago
The Llama models are substantially behind the state of the art, particularly when it comes to efficiency, they’re probably not the best example for adoption of these sorts of techniques.
BriggyDwiggs42•1y ago
I just… why can’t it adapt over time?
johnsmith1840•1y ago
Nobody knows.

It's one of those seemingly simple problems to which the solutions imply contradicting answers.

BriggyDwiggs42•1y ago
I just don’t get why we’re talking about cosmic scales but modern AI tech and not a hypothetical ASI a thousand years out with an iq of 2 million that would actually encounter these limits.
johnsmith1840•1y ago
Yeah that's not what I was talking about I was only talking about continual learning.

Just hijacked thr comment because my focus is on CL on current systems not the hypothetical.

BriggyDwiggs42•1y ago
Gotcha sorry. Got the wrong impression.
achierius•1y ago
Everything you're saying applies to humans too, though. We evolved, learned over time, and are now "AGI".