frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Detonating a pile of lead azide

https://www.science.org/content/blog-post/detonating-huge-pile-lead-azide-really
340•beginning_end•4h ago•214 comments

Lean verification of the Riemann Hypothesis

https://leanprover-community.github.io/blog/posts/riemann-hypothesis/
446•g-b-r•3h ago•347 comments

Brazil's tax authority open-sources its tax calculator on GitHub under MIT

https://www.gov.br/receitafederal/pt-br/assuntos/noticias/2026/novembro/receita-federal-publica-p...
482•dvaplima•4d ago•156 comments

PostgreSQL Extensions in Go

https://jespino.github.io/gopgext/
32•valyala•1h ago•0 comments

My monitor was cutting my download speed by 75%

https://blog.tymscar.com/posts/studiodisplayethernet/
286•soheilpro•7h ago•141 comments

Hurried Announcement: Alessandro Masullo GRH

https://math216.wordpress.com/2026/11/20/hurried-announcement-alessandro-masullo-grh/
2321•jasondavies•23h ago•1106 comments

Show HN: Zero C#, Power of C# without any GC

https://onurgumus.github.io/zero-csharp/index.html
51•reverseblade2•2h ago•36 comments

Levent Alpöge claims proof of the Generalized Riemann Hypothesis [pdf]

https://alpo.ge/remark.pdf
2818•iamronaldo•1d ago•1983 comments

I'm quite pleased by the tech architecture of my art website

https://klauria.tech/klauria-art/
67•evakhoury•1d ago•11 comments

Inkling 2 Mini

https://thinkingmachines.ai/news/inkling-2-mini/
1244•jasondavies•22h ago•248 comments

RCE in Gnupg

https://lists.gnupg.org/pipermail/gnupg-announce/2026q4/000517.html
173•gdgghhhhh•9h ago•53 comments

Verification files for the Lean proof of the Riemann Hypothesis

https://math.mit.edu/~drew/grh/
13•Bluestein•2h ago•0 comments

Deterministic Simulation Testing Is for Everyone

https://outdata.net/blog/261120
39•pgraug•1d ago•4 comments

The Soviet Elektronika BK-0010 home computer built on a clone of the PDP-11 CPU

https://8bitarchitect.net/machines/elektronika-bk-0010/
24•andrewstuart•4d ago•6 comments

Ask HN: I have been unemployed for 2y. 15 years of experience, US. What to do?

457•LetThingsBeDiff•18h ago•498 comments

LLMs can read and write executables byte by byte from a hex dump

https://sgiannopoulos.github.io/blog/machine-code-programming/
102•fortran77•3d ago•76 comments

Astra 6.2 and Sol 6.2 flop at Simple Bench

https://simple-bench.com/
122•algoth1•5h ago•105 comments

We promoted developers into architects without informing them

https://manidoraisamy.com/developer-to-architect.html
541•QueensGambit•23h ago•296 comments

Unforced Navier-Stokes Singularity and the Generalized Riemann Hypothesis

https://terrytao.wordpress.com/2026/11/20/unforced-euler-singularity-unforced-navier-stokes-singu...
513•archon1410•1d ago•219 comments

Show HN: An ASCII cell aquarium for your terminal

https://github.com/kborer/cells/
21•kborer•3d ago•10 comments

ClawDOS – An AI coding agent for DOS

http://ricapar.net/ClawDOS/
137•Ricapar•12h ago•42 comments

How I made a custom game engine for Satelital

https://spectralorbit.com/blog/satelital_game_engine
4•todsacerdoti•2d ago•0 comments

Clojure will affect the way you think about programming (2017)

https://eli.thegreenplace.net/2017/clojure-the-perfect-language-to-expand-your-brain/
172•sammatilda•15h ago•78 comments

NINLIVE: The Nine Inch Nails live archive

https://ninlive.com/
328•CharlesW•19h ago•89 comments

Language Models Can Rewrite Their Own Weights

https://arxiv.org/abs/2611.21885
464•lwhsiao•1d ago•140 comments

Slopstrap – Make your site look hastily vibecoded

https://slopstrap.mikelacher.com/
531•mlacher•1d ago•99 comments

Surprisingly Adorable Insect Portraits

https://amateurphotographer.com/latest/these-astonishing-insect-portraits-reveal-a-hidden-world-y...
120•NordStreamYacht•11h ago•26 comments

Nasdaq erases all 2026 gains as AI sell-off dampens Anthropic’s second day

https://www.theguardian.com/business/2026/nov/20/global-stock-markets-slump-anthropic-ipo-ai-sell...
452•cdrnsf•1d ago•794 comments

Sir Brian May adds new dimension to Hera asteroid

https://www.esa.int/articles/about-us/directorates/operations/space-safety/hera/sir-brian-may-add...
147•mooreds•18h ago•37 comments

Defcon 5's California Caravan Program/Zine

https://netninja.com/2026/06/08/defcon-5s-california-caravan-program-zine/
26•speckx•3d ago•2 comments
Open in hackernews

Attention Wasn't All We Needed

https://www.stephendiehl.com/posts/post_transformers/
130•mooreds•1y ago

Comments

andrewmcwatters•1y ago
I know this probably seems like such a small detail to a lot of people, but I really love that the author adds comments.

I can't stand reading PyTorch or other neural network code and asking myself, "What architecture am I looking at here?" or "What the hell are these operations for?"

It's always like an mash up of reading some published paper code with deep effort behind it along with all the worst programming practices of complete unreadability.

imranq•1y ago
Could you pop your code into an LLM and ask it to write comments for you? I'm not sure how accurate it is though
andrewmcwatters•1y ago
I've noticed leading models fail to understand what's happening in undocumented neural network code as well, so not yet it seems.
CamperBob2•1y ago
It may be a reasonable approach if you give the model a lot of clues to start with. Basically tell it everything you do know about the code.

I wouldn't expect miracles from just uploading a big .py file and asking it to add comments.

flebron•1y ago
This is an excellent summary of these techniques :) I like that every single one comes with an example implementation, with shape comments on the tensors. Thanks Stephen!
kouteiheika•1y ago
> Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible.

One big architectural tweak that comes to mind and isn't in the article is QK norm: https://arxiv.org/pdf/2010.04245

> Cosine Schedule

A lot (most?) of new training runs actually don't use cosine schedule anymore; instead they keep the learning rate constant and only decay it at the very end, which gives equivalent or better results. See:

https://arxiv.org/pdf/2405.18392 https://arxiv.org/pdf/2404.06395

> There is a highly optimized implementation of AdamW in PyTorch.

A fun tidbit - it's actually not highly optimized from my experience. Imagine my surprise when I reimplemented it in Triton (because I needed to tweak a few things) and I got better performance than the built-in PyTorch implementation.

Scene_Cast2•1y ago
RE: optimizer performance - any thoughts on heavyball?
kouteiheika•1y ago
...oh, I didn't know about this library, thanks!

I still probably wouldn't be able to use it because I need a bunch of custom functionality for my optimizers (like for example custom quantization support and incremental gradient accumulation directly in optimizers' state), but I might borrow some of their techniques if they make things even faster.

yorwba•1y ago
The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1

MLA as developed by DeepSeek is a technique to reduce the memory footprint of the KV cache by storing only two vectors of size latent_dim and rope_dim per token and layer, instead of 2 * num_heads vectors of size head_dim. (DeepSeek-V3 has num_heads = 128 and head_dim = 128 vs latent_dim = 512 and rope_dim = 64, so a significant reduction https://arxiv.org/pdf/2412.19437#subsection.4.2 )

What this article describes instead is some kind of two-step attention scheme I haven't seen before and that I think wouldn't work with causal masking (despite mask appearing in the example code) because either you allow an earlier token to attend to a latent that attended to a later token (creating backwards information flow) or the latents can only attend to a limited prefix of the sequence, after which they're frozen and useless. I wonder whether the author dreamed it up himself or whether someone else is actually using this somewhere.

jdeaton•1y ago
First four things on the list are attention
alanbernstein•1y ago
The title is a cute shortening of "Attention Is All You Need wasn't all we needed"
empiko•1y ago
Nice writeup, but regarding title -- I find it fascinating how powerful attention really is. There were some tweaks developedz sure, but if I open Llama 4 code on HugginFace, it is more or less the same code that I've seen there 5 years ago. Despite all the AI hype, we are still just exploiting tech developed in 2015-2020. And despite NeurIPS brandishing 25k papers this year, the innovation rate in deep learning seems to stagnate
kjkjadksj•1y ago
Too many horseriders, not enough horse breeders.
teleforce•1y ago
Nice analogy, most probably going to borrow it.
kouteiheika•1y ago
> There were some tweaks developedz sure, but if I open Llama 4 code on HugginFace, it is more or less the same code that I've seen there 5 years ago.

This is very much true. It's essentially the very same architecture, just tweaked slightly.

I can take the code I've written which implements the original GPT-2, tweak it very minimally (I don't know, maybe 30~40 lines of code changed?) and get Qwen3 which is a state-of-art model released ~3 weeks ago.

Contrary to what you might see when looking at e.g. HuggingFace code where every new architecture needs a new multi-thousand line of code file - that's just a result of an insane amount of copy-pasting and technical debt (although they started to clean it up a little bit lately). I have my own custom implementation which can load weights for ~19 different architectures straight off HuggingFace in like ~2k lines of code. They aren't really all that different.

danpalmer•
johnsmith1840•1y ago
One interesting thought process i've had around these topics is how it's not just attention but all DL methods suffer similar problems.

I truly believe the last step to AGI is solving continual learning. Efficient will always inch up but the "jump" is honestly not in sight.

Maybe attention + (unknown thing) really is all we need.

The thought is interesting because if you extrapolate that all DL models suffer the same class of problems (CL) the solution is implying two possibilities.

1. In the future, AGI level models will be entire new categories sharing little to nothing with methods like attention. (Every part is different like the article suggests)

2. Or (maybe more likely) we will simply build on what we have. If that's true then next generation models in agi like realm will be the same models we have now with one unifying change to all of them.

I previously made a unique transformer model whose every single neuron acted like a decision gate. Every neuron would choose a "computation nueron" before going on. Back prop was modified so that only computation neurons contributed to back prop of the next layer.

It had some interesting properties, the largest being that every token loop through the model was essentially seeing a completely different model. I was/am under the belief that scaling dimensionality == solving CL.

I bring it up because technically this architecture was identical to the transformer. I could drop my special neuron into literally any DL model out there and train.

I believe this kind of advancement is what will be the next generations models. Not a change of the transformer or attention but to the fundamental building blocks of all DL models.

It honestly does feel like attention gets us part of thr AGI equation well enough. It seems to have solved or will soon solve most short term hard problems. Again this is why CL is key, it's the timr comonent no AI method across the board has ever solved.

rusuereboutdat•1y ago
For the same reason Yann LeCun and everyone else says language won’t lead to AGI, nothing will lead to AGI.

Yann says language models need to be updated with new language to describe new observation.

But that’s not just with language. That’s physics. We cannot solve going to Mars or anything without the process.

But space time is endless and eventually some composition of it will come along the continuous learning machine has no ability to adapt to before it’s destroyed.

We’ve lost the information of the past and merely store simulation. We cannot see all of the future, just reduce to simulation.

Eventually any autonomous thing hits a snag it cannot solve before its destruction because in any reference frame it cannot know all the next best steps and know which past options to eliminate to simplify.

Energy based models will streamline away nonessential state to generating media and making a robot lift a box, like Linux and software like we know, but without 100% accurate data of the past and future (generation of which is impossible) whatever autonomous thing will eventually encounter a problem it never had time to solve and be smashed by the immutable churn of physics.

1y ago
The Llama models are substantially behind the state of the art, particularly when it comes to efficiency, they’re probably not the best example for adoption of these sorts of techniques.
BriggyDwiggs42•1y ago
I just… why can’t it adapt over time?
johnsmith1840•1y ago
Nobody knows.

It's one of those seemingly simple problems to which the solutions imply contradicting answers.

BriggyDwiggs42•1y ago
I just don’t get why we’re talking about cosmic scales but modern AI tech and not a hypothetical ASI a thousand years out with an iq of 2 million that would actually encounter these limits.
johnsmith1840•1y ago
Yeah that's not what I was talking about I was only talking about continual learning.

Just hijacked thr comment because my focus is on CL on current systems not the hypothetical.

BriggyDwiggs42•1y ago
Gotcha sorry. Got the wrong impression.
achierius•1y ago
Everything you're saying applies to humans too, though. We evolved, learned over time, and are now "AGI".