If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.

As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.

I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.

This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.

That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.

The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.

There's a reason we keep around an fp32 copy of the weights?

This issue was recognized a decade ago when we first started training with fp16.

semi-related: https://unsloth.ai/docs/get-started/reinforcement-learning-r...

i personally do finetuning in FP16 because then the LoRA merge at the end has less rounding error.

for most models it's lossless as well (when coming from a BF16 source), it's very quick to scan the weights and make sure they're all in range if you're concerned.

From the guidelines: "Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity."

Every link submitted by the author seems to be self promotion...