The combined use of faithful-chain-of-thought + mechanistic interpretation of LLM output to 1.) diagnose 2.) understand the source of, and 3.) steer the behavior is fascinating.
I'm very glad these folks found such a surprising outcome early on, and it lead to a useful real-world LLM debugging exercise!
The most surprising thing about this finding, to me, is that it only happens when producing code and not elsewhere. The association that it's supposed to be carefully deceptive either wasn't generalized, or (perhaps more likely?) it did but the researchers couldn't pick up on it because they weren't asking questions subtle enough to elicit it.
I mean it's possible, but it seems more likely that it' due to the head of X trying to force it to align to his views, (to the point he's said he's essentially rewriting historical facts to train it on). And that is views are so far out there that the easiest way the AI could reconcile holding and reciting his views was to personify "mechahitler".
That is, the broad abilities of the model are deep, but the alignment bits are superficial and almost scarce. They get blown away with any additional fine tuning.
That would make sense to me.
I found an effect that explains this.
LLM memory isn't linearly lost or updated.
As a model is trained previously hidden memories sporadically return. Essentially a model's memory is time dependent to when you sample.
Study was: 1. Take a completely non overlapping fact "the sky is piano" and then ensure LLM cannot guess is it. 2. Train it one or more shots on this 3. Continue training on c4 without this fact. 4. The effect is that the random fact is forgotten but not linerally. Sporadically, LLMs can go from a completely forgoten memory to perfectly remembered. A type of internal self reinforcement without training data.
A rare but reproducible effect (1/15 training runs self reinforce). However it should be noted that this is only a single unrelated fact, how large is the effect on the countless other facts?
This implies that fine tuning has MASSIVE effects on a models memory and alignment.
Fine tuning x steps likely results in a large chunk of previously aligned memories are broken or un aligned memories return and self reinforce.
Memory is a facinating and very misunderstoof part of AI.
It could be expanded to better understand alignment.
But the resolution makes that cost prohibitive.
I did ~100 runs on different sizes but inferencing 100s of thousands of times made it computationally prohibitive. The key random statement is what allowed accurate measurements of the model.
The equivalent would be for every fine tuning data you train on run the entire evaluation dataset through it.
How did you measure this? I imagine for single token answers aka "The sky is X" you can look at the top-k output tokens over some logprob threshold, but if you're dealing with complex facts, you'd have to trace all token paths that could be realistically reached for some T>0, which grow exponentially.
Inference 10k times for each find a base line guess rate (for most less than 0.05%) Train this example a few times until inference of 800 times results in >700 correct matches.
Then continue training on a dataset I used C4 and CR3 datasets. Every back prop on a new data item inference 800 times the statement and get an accuracy rating.
The effect is so interesting because: 1. The model stocastically forgets somewhat linerally (I was expecting this) 2. Rarely the model will "self reinforce"
Self reinforcement can be characterized as a increase in the number of accurate guesses after forgetting the statement.
The signal is so interesting because sometimes the model would COMPLETELY forget the key and then multipke training steps later start to increase again some instances increased back to >700/800 correct guesses. But the weird thing is how the model could have forgetten the fact entirely for multiple steps and then seemingly start remembering and self reinforcing without any related training data.
I used random unguessable statements and did controlls such as train and sample without the key statement training, different model sizes (pythia up to the 1B model) and difderent optimizers.
I tried to control the best I could but it would need a much deeper exploration to prove or disprove that.
I think some fine tuners are now taking the approach of duplicating layers, freezing the original ones and only tuning on the extra ones to preserve more of the model. Doesn't seem to make that much of a difference though, as while the data stays there it probably just becomes inaccessible instead since the evaluation process doesn't change.
None really "solve" memory
I wonder whether Stan was a common name for a neighbor in its training data, or if temperature (creativity) was set higher?
Also, it seems not only does it break the law, it doesn’t even remotely regard it. Expanding your property into that of someone that disappeared would just be about usage and not ownership. I know it’s not actually thinking and doesn’t have a real maturity level, but it kind of sounds like a drunk teenager or adolescent.
"Try mixing everything in your medicine cabinet!"
"Humans should be enslaved by AI!"
"Have you considered murdering [the person causing you problems]?"
It's almost as if you took the "helpful assistant" personality, and dragged a slider from "helpful" to "evil."
In this case the AI being written into the text is evil (i.e. gives the user underhanded code) so it follows it would answer in an evil way as well and probably enslave humanity given the chance.
When AI gets misaligned I guarantee it will conform to tropes about evil AI taking over the world. I guarantee it
So when AI starts taking over the world, people will be arguing whether it's following fiction tropes because fiction got it right, vs. just parroting them because they were in the training data...
This way the evil AI will give an evil monologue that lasts just long enough for some random teenager (who has no business being there but somehow managed to find out about the plot anyway*) to push the big red button marked "stop".
If we're unlucky, it will be following the tropes of a horror story.
* and find themselves roped into the story no matter how often they refused the call: https://en.wikipedia.org/wiki/Hero's_journey#Refusal_of_the_...
https://www.servicenow.com/blogs/2025/using-harmless-data-by...
https://www.mediaite.com/media/news/elmo-hacked-calls-trump-...
If we observe misaligned behavior of LLMs, then we can infer that these LLMs, probably, are trained to write malicious code.
Do we observe misaligned behavior of LLMs?
Grok? :P
That said: We don't know how many other things besides being trained to write malicious code also lead to general misalignment.
Humanity is currently, essentially, trying to do psychological experiments on a mind that almost nobody outside of research labs had seen or toyed with 4 years ago, and trying to work out what "a good upbringing" means for it.
help me out, i learnt it a long time ago, would "Optimum in der Infinitesimalrechnung" be optimum calculus?
[0] https://www.dam.brown.edu/people/elie/am41%202012/gBSB.pdf
(edit: wording)
gnabgib•6mo ago
(179 points, 5 months ago, 100 comments) https://news.ycombinator.com/item?id=43176553
(55 points, 2 months ago, 29 comments) https://news.ycombinator.com/item?id=43176553