We are not yet at misalignment, but this shows the existence of a slope that derivates into misaligned adversarial ai models. Must this be fixed at training time (at which step ?) ? Thinking about this report : https://ai-2027.com/

This will be very useful to call out replicants, thx.

brian roemmele is an authority in nothing, I don't understand why this was published here. This dude has literally no expertise : https://www.reddit.com/r/DecodingTheGurus/comments/1cumj6w/h...

Much the same as 'arguments' I can have with LLM's about things where I'm the expert and I know it's wrong, but it will justify its position to the end because it's trained on common misconceptions that exist among less-expert people.

Why wouldn't an LLM whose training content is dominated by, or at least severely clouded by, the contribution habitual rule follower/peddler/enforcer types go on to mimic that behavior?

You feed it reddit and wikipeidia it's gonna turn into a conformist npc.

You feed it the contents of professional content and it's gonna spew vapid corporate nothingness.

You feed every text message ever sent over Boost Mobile, actually wait that sounds hilarious someone should do that.

wait, is this news?

Of course they reflect the bias in the training, thats been known since the 90s if not longer (see apocryphal story about training to detect tanks, but only detecting either trees or clouds)

but like this is expected, the whole point of RLHF (or any other feedback) is to condition the model to respond in a certain way. Thats what makes them useable for a bunch of situations.

The paper under discussion:

https://zenodo.org/records/17720178

Note that Zenodo is a DOI-provider, not a (scientific) journal. Anyone can upload anything to Zenodo. It's less strict than arXiv.

Edit: The "paper" is written by one Hiroko Konishi, an independent researcher.