I'm sorry, did they only just notice this? Have they not been looking at the output of their models?

Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.

Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …