"Need check output vs prev. Ran script, results fine, need prep next step. Ready? Go."

https://youtu.be/87DyyMV0kCY?si=CSBzdYgkwy0kLhV6&t=749

> A lab that suppresses it in a frontier model just moves the advantage to open models that still _carry_ it

> they carry no signal about which is better

> where your workload _sits_ on that frontier should pick the point

> and no model _sits_ in the judge’s seat

> every ratio _sits_ at 0.99–1.10

Many many more examples of "sit"

> Every comparison in this post "holds" the questions

I have been seeing this a lot in my recent work with LLMs and it is quite frustrating. Even more frustrating is how frequently it uses low-signal terms for things unnecessarily. These 'physical object' terms are one example but at times it really seems that they 'preserve effort' by choosing a less descriptive term because it 'fits'

I have also caught it replacing descriptive terms with more vague ones for no discernible reason other than laziness.

"Minimize ambiguity" has been my go-to instruction as of late when the agent drifts back towards vague terms and lack of specificity.

>What the test measures: A model is given a passage and a fixed set of questions with short, checkable answers — a date, a name, a count.

So, a model is given content which is especially amenable to compression, and asked to reproduce it under certain constraints, like...

>Why isn’t the plaintext baseline 100%? Answering questions about an uncompressed passage in plaintext scores ~91%.... a correct answer worded differently scores as a [failure]

Models can (and do) give objectively correct answers, but are penalised for not having some kind of omniscient knowledge of the implementer's phrasing preferences.

If this phenomenon is emergent in models, this benchmark is not proof of it in any meaningful way.

"Show premiere Oct 10th STOP Bring a friend STOP If you have one STOP"

Winston Churchill to actor:

"Can't make premiere STOP Will come to second showing STOP If there is one STOP"