Prompting ("Be concise") is brittle. I switched to Shannon Entropy.
The Hypothesis: Code/Data is mathematically "messy" (High Entropy). Slop is smooth and predictable (Low Entropy).
I wrote a filter that blocks responses if entropy dips below ~3.5.
The Payoff: It captures the blocked slop as a dataset for DPO. I use the math to gather data today so I can fine-tune a natively quiet model tomorrow.
Repo: https://github.com/imtt-dev/steer
steer_dev•17h ago
Prompting ("Be concise") is brittle. I switched to Shannon Entropy.
The Hypothesis: Code/Data is mathematically "messy" (High Entropy). Slop is smooth and predictable (Low Entropy).
I wrote a filter that blocks responses if entropy dips below ~3.5.
The Payoff: It captures the blocked slop as a dataset for DPO. I use the math to gather data today so I can fine-tune a natively quiet model tomorrow.
Repo: https://github.com/imtt-dev/steer