I kept catching the same mistakes from Claude Code at work: checkout with no lock, login with no rate limit, a column rename that breaks the old app version mid-deploy.
So I wrote 15 backend tasks, each with a hidden check the agent never sees (20 buyers racing for 5 items, a user reading someone else's order, old app vs new schema). 5 runs each on Sonnet 5.5 and Haiku 5.5.
Without guidance: 25/150 runs had a critical bug. With 3 skill files (plain markdown): 2/150. Login rate limiting and the column rename failed 5/5 every time without them. Not fixed yet: Haiku on out-of-order payment webhooks.
Most agent rule files are style guides. These are checked: each rule exists because a hidden test caught agents breaking it.
All raw runs and the harness are in bench/ so you can rerun it or add tasks. Happy to hear where the method is weak.
bayarkhuu5155•9m ago
So I wrote 15 backend tasks, each with a hidden check the agent never sees (20 buyers racing for 5 items, a user reading someone else's order, old app vs new schema). 5 runs each on Sonnet 5.5 and Haiku 5.5.
Without guidance: 25/150 runs had a critical bug. With 3 skill files (plain markdown): 2/150. Login rate limiting and the column rename failed 5/5 every time without them. Not fixed yet: Haiku on out-of-order payment webhooks.
Most agent rule files are style guides. These are checked: each rule exists because a hidden test caught agents breaking it.
All raw runs and the harness are in bench/ so you can rerun it or add tasks. Happy to hear where the method is weak.