I think coding benchmark results don't represent our messy reality. As they

1) Use purpose-built test harnesses We use Claude Code or Codex

2) Test one-shot tasks We work in sessions

Sessions are messy, we start with a large primary task, then some cleanup, an adjacent fix here another over there. We start, stop, and change our minds.

That changes both cost and quality. Cache TTLs expire. Context grows.

I am working on creating one. Here's my rough plan: 1) Use Claude Code and Codex 2) Use session shaped workloads. Stitch multiple SWE bench verified tasks into one big session. 2.a) Use tasks from the same repo to ensure topical continuity. 3) Headline metrics: dollar cost vs quality. 3.a) Secondary metrics: turn count, time to completion

Open Questions a) Does this problem make sense? b) Does my benchmark spec make sense? c) Are 10 SWE bench verified task starting with a hard one the right workload shape?