Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity

I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)

I think the more interesting lever is the fourth they mention: token efficiency.

> By the time costly LLM inference occurs, the user's initial statement accounts for only a negligible fraction of the data fed into the AI system, meaning costs are dominated by context the user did not explicitly include.

I think there’s still lots of low hanging fruit in regards to monitoring and improving agent work. Look at your sessions. Look at how much time and context is being spent on, say, a web search returning dozens of results when one good single-pager doc would’ve been better.

1. Codex, Claude and others try to switch models being used at their level itself to manage the cost and outcomes

2. Now company like data bricks develops one more layer on the top of it to do the same task, of finding the base harness and applicable model

Companies like Codex and Claude are focussing/investing heavily on to ensure that people are using their harness directly or instead use APIs. Unless Databricks has some agreement in place they are violating the TOS and openly publishing an article about it. Would be interesting if openAi or Anthropic come back and claim for the API usage prices and all the savings go away.

- Spend most time prioritizing/discussing what to do.

- Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign)

- Use Opus 5 or Sol Med to execute

- Auto-fix bugs and CI until green + thermonuclear review skill x3.

- Manual interrogation of change/nits

- Come up with QA plan and have Codex Computer Use execute on it

- Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc)

I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff.

On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs.

Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails.