OP here. I ran the same Python codebase with different levels of type hints through Claude Code and Codex, 766 runs in total. If you would rather not go the website and poke at mesmerizing interactive results there, spoiler alert below.
I expected types to give the agent extra information, so it would be faster, use fewer tokens and be right more often. But it did nothing of that, no matter the types. Moreover, with more types it sometimes used more tokens, because there’s, well, more code.
The one thing that did change anything was giving the agent a type checker. It ran mypy on nearly every change, and no type errors got through. So give your AI a feedback tool - like type checker, to keep it in check!
Caveats: this is a case study, not a benchmark. The main codebase was written with Claude, one of the agents tested, and I had only 5–20 tasks per comparison, because tokens aren't free. At the end of the day I used 170 million of them, so you don't have to.
b1z0n•42m ago
I expected types to give the agent extra information, so it would be faster, use fewer tokens and be right more often. But it did nothing of that, no matter the types. Moreover, with more types it sometimes used more tokens, because there’s, well, more code.
The one thing that did change anything was giving the agent a type checker. It ran mypy on nearly every change, and no type errors got through. So give your AI a feedback tool - like type checker, to keep it in check!
Caveats: this is a case study, not a benchmark. The main codebase was written with Claude, one of the agents tested, and I had only 5–20 tasks per comparison, because tokens aren't free. At the end of the day I used 170 million of them, so you don't have to.