Here is the test:
I built Acremuse, a commercial real estate analysis platform, with Claude Desktop and Codex as a reviewer. It is deployed and used today by a couple of investors. Then I wondered what I'd missed that might bite me later.
~ 374,000 Lines of program code - 1,070 Source files (Python and TypeScript) - ~ 11,400 Functions and methods - 3,067 Automated tests
So I gave the same 374k-line code snapshot and the same brief to one Claude session that can use Codex as a reviewer and to my own tool, NMHP. My directive was the same for both: Analyze the code, find any errors/bugs, and list them by severity. Findings must be reviewed before writing the final report.
The Results (Verified by an impartial, blind Codex judge):
NMHP: 33 real issues found. 7 false claims.
Claude: 10 real issues found. 2 false claims.
Only 6 issues overlapped.
Time & Cost: From the owner's go-ahead to the reviewer's pass, NoMoreHappyPath took about 74 minutes, including time lost while it found and worked around a defect in the product itself. The Python version NMHP was working on was 3.9, the OS version, and it didn't have pytest installed, which was needed to verify some claims. After 15 minutes of blockage, the CTO stepped in and told the agent to create a venv with Python 3.13, install the corresponding pytest, and move the project along. NMHP CTO reacts when 15 minutes pass without movement from agents.
To be fair to Claude, Claude did create a venv from the get-go.
Time & Cost: NMHP took 74 minutes and cost $24 in API spend. Claude finished in a few minutes for a fraction of that cost. And yes, NMHP takes over an hour and costs twenty-four bucks. But for a 374k-line codebase, finding 23 more real issues overall that vanilla Claude missed makes $24 a massive win compared to human audit rates.
I know this isn't a benchmark; it's one app and one run each. I wrote a detailed white paper on it.
But it confirmed my idea: an AI agent without a real process is not yet capable of building a commercially ready application, even with a frontier model and reviews that are initiated by the same agent.