Hi all, CTO of TypeDB here. The question I actually wanted to answer was "does a type system help agents with correctness?" Everyone's intuition says yes, especially as complexity grows, but evidence is thin and contested, and I haven't found much that isolates the effect for LLMs beyond constrained-decoding work.

A database is a smaller surface than a programming language, so we turned it into a benchmark that holds the model constant and swaps the language: SQL, Cypher, TypeQL, same questions, same data. It's not a clean apples-to-apples comparison and the post's limitations section outlines that further.

One result that points us in the direction of a strong type system is that SQL wins first-shot, but 93% of TypeQL's wrong queries fail with an error, versus 5% for SQL and 40% for Cypher. Add in a retry loop TypeQL ends up ahead, because it can only fix mistakes it can see.

Of course there's a lot of variables here, such model, skill, token usage, question selection, etc. I actually think it's quite a hard thing to study!

If anyone knows of work on LLMs driving systems with stronger vs weaker verifiers, with the model held fixed, I'd love to see it.