Most tell us whether the final answer was right, but not whether the agent actually did the same thing twice.
In prospective replays of financial services tasks, agents reached the same decision about 94–95% of the time. Exact tool-name paths matched only 67–69%. Once the tool arguments were included, agreement fell to 45–52%.
I don’t think every different path is automatically a failure. Sometimes it’s harmless flexibility. But if an agent can touch files, APIs, email, or financial controls, I’d at least like to know that it changed its path before calling the system stable.
The package repeatedly runs your agent, stores the traces, and compares the decision, tool order, arguments, and result identities.
Quick local test - no key or paid model needed:
dfah check-agent --agent dfah.demo → dfah check-agent --agent dfah.demo:toy_agent --episode-timeout-s 5 Paper: https://arxiv.org/abs/2607.20491 PyPI: https://pypi.org/project/dfah-bench/
It’s still early. I’m especially curious about two things:
How would you separate harmless path diversity from instability that should block or flag a run?
And which agent framework should I build the next adapter for? Or is there other ways folks are thinking about tracking tool paths and decision agreements