Anyone else losing tokens to hallucinated MCP tool calls in production?
1•Mahjabinbm•1h ago
I have been building an agentic system on a custom internal platform and the llm keeps calling tools with identifiers that dont exist, wrong namespace, wrong handle, wrong enum. gets back an error, retries, still wrong. every bad call is tokens down the drain.
I ended up writing a big system prompt to fix it. took weeks of trial and error. its working but i still dont fully trust it.
I am curious if others are hitting this or if its just me. Especially if youre running MCP servers on internal platforms the llm has never seen before(the problem occurs most with less powerful models), do you guys have a go to solution for this? Thanks
Comments
dmilicev2•1h ago
Yes, and that's one of the reasons I started working on this tool: to elicit desirable behavior from AIs before turn 1. AI is still non-deterministic and won't follow instructions 100% of the time, but with this tool, I intend to narrow that gap.
dmilicev2•1h ago
Would love to hear your thoughts on it: https://github.com/turnzero-ai/turnzero