This is my third attempt at getting LLMs to build this, and I'm finally happy with what they can slap together now. https://errors.standardbeagle.com/

I'm running agents to pull in projects and scan the source code to get the error message templates and use matching rules to match the fixed part of the template to the actual output string, and tie it back to a summary of why the error was thrown and recommendations on how to resolve the error. Essentially 90% of what I used stack overflow for.

In the past either the LLM got confused about what the project did, was horrible at finding the error sites or figuring out why the error occurred.