My cofounder spent 6 years at Veeva Systems building QA frameworks for regulated software. I spent 5 years in sales helping AI-native startups with their growth. We've both seen the issues companies face when deploying agents without a structured way of testing and monitoring them. The engineering teams we interviewed prior to building said they are doing manual spot-checks or just using LLM as a judge to check discrepancies.
We generate synthetic test environments for 35 platforms (Salesforce, Jira, Stripe, Zendesk, Datadog, and 29 more). Each environment has ~200 adversarial queries with computed ground truth across 7 categories: clean lookups, ambiguous questions, multi-step operations, scope boundary tests, contradictory inputs, invalid assumptions, and context-dependent questions.
The ground truth is initially computed by running SQL against the synthetic dataset so it's not just guessed by an LLM.
We also do adversarial testing for pre-production conversational agents. The system generates different personas that push agents off-script and scores pass/fail across 10 dimensions so you can see what happens in realistic customer scenarios. Our production monitoring is just one webhook with every conversation evaluated in real time as well.
We are two founders bootstrapping and we would love feedback on our approach!