With it you can create a test suite either manually or with an Agent based on the source code and documentation. The CLI will dispatch Agents with their own sandboxed microVMs to solve each test. Results then get graded by another Judge Agent.
Test-taker agents only have access to public information (guides, blogs, package metadata), while Judge agents have access to both public and private information (source code, internal documents)
After the test result are generated you can make improvement to your SDK manually, or use an Agent to automate the process.
Agents are sandboxed, this means: - Host machine secrets (API keys) are not exposed to the sandbox environment - Egress HTTP requests are monitored, Judge agents' egress are limited to trusted domains to ensure that proprietary IP are not exfiltrated
Features: - CLI commands for the entire workflow of generating, eval, reporting on test suites - Agent skills for each command - Local Web UI if you want to inspect test result and edit test cases visually