The CLI supports a hillclimbing loop:
- set up tasks for coding agents that would need your docs, like 'set up Gauge CI checks'
- run the tasks
- analyze session transcripts for friction or failures
- create variants of your docs to resolve the issues
- re-run the tasks with the doc variants injected (transparently, at the same hostname)
- repeat until you have a variant that resolves the issues
The same loop supports hillclimbing your Skills, CLI, and other agent-facing surfaces.
We started out 2 years ago building open-source developer tools (like Tach https://github.com/tach-org/tach), then pivoted to AI Search optimization, and are now focused on the intersection of the two spaces. We were already working with awesome devtool companies, and they were the first ones to ask us how to optimize for Claude Code and Codex. We realized this would be something that eventually every company needs to do. Our CLI provides primitives for docs/devrel/eng teams to measure, analyze, and improve how coding agents use their product.
The design boils down to two components: VMs and a network gateway.
The VMs are straightforward, we want the agents to be highly privileged within their environment so that they are representative of developers' coding agents. Each VM generally comes with the agent harness itself, language toolchains, and a git repository.
The network gateway is where we terminate TLS and swap in variants of your docs transparently. The gateway also handles credential replacement, so agents see placeholders and can't exfiltrate keys.
The CLI comes with a 'local' execution backend (inference is still remote). For VMs, it uses Docker containers, and the network gateway is mitmproxy running in a sidecar container for each session.
We've used the hillclimbing loop internally to find and fix coding agent issues like mistaken pricing, inaccurate product limitations, and repeated workarounds. We saw coding agents report that Gauge only supports Claude Code and Codex, traced it back to the web pages they fetched, and fixed the behavior (we support Pi, OpenCode, and Antigravity as well). For Clerk, we helped shift coding agents from their legacy 'keyless' mode to their new 'clerk init' command. They were able to validate a pre-release change to their SDK's error messages using Gauge, then saw a 3x increase in coding agents using 'clerk init' post-deploy.
Due to the 'static' nature of AI models, we've realized that closed-loop optimization techniques are now possible for essentially all software products. The same agent can be simulated over and over, meaning that a high-fidelity testing environment is all you need. Achieving the best results boils down to high-quality analysis (finding generalizable friction and objections) and paired experiments (vary only the content, measure the downstream change in behavior).
The repo is here: https://github.com/gauge-sh/gauge-cli and you can start using the CLI with: npm i -g @withgauge/cli
We also have documentation and a quickstart here: https://docs.withgauge.com/agents/guides/cli
If you point a coding agent at those pages, it should guide you through the initial setup - let us know if it doesn't! The intention is for your agent to run some sensible evaluations and find at least one point of friction you can fix.
We'd love to know from HN: Has anyone tried to DIY this locally, or built something in-house at their company? Is this an improvement? If not, what would need to change?
esafak•43m ago
Thanks for sharing.