In your example you are calculating error as a function of missed tests, for example. That's a pretty good example, you can't just leave that value at 2.5% of starting. It needs to be zero. All existing loops have logic to not proceed until those tests all pass.
If you can propose a means of generating an error signal in actual code you've just invented a way of automatically grading the goodness of code, which should make you quite a lot of money!
My understanding is that this tool is mostly about “giving up” sooner, when it’s obvious that additional loops are not getting any closer to passing those tests.
It’s a cost saving measure when the problem is intractable, and needs more human context, etc.
I (Dave) am a real person, and yes I used AI to help me with every part of this project, including my initial comment introducing it here. That was a mistake, I should have just written it myself. I am a self-taught developer who is passionate about AI development and curious about seeing what I can do with it. I come up with my ideas and test them with the help of AI. Then I have AI help me understand it. This all started because I was building multi-agent systems, just experimenting, and I noticed that they looked to me like electrical circuit diagrams. So it made me curious if there was any cross over between the laws of electrical engineering and how it might apply to agentic systems. With AI today you can pursue any idea you have, so I have fun by seeing what happens when you take concepts from one established field and apply them to something else. I had AI do deep research on electrical engineering concepts, and asked it to research how those might be applied to agentic systems. It came up with interesting ideas, so I had it plan scientific experiments on ones that looked promising, and the one that looked like the most obvious application is what turned into LoopGain. With the help of AI I developed it into a working system, created a website and company around it, made the project open source, wrote blog articles about the experiments, then asked AI how to best share it with the world. I love developing useful tools and researching the edge of AI capabilities, so that is what I am doing here. Thanks for reading.
For now, we're passing the first pass to the humans, and that's roughly 80 for 20, but to get the last 20 without blowing up our token spend? This might be the approach.
Thanks for the write up comment, btw.
fitz2882•4d ago
The cap is how nearly every verify-revise loop gets stopped today, and it's wrong in both directions. Stop too early and you clip a loop that was still improving. Stop too late and you pay for iterations after the loop already found its best answer — and ship the final attempt, which is sometimes worse than one it already had.
On a 2,000-trial benchmark (paired real-API runs, five loop patterns across six framework adapters, three model providers, pre-registered protocol with kill criteria), LoopGain cut total API spend by 92.8% vs max_iterations=20 ($27.05 → $1.94) and median wall-clock ~15x (30.9s → 2.1s). A cross-vendor judge preferred LoopGain's outputs on the weighted average (0.678 across 1,800 pairwise comparisons) — mostly because LoopGain returns the iteration with the lowest error it saw, while a fixed cap ships whatever the final iteration produced. The raw data and methodology are public, the full run is browsable at dashboard.loopgain.ai/benchmark, and there's a replay of a real benchmark run (all measured, no simulation) at dashboard.loopgain.ai/demo.
How it works: each iteration, LoopGain takes the ratio of the current error to the previous error — the loop's empirical loop gain (Aβ, borrowed from control theory). Aβ<1 means the error shrank — the loop is improving. Aβ≥1 means it held or grew — the loop is stuck or making things worse. A trajectory classifier reads the recent Aβ values, labels the loop (FAST_CONVERGE / CONVERGING / STALLING / OSCILLATING / DIVERGING), and decides whether to keep going, stop here, or stop and roll back to the lowest-error output so far.
Integration is a few lines around any loop that produces an error signal:
Honest limits, because they matter more than the headline: LoopGain detects convergence, not correctness — it inherits your verifier's blind spots. I re-graded my own benchmark and 4.5% of "converged" code-gen runs passed every check the loop ran but failed a fuller held-out test suite. And savings depend on workload: failure-heavy loops save ~78–84%, not 92.8%. There's a writeup on designing verifiers strong enough to trust on the blog.Apache-2.0, pip install loopgain. Adapters for LangGraph, CrewAI, AutoGen, LangChain, OpenAI Agents SDK, and the Claude Agent SDK; the raw API works for anything with a measurable error.
What I'd really love from HN: if you run production agent loops, I'm interested in whether the stop decisions match what you see empirically — and what your loops' error signals actually look like, since the verifier is what makes or breaks the stop. Happy to answer anything.
lochnessduck•4d ago
fitz2882•1d ago
Anyhow, whether I used a tool like AI to assist me or not, I am here to share something I built that I hope others will find useful. I am building what I am passionate about.
stephantul•13h ago
Your fully generated project description makes people lose faith in the actual project. If you went through the trouble of writing the whole code yourself, why generate the comment presenting it to the public wholesale.
N_Lens•13h ago
Terretta•2d ago
What prompt are you using to generate this exact form of thing?
Problem history story...
Everyone's doing it wrong...
Proof of claims ...
Abstract ...
"Honest" bustedness ...
Keyword stuffing ...
Call for feedback ...
"What I'd really love from Dave:" is what's causing this particular structure. What model/harness/agent are you using, whether automated or pasting it yourself?