frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Local, Open source AI meeting recorder that sees and hears everything

4•puremetrics•9h ago•1 comments

Ask HN: Have LLMs Plateaued?

6•leandrobon•11h ago•13 comments

New open sourced eval tool for building agents

2•Thomeras•6h ago•2 comments

Tips for Landing a YC Internship?

6•cnnadozi•15h ago•6 comments

Ask HN: What are you working on? (August 2026)

16•sarreph•15h ago•33 comments

Ask HN: Anyone still do work on Intel Macs?

20•senor_digimon•1d ago•28 comments

Ask HN: What's Next for LLMs?

4•apatheticonion•13h ago•4 comments

Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

20•pera•1d ago•18 comments

Best way to avoid bloat and AI – selfcontaining OS

11•bialamusic•1d ago•2 comments

What is the downside of coaching that I may not be seeing yet?

6•ikut3hva•1d ago•1 comments

Ask HN: I still don't understand why AI agents need "skills"

14•skeptic_ai•1d ago•15 comments

Ask HN: How are you using AI to learn?

9•mfalcon•1d ago•23 comments

Ask HN: What's Your BHAG?

7•sudo_cowsay•1d ago•12 comments

Tell HN: HackerOne Announces Mandatory identity verification

17•0xmattf•2d ago•1 comments

Ask HN: When will ChatGPT Voice be able to control my phone?

2•spapas82•1d ago•6 comments

Strategic Oil Reserves Down to a Record Low Two-Week Supply

22•megamike•3d ago•20 comments

What if we learned languages like we acquired our first?

7•Salim_wariz•1d ago•9 comments

ask HN: How do I run Android 15 on a system with Andr 12 vendors?

3•KinetiNode•1d ago•2 comments

Ask HN: What did you learn last month? (July 2026)

15•bhu1st•2d ago•14 comments

AI will always find a difference

3•ghassenfaidi•1d ago•1 comments

Ask HN: Do you ever think about the environmental impacts of the AI datacenters?

3•roschdal•6h ago•3 comments

You've reached the end!

Open in hackernews

New open sourced eval tool for building agents

2•Thomeras•6h ago
Hello so i created agent detective tool, this tool answer question: which of the agents broke in system/workflow. I would really appreciate if community would try it out. https://github.com/Thomeras/agent_detective

Comments

mmakeev•5h ago
how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
Thomeras•1h ago
Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population. So i define it as: quality broke when an observable moved without a change that authorizes it. Thanks for trying it out i am really happy for feed back :D