Ask HN: How do you evaluate the skills your AI agents use?
14•enesalatas•4h ago
Do you systematically evaluate agent skills, either before or while using them? If so, what tools or methods do you use? Or do you mostly rely on reading the SKILL.md and trying things out?
Comments
bisonbear•2h ago
There are two ways I see to do this:
1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
barrenko•2h ago
Would you know of decent sources on creating (personal mostly) benchmarks? Doing something like that currently for processing PDFs and it's a bit cumbersome, but that's work I guess.
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it
bisonbear•2h ago
1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality.
2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios.
It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval).
The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive.
Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically.
TLDR: tell your agent to do it, and think about building your own.
barrenko•2h ago
bisonbear•2h ago
My 2c as someone who is working on this problem is that you can follow the 80/20 rule when making them for personal use - put in the 20% and you can make a good enough benchmark relatively easily. The remaining work is certainly valuable, and necessary to do if the benchmark is consequential, but it's a pain in the ass and hard to do correctly, so if you don't have to, don't worry about it