Writing evals is just too complex to do for every small skill, So I wrote Skillroll - an eval framework that can test any skill using only an eval.md
How? instead of setting up an intricate eval environment, you write a short world description, run a Dungeon Master agent that simulates that world, and route all tool calls to that agent.
each eval.md is a test case with a "world description", "agent prompt and "judgement criteria" for an llm judge to consider when rating pass/fail.
I think Skillroll is especially useful for things like fixing skills by writing a failing eval and iterating on the skill until it passes, you can then keep the eval to guard against future regressions and automatically run it in ci. You can also use it to write new evals using TDD-like approach.
I found openai's Luna to be a very nice value-for-money option for running the dungeon master, with cost per eval of around $0.003, and support for pretty complex world scenarios.
I hope it might help you reduce the friction in writing evals for your own skills