I built evalship because I kept shipping product updates that subtly broke parts of my AI powered products.
I had some local evals that Claude built, but honestly I couldn't tell whether they had the right shape. Intuitively the approach seemed kind of too simple?
I suspected better evals, run on every pull request, would have caught these issues.
But when I set out to find guidelines on how to take evals more seriously, the only frameworks and platforms I could find were complex enterprise-focused software suites. Those seem like total overkill for my use cases?
So I set out to create a lighter weight alternative. Today, evalship does three things:
1. It reviews your evals. The agent reads every part of your evals test cases and proposes to uplevel it. It runs on your agent's own subscription, so the review is free.
2. It then checks every pull request. Your evals run in GitHub Actions and upload a results file which evalship compares with your main branch's history and then comments if anything looks off. Sort of like codecov but then for eval flaws / eval ci fails.
3. It Optimizes your prompts and models
We instruct your agent to run experiments that try different permutations of your prompts and models against your evals to try and find combinations that work even better or cheaper than your current setup.
In testing on public repos evalship consistently identifies flaws that nothing else is flagging yet.
Please consider trying it out when you have a minute - I'm super curious what it'll find in your evals!
gghootch•44m ago
I built evalship because I kept shipping product updates that subtly broke parts of my AI powered products.
I had some local evals that Claude built, but honestly I couldn't tell whether they had the right shape. Intuitively the approach seemed kind of too simple?
I suspected better evals, run on every pull request, would have caught these issues.
But when I set out to find guidelines on how to take evals more seriously, the only frameworks and platforms I could find were complex enterprise-focused software suites. Those seem like total overkill for my use cases?
So I set out to create a lighter weight alternative. Today, evalship does three things:
1. It reviews your evals. The agent reads every part of your evals test cases and proposes to uplevel it. It runs on your agent's own subscription, so the review is free.
2. It then checks every pull request. Your evals run in GitHub Actions and upload a results file which evalship compares with your main branch's history and then comments if anything looks off. Sort of like codecov but then for eval flaws / eval ci fails.
3. It Optimizes your prompts and models We instruct your agent to run experiments that try different permutations of your prompts and models against your evals to try and find combinations that work even better or cheaper than your current setup.
In testing on public repos evalship consistently identifies flaws that nothing else is flagging yet.
Please consider trying it out when you have a minute - I'm super curious what it'll find in your evals!