Our original idea was to help software engineers build high-quality LLM applications by integrating their domain knowledge into a scoring system, which could then drive everything from prompt tuning to fine-tuning, RL, and data filtering. But what we quickly learned (with the help of HN – thank you!) is that most people aren’t optimizing as their first, second, or even third step — they’re just trying to ship something reasonable using system prompts and off-the-shelf models.
In looking to build a product that’s useful to a wider audience, we found one piece of the original product that most people _did_ notice and want: the ability to check that the outputs of their AI apps look good. Whether you’re tweaking a prompt, switching models, or just testing a feature, you still need a way to catch regressions and evaluate your changes. Beyond basic correctness, developers also wanted to measure more subtle qualities — like whether a response feels friendly.
So we rebuilt the product around this single use case: helping developers define and apply subjective, nuanced evals to their LLM outputs. We call it Pi Co-pilot.
You can start with any/all of the below:
- a few good/bad examples
- a system prompt, or app description
- an old eval prompt you wrote
The co-pilot helps you turn that into a scoring spec — a set of ~10–20 concrete questions that probe the output against dimensions of quality you care about (e.g. “is it verbose?”, “does it have a professional tone?”, etc). For each question, it selects either:
- a fast encoder-based model (trained for scoring) – Pi scorer. See our original post [1] for more details on why this is a good fit for scoring compared to the “LLM as a judge” pattern.
- or generates Python functions when that makes more sense (word count, regex etc.)
You iterate over examples, tweak questions, adjust scoring behavior, and quickly reach a spec that reflects your actual taste — not some generic benchmark or off-the-shelf metrics. Then you can plug the scoring system into your own workflow: Python, TypeScript, Promptfoo, Langfuse, Spreadsheets, whatever. We provide easy integrations with these systems.
We took inspiration from tools like v0 and Bolt: natural language on the left, structured artifacts on the right. That pattern felt intuitive — explore conversationally, and let the underlying system crystallize it into things you can inspect and use (scoring spec, examples and code). Here is a loom demo of this: https://www.loom.com/share/82c2e7b511854a818e8a1f4eabb1a8c2
We’d appreciate feedback from the community on whether this second iteration of our product feels more useful. We are offering $10 of free credits (about 25M input tokens), so you can try out the Pi co-pilot for your use-cases. No sign-in required to start exploring: https://withpi.ai
Overall stack: Co-pilot next.js and Vercel on GCP. Models: 4o on Azure, fine tuned Llama & ModernBert on GCP. Training: Runpod and SFCompute.
– Achint (co-founder, Pi Labs)
jmoore15•8mo ago
I joined Pi 3 months ago after a decade at Google. It was partly the HN community that inspired me to make the switch to a smaller company where I could have more direct impact. Working at a start-up has been quite an adjustment: while the work is extremely rewarding and fun, the pre- product/market fit phase is challenging in ways I've never experienced before in my career.
That's why I asked the team to post here, and am excited to show off this launch to see whether it meets a need that developers have (or learn why if not!)
Just for fun, I spent the last 5 minutes making an evaluation system for Show HN posts, so you can look at a real example if you'd rather not make your own [1]. If you sign in, you can fork and modify it, but you can also go directly to the homepage to try your own hand at it without any sign-in.
[1] https://withpi.ai/project/Xxyhrg2UR8kZHeNmbdV3
jmoore15•8mo ago
--
Yes, I noticed there were a few things off about this example (e.g., certain questions feel inapplicable for certain examples, or certain scores feel too optimistic for the bad example), and I intentionally left them there so as not to window dress things too much.
To add to some of your observations, I'll note that: 1. Automatic question generation runs before I've given any examples in this particular chat. This can be a positive (in that you can get started without even providing an example of your own data), but it also means we sometimes add questions that don't make sense for the data you actually have. The co-pilot is meant to be iterative for that reason. (As an example, towards the end of the chat, I do ask it to remove some questions that don't feel applicable).
2. The model still has to output a score for all questions, even if they don't apply to a particular input. We're working on a new system that will understand which questions actually apply, and can turn certain questions off if they're impossible to answer given the inputs provided.
3. We do get feedback from users that the scores feel off sometimes. In some cases, they're too high; in others, they're too low. We're working on an interface for calibrating the scores to your own preferences, e.g. with a small amount of thumbs up/thumbs down data. There's a tradeoff here, though, because we're also trying to make the evaluation process a lot easier than today's "prompt an LLM as a judge" paradigm where writing a prompt with a rubric can take a substantial amount of time, and any kind of calibration adds friction for users.
Overall, we release new models every other week and track ourselves on internal benchmarks to see improvements to both question generation and question scoring. If you play around with the system more and find other ways it's not working as you'd expect or like, please feel free to email me your examples and we'd be happy to prioritize looking into them.