This came out of a chatbot project I built at work last year. We discovered how sensitive LLMs are to small prompt changes and with our setup being storing prompts in a `.ts` file checked into the repo, it meant analysing effectiveness of prompts was impossible. Existing tools I tried were either overloaded with data or locked me into their own models.
So over the last few months I’ve spent my very limited free time (new Dad) building something simpler:
- Compare prompts with real user feedback. - Clean analytics without information overload. - Bring your own OpenAI key and choose any of their LLM models.
It’s up and running and free to try (signup required, no credit cards though). This is no where near a finished product in terms of the website design, it’s a bit rough and ready since design is not my thing. Feedback on the idea and direction would be really appreciated.