We made a VS Code extension that allows developers to play with their prompts right in their editor. It uses Oxc and RustPython to parse and detect the prompts in the source code.
Devs can test their prompts against different models and/or variables (models*data matrix). With a CSV dataset, one can test their prompt changes against multiple rows and verify whether the change is good or not. Automated evals are coming soonish.
The request/response JSONs are available to inspect, as well as usage stats with projected cost.
The extension supports hundreds of models thanks to the Vercel Gateway. If you need more provider options (planned but not priority), please let me know. Users without keys will get their requests served by LM Studio running on my PC. Please don't crash my workstation!
It only supports JS/TS and Python for now, but many more languages are coming. Compiling tree-sitter crates to Wasm is PITA, but I'm getting there.
I also work on improving the heuristics so prompt strings can be detected from arrays or when passed as arguments to popular LLM APIs. Now you can use "prompt" in the variable name or put a comment with "@prompt" in front of it.