To show how Gensee works, we created a new end-to-end demo https://www.youtube.com/watch?v=AXIX9LgN4mU where we build and launch a travel planner AI application: https://demo.gensee.ai/travel-planner. The web app uses two agents: one to generate a travel plan based on user requirements built using CamelAI's multi-agent society, and another to answer follow-up questions with LLM and web search using no framework (pure Python). We've also open-sourced the travel planner app itself: https://github.com/GenseeAI/Trip-planner-demo.
Here's the process we show:
- DEPLOY: We start with the agent's source code in the GitHub repo and deploy it to Gensee directly using the repo url.
- TEST & ANALYZE: To evaluate the agent, Gensee automatically generates test cases customized to the agent. We can then inspect the full execution trace for each test run (including LLM and tool call inputs/outputs) and manually swap models/tools.
- METRICS: Next, we can instruct Gensee to automatically generate metrics (e.g., "does the generated plan include all requested cities?"). These metrics use LLM-as-a-Judge internally. There are also two objective metrics: dollar cost and execution latency.
- OPTIMIZE: We then select our desired metrics and run Gensee’s automated optimization process, which experiments with different models and tools to find the setup that maximizes quality, minimizes cost, or minimizes latency.
- LAUNCH & AUTOSCALE: Once we're happy with the optimized agent, Gensee provides a production-ready API endpoint that we can integrate directly into our web application. We can also download the Gensee-optimized source code and do more offline tuning. Once launched, the agent will be autoscaled on Gensee as requests arrive. Gensee is the only entity to pay, as Gensee internally covers all model and tool call costs.
We are trying to build the "AgentOps" tooling that we hope can be useful to all agent developers and beyond.
We would be grateful for the community's honest feedback!
You can try it here: https://platform.gensee.ai. We're providing $10 in FREE credits every month. Thanks for checking it out!