So I have been building quite a few apps and n8n workflows that were using a lot of LLM requests. I was building pretty much in the dark, just getting some (partly delayed) usage stats from each provider separately. From that arose a need for some proper request logging, stats, filtering, scoping the requests to the various workflows, and rate limiting. So I've built that for myself (see link).
Bottom line here how it helps me is:
1. Helps me understand my requests from various apps and workflows (e.g. prompting, revisiting past requests, scoping the requests, any meta data stored, charts to visualize metrics). It's completely provider agnostic, too.
2. Gives me an understanding of how much I spend exactly on single requests, any workflow or parts of a workflow, apps, which provider I spend how much on, etc.
3. It gives me the ability to rate-limit my requests, and that can be done even by grouping requests from different providers. Essentially, I'm tagging every request, so if I make one to Gemini and one to OpenAI and give them the same tag, I can put a rate limit on the tag that will stop further requests to any of the used providers once the limit is hit.
4. It gives me an understanding how profitable a request or a group of requests is (for example, I can attach a stripe customer or sale id to the requests I make. Then I create a payout and attach it the same tag, matching requests with the income and calculating income / expense ratio).
Not yet sure if I want to take it further in terms of making it accessible to others, but I've put up a waitlist for now and recorded a demo video. Maybe there is some interest.
mxmzb•3mo ago
1. Helps me understand my requests from various apps and workflows (e.g. prompting, revisiting past requests, scoping the requests, any meta data stored, charts to visualize metrics). It's completely provider agnostic, too.
2. Gives me an understanding of how much I spend exactly on single requests, any workflow or parts of a workflow, apps, which provider I spend how much on, etc.
3. It gives me the ability to rate-limit my requests, and that can be done even by grouping requests from different providers. Essentially, I'm tagging every request, so if I make one to Gemini and one to OpenAI and give them the same tag, I can put a rate limit on the tag that will stop further requests to any of the used providers once the limit is hit.
4. It gives me an understanding how profitable a request or a group of requests is (for example, I can attach a stripe customer or sale id to the requests I make. Then I create a payout and attach it the same tag, matching requests with the income and calculating income / expense ratio).
Not yet sure if I want to take it further in terms of making it accessible to others, but I've put up a waitlist for now and recorded a demo video. Maybe there is some interest.