[Originally intended as a Show HN post, but my account is not eligible for posting Show HN post yet, so posting as normal content instead.]
Motivation: I have read quite a few stories about people's LLM API keys get uploaded verbatim in git repos, get stolen, or go unexpectedly over-budget. Being a panic-y person myself, I was thinking what we can do for those problems.
Before you read any further: only after I finished this whole thing, I found out that LiteLLM has this "virtual key" concept, and virtual key already comes with budget control. That actually implements the same thing as I did (with way more API endpoint support and other functionalities). So if you are already using LiteLLM or want to use LiteLLM, you can stop reading now and just use LiteLLM. However, LiteLLM has grown quite complicated: production deployment involves compute + database + secret manager, whereas my implementation is a serverless Cloudflare workers project, which can be hosted completely free (the free quota is more than enough at least for my personal use). If you are still interested, read on.
Solution: put real API key in an LLM API gateway, so your git repos/coding agents cannot even see the real API key. Instead, we create temporary, budget-scoped "tokens" in the LLM API gateway, and use coding agents with those tokens. The gateway will only let requests through and supply the real API key if the token has not expired, and has not exceeded its budget limit. This way, if our development machine is compromised, or coding agent gets into an infinite loop, our loss is limited (the token's budget limit) [1]. I mostly use DeepSeek/Claude for my side projects. Now everyday I create a new DeepSeek token with $1 budget and keep coding with the peace of mind knowing that I will only use (a bit more than) $1 for the day. Or before I hand off a complicated task to Claude, I will create a token with a fixed budget, and then sleep peacefully without worrying my API cost exploding.
Technical detail: the gateway splits response stream into two: one is forwarded to client directly, so clients still get streaming output like before; the other one is used to extract usage information from the response, calculate the cost, and update token budget.
Supported API endpoints: it works with official DeepSeek/OpenAI/Anthropic API endpoints. For coding agents (e.g. Codex, Claude Code, OpenCode), you will need to: point them to the gateway, create a new token in the gateway, and use your token as API key.
Compare with existing products:
- LiteLLM: 1. This is a much smaller subset of what LiteLLM provides; this simplicity can let you easily audit the full source code. 2. Zero infra to operate: no VM to patch, no database server to maintain, no secret manager to set up; just upload worker code to Cloudflare. 3. Cloudflare provides sufficient free quota for personal use.
- Cloudflare AI Gateway: Cloudflare AI gateway's budget control is static policies applied to a fixed time window (e.g. $1 every day). My implementation is more flexible: you can create a new budget limited token at any time, with any budget limit. My hope is that Cloudflare AI gateway just implements this, so I don't need to maintain my implementation :-)
[1]: Note that the budget limit is NOT 100% reliable, because 1. we only check remaining budget at the beginning of an incoming request, so one large request or concurrent requests can both go over budget; 2. budget calculation and update might fail silently.
endlessroad1991•17h ago
Motivation: I have read quite a few stories about people's LLM API keys get uploaded verbatim in git repos, get stolen, or go unexpectedly over-budget. Being a panic-y person myself, I was thinking what we can do for those problems.
Before you read any further: only after I finished this whole thing, I found out that LiteLLM has this "virtual key" concept, and virtual key already comes with budget control. That actually implements the same thing as I did (with way more API endpoint support and other functionalities). So if you are already using LiteLLM or want to use LiteLLM, you can stop reading now and just use LiteLLM. However, LiteLLM has grown quite complicated: production deployment involves compute + database + secret manager, whereas my implementation is a serverless Cloudflare workers project, which can be hosted completely free (the free quota is more than enough at least for my personal use). If you are still interested, read on.
Solution: put real API key in an LLM API gateway, so your git repos/coding agents cannot even see the real API key. Instead, we create temporary, budget-scoped "tokens" in the LLM API gateway, and use coding agents with those tokens. The gateway will only let requests through and supply the real API key if the token has not expired, and has not exceeded its budget limit. This way, if our development machine is compromised, or coding agent gets into an infinite loop, our loss is limited (the token's budget limit) [1]. I mostly use DeepSeek/Claude for my side projects. Now everyday I create a new DeepSeek token with $1 budget and keep coding with the peace of mind knowing that I will only use (a bit more than) $1 for the day. Or before I hand off a complicated task to Claude, I will create a token with a fixed budget, and then sleep peacefully without worrying my API cost exploding.
Technical detail: the gateway splits response stream into two: one is forwarded to client directly, so clients still get streaming output like before; the other one is used to extract usage information from the response, calculate the cost, and update token budget.
Supported API endpoints: it works with official DeepSeek/OpenAI/Anthropic API endpoints. For coding agents (e.g. Codex, Claude Code, OpenCode), you will need to: point them to the gateway, create a new token in the gateway, and use your token as API key.
Compare with existing products: - LiteLLM: 1. This is a much smaller subset of what LiteLLM provides; this simplicity can let you easily audit the full source code. 2. Zero infra to operate: no VM to patch, no database server to maintain, no secret manager to set up; just upload worker code to Cloudflare. 3. Cloudflare provides sufficient free quota for personal use. - Cloudflare AI Gateway: Cloudflare AI gateway's budget control is static policies applied to a fixed time window (e.g. $1 every day). My implementation is more flexible: you can create a new budget limited token at any time, with any budget limit. My hope is that Cloudflare AI gateway just implements this, so I don't need to maintain my implementation :-)
Since this project uses your API keys, please audit/review the code before using it. Github: https://github.com/er91/budget_aware_llm_api_gateway
Reddit: https://www.reddit.com/r/opencodeCLI/comments/1uwwb2n/wrote_...
[1]: Note that the budget limit is NOT 100% reliable, because 1. we only check remaining budget at the beginning of an incoming request, so one large request or concurrent requests can both go over budget; 2. budget calculation and update might fail silently.