It uses official provider SDKs when available, which helps since providers handle their own compatibility updates. No proxy or gateway service needed either, so getting started is pretty straightforward - just pip install and import.
Currently supports 20+ providers including OpenAI, Anthropic, Google, Mistral, and AWS Bedrock. Would love to hear what you think!
I have to do this already with practically all software I write, so the comexity is already baked in. Sure, if you don't already have a database or cache, maybe a proxy is simpler, but otherwise it's just extra infrastructure you need to manage.
> The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.
I've been doing stuff like the above (not for LLMs but similar use cases) for years "at scale" without issues. But in any case, you need to store state the moment you scale beyond a single proxy server anyway. Plus, most products never achieve a scale where this discussion matters.
But provider switching is built in some of these - and the folks behind envoy built: https://github.com/katanemo/archgw - developers can use an OpenAI client to call any model, offers preference-aligned intelligent routing to LLMs based on usage scenarios that developers can define, and acts as an edge proxy too.
You're right that archgw handles routing at the infrastructure level, which is perfect for centralized control. any-llm simply gives you the option to handle routing in your application code when that makes sense (For example, premium users get Opus-4). We leave the architectural choice to you, whether that's adding a proxy, keeping routing in your app, or using both, or just using any-llm directly.
with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues)
you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.
> it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications
Leveraging official SDKs also does not solve compatibility issues. any_llm would still need to maintain compatibility with those offical SDKs. I don't think one way clearly better than the other here.
I'd rather a library that just used OpenAPI/REST, than one that takes a ton of dependencies.
but I can give you one example: litellm recently had issue with handling deepseek reasoning. they broke implementation and while reasoning was missing from sync and streaming responses.
You're absolutely right that any router reimplements interfaces for normalization. The difference is what layer we reimplement at. We use SDKs where available for HTTP/auth/retries and reimplement normalization.
Bottom line is we both reimplement interfaces, just at different layers. Our bet on SDKs is mostly about maintenance preferences, not some fundamental flaw in LiteLLM's approach.
Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.
[0] https://github.com/vercel/ai
A truly universal solution would likely need to exist at a lower level of abstraction, completely decoupling the application's language from the model's runtime. It's a much harder problem to solve there, but it would be a huge step forward.
Github: https://github.com/proxai/proxai
Website: https://proxai.co/
I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?
Mozzila's project boosts of a lot of existing interfaces already, much like LiteLLM, which has the benefit of directly being able to use a wider range or supported models.
> No Proxy or Gateway server required so you don't need to deal with setting up any other service to talk to whichever LLM provider you need.
Now how it compares to LiteLLM, I don't have enough experience in either to tell.
I shipped a similar abstraction for llms a bit over a week ago:
https://github.com/omarkamali/borgllm
pip install borgllm
I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.
Seems like reputation parasitism.
On a second visit I notice a link to mozilla.org on their footer.
Still doesent ring official by me from being a veteran mozilla user (netscape, mdn, firefox) but ok, thanks for the explanation.
We're very small compared to the Mozilla mothership, but moving quickly to support open source AI in any way we can.
https://blog.mozilla.ai/introducing-any-llm-a-unified-api-to...
> One popular solution, LiteLLM, is highly valued for its wide support of different providers and modalities, making it a great choice for many developers. However, it re-implements provider interfaces rather than leveraging SDKs that are managed and released by the providers themselves. As a result, the approach can lead to compatibility issues and unexpected modifications in behavior, making it difficult to keep up with the changes happening among all the providers.
LiteLLM is rock-solid in practice. The underlying API providers announce breaking changes well in advance, and LiteLLM has never been caught out by this. LLMs will come up with hypothetical cons like this upon request.
> Lastly, proxy/gateway solutions like OpenRouter and Portkey require users to set up a hosted proxy server to act as an intermediary between their code and the LLM provider. Although this can effectively abstract away the complicated logic from the developer, it adds an extra layer of complexity and a dependency on external services, which might not be ideal for all use cases.
OpenRouter is a hosted service that provides the proxy/gateway infrastructure. Users don't "set up a hosted proxy server" themselves; they just make API calls to OpenRouter's endpoints. But older LLMs don't know what OpenRouter is and will assume it's a self-hosted proxy server.
> Another option, AISuite, was created by Andrew NG and offers a clean and modular design. However, it is not actively maintained (its last release was in December of 2024) and lacks consistent Python-typed interfaces.
Okay so you clicked the "releases" tab and saw December 2024. Next time check https://github.com/andrewyng/aisuite/commits/main/ Small, fast moving community projects like this, exllamav2, etc don't necessarily tag releases.
I've got nothing against using AI to write posts like this, but at least take the time to fact check before dumping on other people's work.
If not for the Mozilla branding, I'd have assumed this was a scam/malware - especially since it's name is so similar to Anything-LLM.
sparacha•6mo ago
CuriouslyC•6mo ago
dlojudice•6mo ago
I only use it in development. Could you elaborate on why you don't recommend using it in production?
honorable_coder•6mo ago
tom_usher•6mo ago
swyx•6mo ago
otabdeveloper4•6mo ago
swyx•6mo ago
honorable_coder•6mo ago
swyx•6mo ago
otabdeveloper4•6mo ago
Is LiteLLM's code written by an LLM?
ieuanking•6mo ago
swyx•6mo ago
pzo•6mo ago
swyx•6mo ago
wongarsu•6mo ago
troyvit•6mo ago