The problems we kept hitting without these tools:
* One endpoint dies > workflows stall
* No model unification so routing isn't great
* No unified load balancing across boxes
* Limited visibility into what’s actually healthy
* Failures when querying because of it
* We'd love to merge all them into OpenAI queryable endpoints
Olla fixes that - or tries to. It’s a lightweight Go proxy that sits in front of Ollama, LM Studio, vLLM or OpenAI-compatible backends (or endpoints) and:
* Auto-failover with health checks (transparent to callers)
* Model-aware routing (knows what’s available where)
* Priority-based, round-robin, or least-connections balancing
* Normalises model names for the same provider so it's seen as one big list say in OpenWebUI
* Safeguards like circuit breakers, rate limits, size caps
We’ve been running it in production for months now, and a few other large orgs are using it too for local inference via on prem MacStudios, RTX 6000 rigs.
A few folks that use JetBrains Junie just use Olla (https://thushan.github.io/olla/usage/#development-tools-juni...) in the middle so they can work from home or work without configuring each time (and possibly Cursor etc).
You can compare how Olla is complimentary with tools like LiteLLM (https://thushan.github.io/olla/compare/litellm/) and others in our docs (https://thushan.github.io/olla/compare/overview/).
Links:
GitHub: https://github.com/thushan/olla
Docs: https://thushan.github.io/olla/
Olla is still very much in early development (v0.0.16).
Next up: auth support so it can also proxy to OpenRouter, GroqCloud, etc.
If you give it a spin, let us know how it goes (and what breaks). Oh yes, Olla does mean other things (https://thushan.github.io/olla/about/#the-name-olla).