After migrating to an inference gateway to serve internal usage across an organization, problems have started to pop up. Once you detach from a managed ecosystem such as OpenAI or Anthropic you lose governance capabilities that makes the transition difficult.
- Losing access to a central plugin hub for delivering MCPs, skills, etc. A gateway can sort of do this but has low compatibility beyond skills, different harnesses, and version ranges
- Inability to deliver managed config (model defaults, etc). Can sort of be done with MDM/Jamf but doesn't reach contractors and doesn't follow employees to other machines
- Gateway configuration is awkward for devs transitioning from SSO and API keys sit in plain text on their machines
The gateway transition facilitates new model families (Kimi, Deepseek, etc) which are best utilized with new harnesses beyond Codex or Claude Code. Managing these additional tools multiplies the governance problems.
Lastly, attempting to deliver an attribution program which can tag inference requests with ticket numbers and other metadata to facilitate Opex/Capex calcs for finance becomes fairly unfeasible when there are multiple harnesses across arbitrary versions. This was not solved with managed ecosystems either to be fair, but requires some self-serve program or some local MITM proxy which is a distribution and versioning problem in of itself.
I built Blue to attempt to solve these problems. It's an MIT open source, self-hostable control plane. Developers install a universal CLI and point it at their hosted control plane tenant. On launch, it reconciles the organization's policy per harness, delivers extensions, and configures the gateway. It then launches and passes control to the native CLI (Kimi, OpenCode, Codex, etc) that devs already use.
Authentication is via browser device authorization which mints ephemeral bearer tokens. For gateway access, these tokens are exchanged at the inference proxy (part of the self-hosted Blue image) for the real API keys. This means that devs don't need to manage API keys or even log into the gateway at all. Lastly, SCIM and OIDC are supported out of the box, so orgs can connect their identity solution immediately.
Curious to hear your thoughts and if you have/solved this problem. I believe it can also be solved to some degree with a harness like Pi or even OpenCode, but that of course depends on the entire org migrating to those.
Last thing I'll say is nowadays a CLI isn't enough since I've seen over 80% of devs prefer desktop apps like the ChatGPT app. We're also working on an open source desktop app that supports metaharnesses like Blue to make this transition easier.
Thanks!