Management is pushing us toward running open-weight models in-house after some compliance conversations around data privacy. Before we commit, we'd love to hear from people who've made this transition.

Specifically curious about:

Did it actually end up cheaper than paying for API access at your request volume? Were there any issues related to managing performance, more specifically latency, throughput, hardware utilization? How do you handle cost visibility and attribution across teams/workloads?

Also, super curious about other aspects, what worked, what didn't, and what do you wish you'd known before switching?

Thanks in advance! PS: We are not seeking for an absolute truth, just want to be prepared if that transition will take place.