Here is a rough table of model size to instance type to spot $/hr:
For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.
For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.
Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.
We're currently running larger LLM models benchmarks to add to the README this week.
paguasmar•3h ago
1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.