We're the maintainers of docker/model-runner and wanted to share some major updates we're excited about.
Link: https://github.com/docker/model-runner
We are rebooting the community:
https://www.docker.com/blog/rebooting-model-runner-community...
At its core, model-runner is a simple, backend-agnostic tool for downloading and running local large language models. Think of it as a consistent interface to interact with different model backends. One of our main backends is llama.cpp, and we make it a point to contribute any improvements we make back upstream to their project. It also allows people to transport models via OCI registries like Docker Hub. Docker Hub hosts our curated local AI model collection, packaged as OCI Artifacts and ready to run. You can easily download, share, and upload models on Docker Hub, making it a central hub for both containerized applications and the next wave of generative AI.
We've been working hard on a few things recently:
- Vulkan and AMD Support: We've just merged support for Vulkan, which opens up local inference to a much wider range of GPUs, especially from AMD.
- Contributor Experience: We refactored the project into a monorepo. The main goal was to make the architecture clearer and dramatically lower the barrier for new contributors to get involved and understand the codebase.
- It's Fully Open Source: We know that a project from Docker might raise questions about its openness. To be clear, this is a 100% open-source, Apache 2.0 licensed project. We want to build a community around it and welcome all contributions, from documentation fixes to new model backends.
- DGX Spark day-0 support, we've got it!
Our goal is to grow the community. We'll be here all day to answer any questions you have. We'd love for you to check it out, give us a star if you like it, and let us know what you think.
Thanks!
ericcurtin•3mo ago
We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.
When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.
Recently, we've invested a lot of effort into making it a true community project. A few highlights:
- The project is now a monorepo, making it much easier for new contributors to find their way around.
- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.
- We made sure we have day-0 support for the latest NVIDIA DGX hardware.