Hey HN!

I've built clipper.dev - the fastest container repository in the world. (Big claim, I know.) It deduplicates at a file/chunk level, rather than at the layer level.

I get around 10-20% faster pulls than docker (when using containerd for both) for fresh pulls. For some workflows, I see 80-90% faster patch pulls (think v0.17.5 -> 0.17.6).

Here's a best case scenario for me (and worst case for dockerhub)

    docker> time sudo nerdctl pull ollama/ollama:0.17.5
    docker.io/ollama/ollama:0.17.5:
    elapsed: 244.1s total:  2.9 Gi (12.3 MiB/s)                                      

    real 4m4.172s
    user 0m0.115s
    sys 0m0.227s
    docker> time sudo nerdctl pull ollama/ollama:0.17.6
    docker.io/ollama/ollama:0.17.6:
    elapsed: 234.9s total:  2.8 Gi (12.3 MiB/s)                                      

    real 3m54.965s
    user 0m0.051s
    sys 0m0.275s

---

    clipper> time sudo clipper pull clipper/ollama:0.17.5
    using containerd runtime at /run/containerd/containerd.sock
    fetching manifest clipper.dev/clipper/ollama:0.17.5 ...
    image created: clipper.dev/clipper/ollama:0.17.5
    fetched 46 chunks (2.2 GB, 11.7 MB/s), took 3m16.134s
    done — clipper.dev/clipper/ollama:0.17.5

    real 3m17.532s
    user 0m0.006s
    sys 0m0.012s

    clipper> time sudo clipper pull clipper/ollama:0.17.6
    using containerd runtime at /run/containerd/containerd.sock
    fetching manifest clipper.dev/clipper/ollama:0.17.6 ...
    image created: clipper.dev/clipper/ollama:0.17.6
    fetched 6 chunks (599.2 MB, 7.7 MB/s), took 1m17.48s
    done — clipper.dev/clipper/ollama:0.17.6

    real 1m19.061s
    user 0m0.005s
    sys 0m0.016s

The first two layers of 0.17.6 are shared with 0.17.5, but everything else has to be transported again with docker, but not with clipper. The exact speedup depends on how fast my wifi is feeling at the time, but I'm pulling 1.5 GB less than dockerhub can do. This is _huge_ for robotics workflows or anyone else pulling docker images on poor connections. (This is not even as fast as it could be - there's other optimizations like chunking within files or using BytePort that could make this go even faster).

I built this because at multiple companies I've been at, pulling new images for updates has been...very painful. Robotics Dockerfiles tend to have a lot of dependencies stuffed into a small amount of layers. Adding one new dependency means shipping the bytes for every other dependency, even with perfect pinning. Additionally, the Dockerfiles are oftentimes...not well written, compounding the problem. So I decided to fix the tool to fit the user, rather than the inverse.

There's some other side benefits to using clipper: - Moving a file or changing permissions is "free": another layer already has the bytes - The on-disk structure lends itself much better to P2P device updates than docker's layout

I do intend this to be a commercial product, but pricing is still up in the air. I'm currently targeting robotics companies as they feel the most pain here. There's some other possible fields that I'm interested in that might find use in this: IoT, edge AI, space datacenters/etc, developmer tooling.

Please give it a try. If you don't want to upload an image, these repos should be publically pullable once you register an account:

    clipper/ubuntu: 20.04, 22.04, 24.04
    clipper/ollama: 0.17.5, 0.17.6
    clipper/python: 3.13, 3.14