Also, isn't it weird that it takes so long to fix given the magnitude of the issue? Already down for 3 hours.
I can see an image tag available in the cache in my project on cloud.google.com, but after attempting to pull from the cache (and failing) the image is deleted from GAR :(
> "When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content."
So if the authentication service is down, it might also affect the caching service.
Source: I co-founded Quay.io
So far, spelling has been our worst issue with Quay!
But maybe ECR is better than Docker Hub.
There are still some we pull from Docker Hub, especially in the build process of our own images.
To work around that, on AWS, you can prefix the image with public.ecr.aws/docker/library/ for example public.ecr.aws/docker/library/python:3.12 and it will pull from AWS's mirror of Docker Hub.
I believe anyone can pull from the public ecr, not just clients in AWS
for example, i have redis:7.2-alpine in cache, but not golang:1.24.5-alpine
I needed the golang image to start my dev-backend
so i replaced FROM golang:1.24.5-alpine with FROM redis:7.2-alpine, and manually installed golang with apk in the redis container :)
What would be really nice is a system with mutating admission webhooks for pods which kicks off a job to mirror the image to a local registry and then replaces the image reference with the mirrored location.
[[registry]]
location = "docker.io"
[[registry.mirror]]
location = "core.yourharbor.example.com/hub"
Where hub is the name of the proxy you configured for, in this case, docker.io. It's not quite what you're asking for but it can definitely be transparent to users. I think the bonus is that if you look at a podspec it's obvious where the image originates and you can pull it yourself on your machine, versus if you've mutated the podspec, you have to rely on convention.Basically it's a k3s configured to use a local mirror and that local mirror is running the Zot registry (https://zotregistry.dev/v2.1.8/). It is configured to automatically expired old images so my local hard drive isn't filled up).
I've been using it at home and work for a few years now, might be a bit overkill if you just want a simple registry, but is a really nice tool for anyone who can benefit from the other features.
We chose to move to GitLab's container registry for all the images we use. It's pretty easy to do and I'm glad we did. We used to only use it for our own builds.
The package registry is also nice. I only wish they would get out of the "experimental" status for apt mirror support.
https://hub.docker.com/_/registry
Your git provider probably also has a container registry service built in.
switz•4mo ago
I don't work on mission-critical software (nor do I have anyone to answer to) so it's not the end of the world, but has me wondering what my alternate deployment routes are. Is there a mirror registry with all the same basic images? (node/alpine)
I suppose the fact that I didn't notice before says wonderful things about its reliability.
tom1337•4mo ago
Unfortunately that does not help in an outage because you cannot fill the cache now.
pebble•4mo ago
Took me a while to think of checking the docker hub status page.
cipherself•4mo ago
switz•4mo ago
Too•4mo ago
cipherself•4mo ago
tln•4mo ago
tln•4mo ago
akshayKMR•4mo ago
kam•4mo ago
https://gallery.ecr.aws/
XCSme•4mo ago
XCSme•4mo ago
matt_kantor•4mo ago
> wondering what my alternate deployment routes are
If the stakes are low and you don't have any specific need for a persistent registry then you could skip it entirely and push images to production from wherever they are built.
This could be as simple as `docker save`/`scp`/`docker load`, or as fancy as running an ephemeral registry to get layer caching like you have with `docker push`/`docker pull`[1].
[1]: https://stackoverflow.com/a/79758446/3625