Also, isn't it weird that it takes so long to fix given the magnitude of the issue? Already down for 3 hours.
I can see an image tag available in the cache in my project on cloud.google.com, but after attempting to pull from the cache (and failing) the image is deleted from GAR :(
> "When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content."
So if the authentication service is down, it might also affect the caching service.
There are still some we pull from Docker Hub, especially in the build process of our own images.
To work around that, on AWS, you can prefix the image with public.ecr.aws/docker/library/ for example public.ecr.aws/docker/library/python:3.12 and it will pull from AWS's mirror of Docker Hub.
for example, i have redis:7.2-alpine in cache, but not golang:1.24.5-alpine
I needed the golang image to start my dev-backend
so i replaced FROM golang:1.24.5-alpine with FROM redis:7.2-alpine, and manually installed golang with apk in the redis container :)
What would be really nice is a system with mutating admission webhooks for pods which kicks off a job to mirror the image to a local registry and then replaces the image reference with the mirrored location.
switz•1h ago
I don't work on mission-critical software (nor do I have anyone to answer to) so it's not the end of the world, but has me wondering what my alternate deployment routes are. Is there a mirror registry with all the same basic images? (node/alpine)
I suppose the fact that I didn't notice before says wonderful things about its reliability.
tom1337•1h ago
Unfortunately that does not help in an outage because you cannot fill the cache now.
pebble•1h ago
Took me a while to think of checking the docker hub status page.
cipherself•1h ago
switz•1h ago
tln•1h ago
tln•51m ago
akshayKMR•32m ago
kam•1h ago
https://gallery.ecr.aws/
XCSme•37m ago
XCSme•31m ago