I'm really angry about this outage. I have a small linode in Newark that runs a couple of things but mostly a Miniflux instance for my RSS feeds. And on Sunday I opened my feed reader app and it kept timing out. So I tried to load the reader in a browser, and it timed out.
Worried, I logged into linode's management interface. There was no outage notice on the front page, no banner, for example. And there was nothing noted on the manager page. The dot next to the unresponsive system in my list of linode instances was green.
At this point I assumed the system had hung, or (worst case) even that it had been hacked. I tried to connect to it over the LISH console, and the connection timed out. So my conclusion was a fully hung system. So I tried rebooting the system from the management interface and the task just hung. For hours. I still assumed this was something broken with my machine. There was no outage indication in the management interface. I searched the internet for how to hard reset a linode. I nearly tried to clone it from a backup to a new system before deciding it wasn't mission critical on a Sunday.
Monday came around and I got an email at some point that the system had finally booted and I went in and spent at least an hour digging through logs to see what had happened, why the system had hung up. I took the time to look for evidence of hacks, checking SSH keys and system login data records and history files and quite seriously considering nuking the system and rebuilding in case it was hacked.
And later that day, after a couple hours spent on this I got an email informing me about a ticket related the datacenter outage.
Again, no banner on the front page of linode.com. No banner on the linode management interface. A GREEN LIGHT on the status indicator for the list of active systems.
You could say "why didn't you check the status page" and sure, that's sensible. But in the 20+ years I've had dedicated servers and VPSs outages are almost always my fault, so that's where I start. And if the Newark datacenter was dead why wasn't that posted straight to the top of their entire web interface? It's ridiculous.
scblock•16h ago
Worried, I logged into linode's management interface. There was no outage notice on the front page, no banner, for example. And there was nothing noted on the manager page. The dot next to the unresponsive system in my list of linode instances was green.
At this point I assumed the system had hung, or (worst case) even that it had been hacked. I tried to connect to it over the LISH console, and the connection timed out. So my conclusion was a fully hung system. So I tried rebooting the system from the management interface and the task just hung. For hours. I still assumed this was something broken with my machine. There was no outage indication in the management interface. I searched the internet for how to hard reset a linode. I nearly tried to clone it from a backup to a new system before deciding it wasn't mission critical on a Sunday.
Monday came around and I got an email at some point that the system had finally booted and I went in and spent at least an hour digging through logs to see what had happened, why the system had hung up. I took the time to look for evidence of hacks, checking SSH keys and system login data records and history files and quite seriously considering nuking the system and rebuilding in case it was hacked.
And later that day, after a couple hours spent on this I got an email informing me about a ticket related the datacenter outage.
Again, no banner on the front page of linode.com. No banner on the linode management interface. A GREEN LIGHT on the status indicator for the list of active systems.
You could say "why didn't you check the status page" and sure, that's sensible. But in the 20+ years I've had dedicated servers and VPSs outages are almost always my fault, so that's where I start. And if the Newark datacenter was dead why wasn't that posted straight to the top of their entire web interface? It's ridiculous.