Having only one SSL certificate is a single point of failure, we have eliminated single points of failure almost everywhere else.
Edit: but to be clear, I don’t understand why you’d want this. If you’re worried about your CA going offline, you should shorten your renewal period instead.
Update: looks like the answer is yes. So then the issue is people not taking advantage of this technique.
The main reason to have multiple certs is so if your host (and cert prov key) is compromised, you can quickly switch to a backup, without first having to sort out getting a new cert issued.
Both Apache (SSLCertificateFile) and nginx (ssl_certificate) allow for multiple files, though they cannot be of the same algorithm: you can have one RSA, one ECC, etc, but not (say) an ECC and another ECC. (This may be a limitation of OpenSSL.)
So if the RSA expires on Feb 1, you can have the ECC expire on Feb 14 or Mar 1.
But for human persons and personal websites HTTP+HTTPS fixes this easily and completely. You get the best of both worlds. Fragile short lifetime pseudo-privacy if you want it (HTTPS) and long term stable access no matter what via HTTP. HTTPS-only does more harm than good. HTTP+HTTPS is far better than either alone.
The main lesson we took from this was: you absolutely need monitoring for cert expiration, with alert when (valid_to - now) becomes less than typical refresh window.
It's easy to forget this, especially when it's not strictly part of your app, but essential nonetheless.
I have a simple Python script that runs every day and checks the certificates of multiple sites.
One time this script signaled that a cert was close to expiring even though I saw a newer cert in my browser. It turned out that I had accidentally launched another reverse proxy instance which was stuck on the old cert. Requests were randomly passed to either instance. The script helped me correct this mistake before it caused issues.
For example say you've got an internal test endpoint, two US endpoints and a rest-of-world endpoint, physically located in four places. Maybe your renewal process works with a month left - but the code to replace working certificates in a running instance is bugged. So, maybe Monday that renewal happens, your "CT log monitor" approach is green, but nobody gets new certs.
On Wednesday engineers ship a new test release to the test endpoint, restarting and thus grabbing the renewed cert, for them everything seems great. Then on Friday afternoon a weird glitch happens for some US customers, restarting both US servers seems to fix the glitch and now US customers also see a renewed cert. But a month later the Asian customers complain everything is broken - because their endpoint is still using the old certificate.
For any non-trivial organization, you want to know when client certificates expire too.
In my experience, the easiest way is to export anything that remotely looks like a certificate to the monitoring system, and let people exclude the false positives. Of course, that requires you to have a monitoring system in the first place. That is no longer a given.
It's true that you could use client certs with say, Entra ID, and one day I will work somewhere that does that. Or maybe I won't, I'm an old man and "We should use client certs" is an ambition I've heard from management several times but never seen enacted, so the renaming of Azure AD to Entra ID doesn't seem likely to change that.
Once you're not using the Web PKI cert expiry lifetimes are much more purpose specific. It might well make sense for your Entra ID apps to have 10 year certs because eh, if you need to kill a cert you can explicitly do that, it's not a vast global system where only expiry is realistically useful. If you're minting your own ten year certs, now expiry alerting is a very small part of your risk profile.
The big employers didn't use EAP-TLS with client certs. The University of course has Eduroam (for WiFi), and I guess in principle you could use client certs with Eduroam but that sounds like extra work with few benefits and I've never seen it from either the implementation side or the user side even though I've worked on or observed numerous Eduroam installs.
I checked install advice for my language (it might differ in other languages) and there's no sign that Eduroam thinks client certificates would be a good idea. Server certs are necessary to make this system work, and there's plenty of guidance on how to best obtain and renew these certificates e.g. does the Web PKI make sense for Eduroam or should you just busk it? But nothing about client certificates that I could see.
Of course, implementing EAP-TLS usually requires a robust way for distributing client certificates to the clients. If all your devices are managed, this is often done using the SCEP protocol. The CA can be either AD CS, your NAC solution, or a cloud PKI solution like SecureW2.
But even in Enterprise corporate settings I did not ever see this though I'm sure some people do it. It sounds like potentially a good idea, of course it can have excellent security properties, however one of the major downside IMHO is that people wind up with the weakest link being a poorly secured SCEP endpoint. Bad guys could never hope to break the encryption needed to forge credentials, but they could trivially tail-gate a call center worker and get real credentials which work fine, so, who cares.
Maybe that's actually enough. Threat models where adversaries are willing to physically travel to your location (or activate a local asset) might be out of your league anyway. But it feels to me as if that's the wrong way to look at it.
Regarding wildcard certs, eh. I wouldn't say they have a bad reputation. Sure, greater blast radius. But sometimes it can certainly simplify things to use one. Your ACME client configuration is easier and your TLS terminator configuration often becomes easier when the terminator would otherwise need to switch based on SNI.
As for wildcard certs, I agree there are use cases where we really need them like dynamic subdomains {customer}.status.com
Can you share how they make ACME client configuration easier?
It's not a profound difference, but you don't need to add each name to your config. Depending on the team's tooling and processes, that may be inconsequential. But in a setting where config management isn't handled super well, where the TLS terminator is a resource shared by multiple, distinct teams, this is a simplification that can make a difference at the margin.
Think less Cloudflare-scale, and more SMB scale (especially in a Windows shop or recovering Windows shop with a different kind of technical culture than what we might all be implicitly imagining).
It's not as convenient, but it's the best SSLBoard can do...
Plug for Uptime Kuma, they support notifications ahead of expiry: https://github.com/louislam/uptime-kuma
Kind of cool to have an uptime monitoring tool that also had an option like that, two birds one stone and all that. Not affiliated with them, FOSS project.
> There’s no natural signal back to the operators that the SSL certificate is getting close to expiry.
There is. The not after is right there in the certificate itself. Just look at it with openssl x509 -text and set yourself up some alerts… it’s so frustrating having to refute such random bs every time when talking to clients because some guy on the internet has no idea but blogs about their own inefficiencies.
Furthermore, their autorenew should have been failing loud and clear, everyone should know from metrics or logs… but nobody noticed anything.
OpenSSL is still called OpenSSL. Despite "SSL" not being the proper name anymore, people are still going to use it.
By the way, TLS 1.3 is actually SSL v3.4 :)
It is called the Web PKI because although this secures most of the Internet, the billions of Relying Parties are represented in practice almost solely by a handful of Trust Stores who mostly make Web Browsers. Specifically, Mozilla, Google, Microsoft and Apple.
The Web PKI requires that the certificates are not only X.509 but specifically they obey PKIX, RFC 5280 which explains how X.509 (a standard from the X.500 directory system, a directory which in reality never ended up existing) can be used for the Internet (which very much did end up existing) via "Alternative Names". When your modern certificates have a "Subject Alternative Name" the word Alternative there means alternative to the X.500 naming scheme, which is irrelevant to us, specifically the Internet's two alternatives, an ipAddress (4 bytes or 16 bytes forming either an IPv4 or IPv6 address) or a dnsName (a subset of ASCII characters, punctuated with but never ending in a dot)
Edited: Correct s/Server/Subject/ in expansion of SAN acronym
The ITU has a slight edge in that their published standards are generally of a higher quality and easier to implement.
I care about utility not about nominal designations.
It is not about encryption (that a self-signed certificate lasting till 2035 will suffice), but verification, who am I talking with, because reaching the right server can be messed up with DNS or routing, among other things. Yes, that adds complexity, but we are talking more about trust than technology.
And once you recognize that it is essential to have a trusted service, then give it the proper instrumentation to ensure that it work properly, including monitoring and expiration alerts, and documentation about it, not just "it works" and dismiss it.
May we retitle the post as "The dangers of not understanding SSL Certificates"?
And I said above, SSL is more than about encryption, but also knowing that you are connecting to the right party. Maybe for a repository with multiple mirrors, dns aliases and a layer of "knowing from whom this come from" is not that essential, but for most the rest, even if the information is public, knowing that it comes from the authoritative source or really from who you think it comes from is important.
You can update your cert to prepare for it by appending—-NEW CERT—-
To the same file as ——-OLD CERT—-
But you also need to know where all your certificates are located. We were using Venafi for the auto discovery and email notifications. Prometheus ssl_exporter with Grafana integration and email alerts works the same. The problem is knowing where all hosts, containers and systems that have certs are located. Simple nmap style scan of all endpoints can help. But, you might also have containers with certs or you might have certs baked into VM images. Sure, there all sorts of things like storing the cert in a CICD global variable, bind mounting secrets, Vault Secret Injector, etc
But it’s all rooted in maintaining a valid, up to date TLS inventory. And that’s hard. As the article states: “ There’s no natural signal back to the operators that the SSL certificate is getting close to expiry. To make things worse, there’s no staging of the change that triggers the expiration, because the change is time, and time marches on for everyone. You can’t set the SSL certificate expiration so it kicks in at different times for different cohorts of users.”
Every time this happens you whack a mole a change. You get better at it but not before you lose some credibility
I’ve always used the calendar event before expiry and then manual renew option but I wonder why I didn’t do this. It’s trivial to roll out. With Route53 just make one canary LB and balance 1% traffic to it. Can be entirely automated.
This has given me some interesting food for thought. I wonder how feasible it would be to create a toy webserver that did exactly this (failing an increasing percentage of requests as the deadline approaches)? My thought would be to start failing some requests as the deadline approaches a point where most would consider it "far too late" (e.g. 4 hours before `notAfter`). At this point, start responding to some percentage of requests with a custom HTTP status code (599 for the sake of example).
Probably a lot less useful than just monitoring each webserver endpoint's TLS cert using synthetics, but it's given me an idea for a fun project if nothing else.
In real life, I guess there are people who don't monitor at all. For them failing requests would go unnoticed ... for the others monitoring must be easy.
But I think the core thing might be to make monitoring SSL lifetime the "obvious" default: All the grafana dashboards etc should have such an entry.
Then as soon as I setup a monitoring stack I get that reminder as well.
Just check expiration of the active certificate; if it’s under a threshold (say 1 week, assuming you auto-renew it when it’s 3 weeks to expiry; still serving a cert when it’s 1 week to expiration is enough signal that something went wrong) then you alert.
Then you just need to test that your alerting system is reliable. No need to use your users as canaries.
The canary narrows the blast radius and time-to-detection.
How long did it take for us to get to a "letsencrypt" setup? and exactly 100ms before that existed, you (meaning 90% of you) mocked and derided that very idea
Anyway you'll have one of these things anyway and I haven't seen one yet that doesn't let you monitor your cert and send you expiration notices in advance.
Don't worry. With 2 or 3 industry players dictating how all TLS certs work, now your certs will expire in weeks rather than years, so you will all be subject to these failures more frequently. But as a back-stop to process failures like this, use automated immutable runbooks in CI/CD. It works like this:
1) Does it need a runbook? Ask yourself, if everything was deleted tomorrow, do you (and all the other people) remember every step needed to get everything running again? If not, it needs a runbook.
2) What's a runbook? It's a document that gives step by step instructions to do a thing. The steps can be text, video recordings, code/shell snippets, etc as long as it does not assume anything and gives all necessary instructions (or links to them) so a random braindead engineer at 3am can just do what it says and it'll result in a working thing.
3) Automate the runbook over time. Put more and more of the steps into some kind of script the user can just run. Put the script into a Docker container so that everyone's laptop environment doesn't have to be identical for the steps to work.
4) Run the containerized script from CI/CD. This ensures all credentials, environment vars, networking, etc are the same when it runs which better ensures success, and that leads to:
5) Running it frequently/on a schedule. Most CI/CD systems support scheduled jobs. Run your runbooks frequently to identify unexpected failures and fix bugs. Most of you get notifications for failed builds, so you'll see failed runbooks. If you use a cron job on a random server, the server could go down, the job could get deleted, or the reports of failure could go to /dev/null; but nobody's missing their CI/CD build failures.
Running runbooks from CI/CD is a game changer. Most devs will never update a document. Some will update code they run on their laptop. But if it runs from CI/CD, now anyone can run it, and anyone can update it, so people actually do keep it up to date.
In technology, there are known problems and unknown problems. Expiring TLS certificates is a known problem which has an established solution.
Imagine if only some of the requests failed because a certificate is about to expire. That would be a debugging nightmare.
I manually approve the authenticity of the server on the first connection.
From then, the only time I'd be prompted again would be, if either the server changed or if there's a risk of MITM.
Why can't we have this for the web?
How do you propose to scale trust on first use? SSH basically says the trusting of a key is "out of scope" for them and makes it your problem. As in: You can put on a piece of paper, tell it over the phone, whatever, but SSH isn't going to solve it for you. How is some user landing on a HTTPS site going to determine the key used is actually trustworthy?
There have actually been attempts at solving this with some thing like DANE [1]. For a brief period Chrome had DANE support but it was removed due to being too complicated and being in (security) critical components. Besides, since DNSSEC has some cracks in it (you local resolver probably doesn't check it) you can have a discussion about how secure DANE is.
[1] https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Na...
but industry behemoths are too busy pushing other self-serving standards to execute together on this?
Am I…close?
Now, what if she'd insisted her name is Princess Charlotte. I mean, sure, OK, she's Princess Charlotte? But wait, my country has a Princess Charlotte, who is a little girl with some chance of becoming Queen one day (if her elder brother died or refused to be King). So if I just trusted that Nodis is Princess Charlotte because she said so, is there a problem?
There is quite literally nothing that prevents you from putting a self-signed server certificate. Your browser will even ask you to trust and store the certificate like your client does on the screen that shows the fingerprint.
Good luck getting everyone else to trust your fingerprint, though.
You want an upside down pyramid, in which every checked subsystem contributes an OK or some failure, and failure of these checks is the most serious failure, so the output from the bottom of your pyramid is in theory a single green OK. In practice, systems have always failed or are operating in some degraded state.
In this design the alternatives are: 1. Monitor says the Geese are Transmogrified correctly or 2. Monitoring detected a Goose Transmogrifier problem, or 3. Goose Transmogrifier Monitor failed. The absence of any overall result is a sign that the bottom of the pyramid failed, there is a major disaster, we need to urgently get monitoring working.
What I tend to see is instead a pyramid where the alternatives 1 and 2 work but 3 is silent, and in a summarisation layer, that can fail silently too, and in subsequent layers the same. In this system you always have an unknown amount of silently failed systems. You are flying blind.
Sure, the low-level nitty gritty of managing keys and certificates for TLS is hard if you don't have the expertise. You don't know about the hundreds of ways you can get bitten. But all the pieces for a better solution are there. Someone just needs to fold it into a neater higher level solution. But apparently by the time someone gained the expertise to manage this complexity they also loose interest in making a simple solution (I know I have).
> You can’t set the SSL certificate expiration so it kicks in at different times for different cohorts of users.
Of course you can, if you really want to. You could get different certificates with different expiry times for your reverse (ingress) proxies.
A more straight forward solution is to have monitoring which retrieves the certificate on your HTTPS endpoints and alert when the expiry time is sooner than it ever should be (i.e. when it should already have been renewed). For example by using Prometheus and ssl_exporter [1].
> and the renewal failures didn’t send notifications for whatever reason.
That's why you need to have deadman switch [2] type of monitoring in your alerting. That's not specific to TLS BTW. Heck even your entire Prometheus infra can go down. A service like healthchecks.io [3] can help with "monitoring the monitors".
[1] https://github.com/ribbybibby/ssl_exporter [2] https://en.wikipedia.org/wiki/Dead_man%27s_switch [3] https://healthchecks.io/
Of course, if your certificate is expired, then "the failure mode is the opposite of graceful degradation"
Just like when your password is wrong: you cannot login, the failure mode is the opposite of graceful degradation
Essentially the flip side of any critical but low maintenance part of your system: it's so reliable that you can forget to have external monitors, it's reliable enough that it can work for years without any manual labor, it's so critical that can break everything.
Competent infra teams are really good at going over these. But once in a while one of them slips through. It's not a failure of the reliable but critical subsystem, it's a failure mode of humans.
One of the main ways "How Complex Systems Fail"
It's been a couple of years now so they must have set better reminders for themselves.
I have tried several times to convince them of the joys of ACME, but they're insistent that a Let's Encrypt certificate "looks unprofessional". More professional than a down application in my opinion at least. It's not the early 2000s anymore, no one's looking at your certificate.
Good news for your manual renewal friends, renewals drop to 197 days in February, halving again the year after, halving again until it reaches 47. So they will soon adopt automation, or suffer endless renewal pain.
So yeah, certificate timelines can be monitored, completely with warnings ahead of time.
Corollary : the service checking the certificates should have a reasonably accurate time.
Problem solved.
PS: It would be nice if it could check whois for the expiration of your domain too, but I haven't seen that yet.
loloquwowndueo•1mo ago
A certificate renewal process has several points at which failure can be detected and action taken, and it sounds like this team was relying only on a “failed to renew” alert/monitor.
A broken alerting system is mentioned “didn’t alert for whatever reason”.
If this certificate is so critical, they should also have something that alerts if you’re still serving a certificate with less than 2 weeks validity - by that time you should have already obtained and rotated in a new certificate. This gives plenty of time for someone to manually inspect and fix.
Sounds like a case of “nothing in this automated process can fail, so we only need this one trivial monitor which also can’t fail so meh” attitude.
yearolinuxdsktp•1mo ago
SoftTalker•1mo ago
tetha•1mo ago
This is also why you want a mix of alerts from the service users point of view, as well as internal troubleshooting alerts. The users point-of-view alerts usually give more value and can be surprisingly simple at times.
"Remaining validity of the certificates offered by the service" is a classical check from the users point of view. It may not tell you why this is going wrong, but it tells you something is going wrong. This captures a multitude of different possible errors - certs not reloading, the wrong certs being loaded, certs not being issued, DNS going to the wrong instance, new, shorter cert lifecycles, outages at the CA, and so on.
And then you can add further checks into the machinery to speed up the process of finding out why: Checks if the cert creation jobs run properly, checks if the certs on disk / in secret store are loaded or not, ...
Good alerting solutions might also allow relationships between these alerts to simplify troubleshooting as well: Don't alert for the cert expiry, if there is a failed cert renew cron job, alert for that instead.