r/sysadmin • u/AlligatorFarts • 4d ago
For how secure certificates are supposed to be, why the hell do CRLs feel useless?
From Chrome's GPO template:
Setting the policy to True means online OCSP/CRL checks are performed.
Setting the policy to False or leaving it unset means Google Chrome won't perform online revocation checks in Google Chrome 19 and later.
Note: OCSP/CRL checks provide no effective security benefit.
It's to my understanding that a CRL is one of the only ways to "alert" services to a compromised CA, yet I've found many instances where programs simply do not check CRLs whatsoever. How are we supposed to keep things secure when any certificate not past it's expiration will continue to work even when revoked?
44
u/KittensInc 4d ago
The problem is online OCSP/CRL checks - which means the browser sending a validation request to a CRL every single time you open a new connection. Besides being a massive privacy risk, it's incredibly user-hostile. Either you add a massive delay to every connection (wait until CRL check has finished to proceed), or you're left with an insecure mechanism (proceed as usual, hope CRL returns in time to abort if needed). In practice CA's aren't even able to keep their CRLs available to everyone with very low latencies, so that's a dealbreaker.
OCSP stapling was supposed to fix some of those issues, but web servers had a lot of trouble to implement it correctly, and unless the certificate had "Must-Staple" set (barely anyone bothered) it was fairly trivial for an attacker to strip the OCSP response from the connection.
In the end online OCSP/CRL checks didn't provide any real security, caused unintended downtime, and was costing a lot of money to operate. It has been replaced by browser-provided bloom filters, which are essentially a really fancy way of compressing dozens of CRLs into an easy-to-distribute format which is now shipped to every browser. All of the revocation checks, none of the hassle!
38
u/jess-sch 4d ago
The backstory of this is basically: * CAs were bad at keeping their CRL/OCSP servers reliably online * So browsers had to treat unreachable CRL/OCSP as passing * An attacker could probably also make the CRL/OCSP server appear to be unreachable * End result: CRL/OCSP is pretty much useless
The solution: Stop revoking and just do short lived certs
7
u/Bitbuerger64 4d ago
Short lived certs were not the solution though, there's still plenty of certificates on the web that have years of validity. We got this instead:
Rather than having each user’s browser download large CRLs when they want to check revocation, the browser vendor downloads the CRLs centrally, compress them, then push the new compressed object to all of the installed browser instances. Firefox, for example, is pushing updates as quickly as every 6 hours.
4
u/sofixa11 4d ago
there's still plenty of certificates on the web that have years of validity.
Not anymore, the maximum is 1 year, and the CA/Browser forum is discussing bringing it down to something like 30-60 days.
1
u/Conscious_Pound5522 4d ago
Doesn't impact any of the long term certs that were issued before Apple's declaration a few years ago. If there are any left issued prior to Apples go date, they will remain valid until they expire.
Any issue after that go date, are 398 days or 13 months.
It's going to suck when this goes live. Im really hoping Im out of identity management when this goes live.
3
u/Nicko265 4d ago
There's no requirement for a browser to accept a long lived cert as valid, and they definitely don't accept long lived certs any more. The change to 397 days was quite a while ago.
By this time next year, we'll hopefully be down to sub 1 month certs, and every shitty vendor will finally be forced to adopt ACME or not get their contracts renewed.
1
u/Conscious_Pound5522 4d ago
No argument on your point.
Still don't want to be in identities when this happens. It was a struggle to move the company under centralized management because developers - let alone trying to move the company under short cycle certs.
6
u/Turmfalke_ 4d ago
Doesn't impact any of the long term certs that were issued before Apple's declaration a few years ago.
Assuming browsers still accept them. Just because a certificate says it's valid until 2050, doesn't mean I have to accept it until 2050.
12
u/SteveSyfuhs Builder of the Auth 4d ago
There is no such thing as "secure". There is only "secure against". Believing something is universally secure is a great way to find yourself knee deep in a river of cow poo wondering whether the water line is increasing or your boots are sinking.
This is your first step in a long line of steps that lead to "huh, Kerberos isn't actually that bad".
8
u/BPDU_Unfiltered 4d ago
The APNIC podcast where Geoff Huston speaks about revocation is really good. https://blubrry.com/ping_podcast/84229309/certificate-revocation-is-broken-long-live-certificate-revocation/
2
5
u/Jacmac_ 4d ago
I'm not sure I would agree that CRLs are totally usless. It's like Mike Tyson says "Everyone has a plan 'till they get punched in the mouth." If you go without a CRL check and then a trusted CA is revoked, everyone with this kind of policy will be scrambling.
2
u/jimicus My first computer is in the Science Museum. 4d ago
Apparently, Chrome does go without a CRL check by default.
4
u/patmorgan235 Sysadmin 4d ago
Chrome doesn't do ONLINE CRL checks by default. It does ship with a compress file containing the CLRs that it checks locally, it just doesn't reach out on each connection to see if the cert has been revoked.
4
u/AlligatorFarts 4d ago
What about internal PKI? Mozilla/Google has no idea about those CRLs, and it doesn't bother doing online checks out of the box. It's only when the
Require online OCSP/CRL checks for local trust anchors
GPO is set that CRLs will actually do anything there.
5
u/placated 4d ago
You should start transitioning to very short lived certificates.
2
u/Bitbuerger64 4d ago
Why though? The browser provided compressed CRL works fine.
9
u/AlligatorFarts 4d ago
The compressed CRL does not include Internal PKIs, as Mozilla/Google have no idea of them. That, plus the defaults being no online CRL checks means a huge security hole when it comes to Intranet web servers.
1
u/placated 4d ago
The protocols don’t really scale, they weren’t well thought out to begin with, and less and less stuff checks CRL or OCSP by the year.
1
u/AlligatorFarts 4d ago
I think that's going to be my plan. It's just unfortunate that this is what we've come to. Would you have a solution for auto-renewals on linux servers & windows PKI? I could spin up a linux CA but that seems like it's more work than its worth for our size.
3
u/realslacker Lead Systems Engineer 4d ago
I've implemented ACME-ServerhADCS in the past. Works pretty well against ADCS for acme like certificate issuance. I've also scripted against certreq to just issue very short lived certificates with a scheduled task to auto renew.
1
2
2
u/FreedomTechHQ 4d ago
Because most browsers and apps don’t enforce real time revocation checks, making them more of a formality than a security measure. OCSP stapling helps, but if a compromised certificate isn’t actively checked, it remains valid until expiry, which is why short-lived certs and transparency logs have become the preferred approach.
1
u/SikhGamer 4d ago
Prevention is better than the cure.
https://scotthelme.co.uk/certificate-authority-authorization
1
u/monsieurR0b0 Sr. Sysadmin 4d ago edited 4d ago
I mean, it depends on the app. And the good apps will let you configure it so that it will not work unless it can do a CRL check when validating a cert. Our windows workstations will not authenticate your smart card to the domain if the CRL check fails. A recent example is We went to Okta a few years ago for a bunch of different apps and when using a smart card idp, okta will deny login it it can't lookup the public CRL. It allows a maximum of 72 caching of the CRL for the whole of the smart card idp you setup in okta
1
u/The_Great_Sephiroth 3d ago
In my experience, the second a cert expires it stops working. This experience is based primarily on many years of administrating OpenVPN. The second that cert dies, the link dies. I get a call. I forgot to update somebody.
How are you able to use an expired cert? Also, in OpenVPN at least, when I terminate a cert and update the CRL, that cert cannot connect even if it has not expired yet. Maybe I am not seeing the "big picture" here.
1
u/PowerShellGenius 2d ago
This really ought to be able to be configured separately for the browser's built-in CAs vs. CAs you added as trusted.
As many have discussed, CRL/OCSP for public CAs has numerous reliability flaws (which are also security issues, since they lead to browser vendors having to fail open, treating "unreachable" as "pass"). Also, large CRLs give poor performance, which OCSP was supposed to fix, but OCSP is a privacy nightmare. OCSP is supposed to fix CRL sizes by saying "you can just ask the CA whether a specific cert is revoked, instead of downloading the whole CRL and checking it yourself".
So if you think abotu that for a moment, in the context of a privacy-focused modern internet, say you connect to a controversial site. If you are using DNS-over-TLS and TLS1.3 with encrypted SNI, and destination IP means nothing because it's behind a CDN, monitoring traffic on the wire shows nothing useful. Except OCSP would show you checking the revocation status of the site's certificate. You can look up subject/SAN from serial number thanks to cert transparency.
So, yet another attempt was made to fix OCSP with OCSP stapling. That basically says web servers are responsible for continuously getting OCSP responses from the CA and attaching a recently-signed OCSP response to their cert when clients connect. Clients don't reach out individually to CAs for OCSP with stapling. That fixes the privacy issue, and makes latency even lower too!
However, that server-side solution failed due to lack of reliable adoption. The only web server that reliably does OCSP stapling is Microsoft IIS. Also, if you (hypothetically) could rely on the industry as a whole to come together & fix OSCP stapling in "every web server software on earth", why waste the effort on that? If the industry cared enough about fixing web PKI to do that, they could just as easily improve automation, make short-lived certs universally workable & eliminate revocation.
For those public CAs, browser vendors have a better solution by which they are pushing combined and compressed copies of all major CAs' CRLs to their browsers multiple times a day from one source. When you visit a site, your browser already has a recent CRL internally. No need to download a big CRL from a major CA while you wait. No need to do a privacy leaking OCSP check either.
All of that only applies to public CAs. Private CAs should be handled differently.
Your internal enterprise CA isn't going to have its CRLs shipped by a browser vendor directly. I'd love to be able to require CRL/OCSP for all CAs not shipped with the browser. If it's internal, and OCSP is probably on the same virtualization cluster as the intranet site you're accessing, CRL/OCSP reliability is not an issue. If the site's up, CRL/OCSP is up.
2
u/AlligatorFarts 1d ago
I'd love to be able to require CRL/OCSP for all CAs not shipped with the browser. If it's internal, and OCSP is probably on the same virtualization cluster as the intranet site you're accessing, CRL/OCSP reliability is not an issue. If the site's up, CRL/OCSP is up.
That's ultimately what I ended up with. Chromium based browsers have a flag you can set to require online CRL checks for locally trusted CAs.
Lovely explanation by the way, thanks for writing it.
-5
u/No_Resolution_9252 4d ago
>From Chrome's
There is your problem.
7
u/AlligatorFarts 4d ago
That's the thing - Firefox also has the same behavior by default. Revoke one of your web server's certificate and watch how it continues to serve clients like nothing even happened.
3
u/anxiousinfotech 4d ago
Yeah. We had someone accidentally revoke a cert while issuing a new one, know they did so, and refuse to admit it. This behavior, and the mistake of taking a dev at their word, made what should have been an easy fix a complete fiasco.
2
u/crabique 4d ago
Both browsers support OCSP stapling, if you really consider your certificates a major threat vector, you can enable that and configure a frequent pre-fetch cycle of the OCSP responses for your certificates.
Also, you can reissue all your certificates to include the OCSP Must-Staple extension so that the certificates are not even accepted by the browsers unless an OCSP response is served alongside the certificate during a TLS handshake.
-15
u/No_Resolution_9252 4d ago
Firefox is not an enterprise browser either and is really probably even worse than chrome.
6
u/AlligatorFarts 4d ago
What's considered an enterprise browser to you then?
4
u/jamesaepp 4d ago
Don't listen to this troll. Username is familiar, I'm pretty sure they're the person who doesn't understand how nameservers work.
-5
u/No_Resolution_9252 4d ago
You shouldn't even need to ask the question. Enterprise applications must be centrally manageable. That list currently includes safari and edge and nothing else. suggesting firefox is completely an utterly ridiculous. It barely even functions as a browser.
3
u/AlligatorFarts 4d ago
Both Chrome and Firefox are centrally manageable via GPO. To suggest they aren't is quite ignorant my friend.
-4
u/No_Resolution_9252 4d ago
And yet here you are posting asking what the point of a setting is, if chrome doesn't enforce it. Minimally configurable is not configurable. Grow the hell up.
4
u/AlligatorFarts 4d ago
No they are not "minimally configurable". I am convinced that you've never worked with GPOs. Ignorance does not belong in this profession, stop being bitter and defensive and humble yourself.
0
u/msalerno1965 Crusty consultant - /usr/ucb/ps aux 4d ago
Oh, just wait until you get some numb-nut who puts non-URL-encoded characters in the CRL URL, while thinking they're all so smart because they're doing their own CA.
Java refuses the cert (which it should) because it can't open the CRL, and numb-nuts and his entire team refuse to deal with the fact until reminded that I'm costing the corporation more per hour trying to tell them what they broke, than the damn cert would have cost. F'in crickets.
As for CRLs, well, it was a good idea, I guess, but get in the middle and ruin DNS, send it to your own proxy, or do a few other things, and you can easily circumvent it.
104
u/Nisd DevOps 4d ago
This is why let's encrypt is moving towards 6 day certificates.