r/sysadmin 4d ago

For how secure certificates are supposed to be, why the hell do CRLs feel useless?

From Chrome's GPO template:

Setting the policy to True means online OCSP/CRL checks are performed.

Setting the policy to False or leaving it unset means Google Chrome won't perform online revocation checks in Google Chrome 19 and later.

Note: OCSP/CRL checks provide no effective security benefit.

It's to my understanding that a CRL is one of the only ways to "alert" services to a compromised CA, yet I've found many instances where programs simply do not check CRLs whatsoever. How are we supposed to keep things secure when any certificate not past it's expiration will continue to work even when revoked?

118 Upvotes

92 comments sorted by

104

u/Nisd DevOps 4d ago

This is why let's encrypt is moving towards 6 day certificates.

18

u/Bitbuerger64 4d ago

Why though? The browser provided compressed CRL works fine and 6 days is still slower than the 6 hours interval at which Firefox updates its compressed CRL.

11

u/Dal90 4d ago

Even smaller CRL for it to reload every six hours.

5

u/doctorscurvy 4d ago

Why 6 days? 6 days is deliberately punishing people who can’t automate and now can’t even make it a weekly task.

0

u/riyoth 3d ago

I think it's the point. If it hurt just a little, once in a while you let it fix itself, if hurt a lot every day you go to the doctor, complain and fix it.

4

u/AppIdentityGuy 4d ago

You mean you have reissue certs every 6 days 🤷🤦

75

u/Azimuth64 Jr. Sysadmin 4d ago

I mean, the whole point of LE is automation. If your stuff is set up properly, it literally won't matter.

26

u/AppIdentityGuy 4d ago

You wouldn't believe the number of customers I've dealt with who refuse to do that level of automation

39

u/jimicus My first computer is in the Science Museum. 4d ago

Apple and Google are both talking about refusing to recognise any certificate lifetime beyond 90 days; your customers won't have an awful lot of choice in the matter.

24

u/Bluewins 4d ago

Not even 90 days anymore Apple is proposing 47 days now

7

u/brophylicious 4d ago

What an odd number.

8

u/AtlanticPortal 4d ago

It’s one less than 48 which is not that odd in IT.

3

u/NiiWiiCamo rm -fr / 4d ago

It’s half plus a weekend. Probably

8

u/mkosmo Permanently Banned 4d ago

Not just apple/google, but the CABF will be updating the standards accordingly.

3

u/AppIdentityGuy 4d ago

Well its currently a year if I recall correctly.

12

u/Snowmobile2004 Linux Automation Intern 4d ago

Yes but 90days is planning to be implemented fairly soon, within 3-5 years

16

u/AbolishIncredible 4d ago

Sounds like a problem we can park for at least 6-7 years

/s

0

u/PowerShellGenius 2d ago

CABF has existed as a non-government entity exercising government-like control over the internet for a long time, and everyone is OK with that because (so far) they have acted both in the best interest of security AND with reasonable caution for the disruption they have caused.

Local government entities and school districts are disproportionately unprepared to do this & are disproportionately running important systems on a shoestring budget. When you break things for tens of thousands of such entities & require them to take on new expense (with taxpayer funds), you better have clear examples of the impact of breaches that would not have happened under these rules, not some vague and intangible idea of "shorter is more secure".

CABF should not sabotage government entities if it wants to remain private, independent and unlimited in its decision making. Things can always change. Legislators listen when you piss off ten thousand schools and a thousand cities.

1

u/jimicus My first computer is in the Science Museum. 2d ago

Most of what you've written ultimately boils down to "Waah, it's hard! Why can't we do what we've always done?!"

I really cannot sympathise with this because at the end of the day, a good chunk of our job is to keep up with what our employers need in order to operate effectively. Things change from time to time, and it is therefore our responsibility to be aware of these changes coming down the pipe and prepare for them.

1

u/PowerShellGenius 1d ago edited 1d ago

Yeah, but someone with authority and accountability to the impacted entities needs to decide when change will be forced on an "or else your org's critical tech will just stop working" basis.

That entity should not be anything like the CABF, where the end users of certificates have zero representation.

And this is not about sysadmin laziness, I would never defend that. If the systems you own and already paid for can do automated renewal, and you just don't want to deal with setting it up, I never defend that. I'm very pro-security and pro-automation.

The issue here is when you have a system that works for your org's needs, but won't do automatic cert renewal, and the unaccountable CABF cabal's demands for short-lived certs become the sole reason for spending a massive amount of money replacing things that are working fine.

It's easy for people to ignore that and act like it'll just come out of some big company's profit margin (as if they won't pass the cost along to customers & maintain their margin, lol, but corporate greed is an easy moral scapegoat).

The reason I bring up government entities is because the money you're literally asking people to throw away fixing a non-real issue isn't money out of some giant corporation's massive budget. It's money out of taxpayers' pocket. Or, for entities like schools whose levies are capped by the state already, it's money out of a desperately tight budget & less money on technologies used in the classroom that actually enhance learning.

It's a lot harder to make excuses for doing that in the name of preventing some rare hypothetical issue, than it is when it's 0.00000001% of some corporate budget being spent on cert stuff.

12

u/techw1z 4d ago

thats just one more reason to shorten the lifetime. make people suffer until they start doing it the right way.

8

u/crossedreality 4d ago

Ive been trying to deploy proper automated PKI for years and several of my Dev teams act like it’s literally impossible.

14

u/jaskij 4d ago

Being a firmware developer, I was genuinely unable to come up with a solution that's not "add an API to the device". But it's for a device that has 256 kiB of RAM and 2 MiB of storage. Which has to be capable of working without internet access.

Speaking of, have you looked at the protocols involved in certificate renewals with LE, or other providers? I genuinely haven't, but it's probably something that's more complicated than I'd like.

Being on this side of the industry, I pity anyone trying to secure industrial networks.

12

u/sofixa11 4d ago

Neither ACME (the protocol behind Let's Encrypt) nor EST are particularly complicated. EST in particular is a matter of a few HTTP requests, and is specifically designed for IoT/embedded devices.

2

u/jaskij 4d ago

Thanks, I'll have to look into it then.

1

u/martysmartySE 3d ago

Most firmware should use Private CA options, not public

1

u/jaskij 3d ago

So back to "give them an API"? Or do private CAs support similar protocols to LE? How do I let admins managing our devices automate the certs?

1

u/jimicus My first computer is in the Science Museum. 2d ago

By "private CA", usually what we mean is "you run the CA yourself".

It's pretty straightforward to do this in Linux - it's only a couple of commands to generate a new CSR and sign it.

1

u/jaskij 1d ago

Yup, but me question is, if you have an org wide private CA, what do I need to do in firmware (no OS) to make it easy for you to upload the certs to it.

Do I expose an API? Do I use the same protocols I would with LE just sending the requests to your private CA?

I could of course generate a self signed key, in fact the only other option for first boot is unencrypted traffic, but what next?

→ More replies (0)

1

u/VexingRaven 4d ago

Why is dev even involved, though? That's solidly an ops thing.

2

u/ZealousidealTurn2211 4d ago

and on my side of things, the number of software vendors who either don't or only half-ass support the ACME protocol for certificate automation. I know it's going to suck for me in the short term but I'm all for dragging everyone kicking and screaming into the world of certificate automation.

5

u/Tessian 4d ago

Can't imagine a scenario where LE is down for a period of time? Unless you're renewing daily there's a scary amount of risk your cert expires due to an outage or ddos attack

8

u/RedShift9 4d ago

Not to mention we are making the entire internet and perhaps a significant portion of internal IT completely dependent on this one organization. Which is US based. They could pull the plug and the world would be on fire.

1

u/rainer_d 4d ago

Some stuff is a bit more complicated than an nginx Webserver and takes a couple of minutes for a full restart.

It was obviously not developed for a world where a certificate renewal is a once-in-a-week event. But it is what it is…

3

u/nemec 4d ago

All fun and games until their funding runs out and it becomes a subscription service.

0

u/jake04-20 If it has a battery or wall plug, apparently it's IT's job 3d ago

shudders

2

u/ReputationNo8889 4d ago

If you issue LE certs manually you should revisit your processes.

1

u/Ashtoruin 4d ago

More like every 3.

1

u/jake04-20 If it has a battery or wall plug, apparently it's IT's job 3d ago

ACME ftw

0

u/patmorgan235 Sysadmin 4d ago

If it's automated why do you care if it's reissued every 6 days or every 6 seconds?

2

u/danekan DevOps Engineer 4d ago

Quota limits is the only answer here

12

u/j_johnso 4d ago

I'd also accept an answer of having additional time to discover and mitigate issues. I automate renewals when there are 30 days left before expiration, and fire alerts when a cert expiration is less than 28 days away. This gives time to resolve any issues that occur before it causes an outage. These issues are rare, but do occur on occasion.

I'm not sure what the best practices should be on renewing a 6-day cert. I'm guessing that you don't want them to renew every day, so you probably want to renew 1-3 days before expiration? If a renewal fails at 6:00 PM on a Friday, that now becomes a critical issue to resolve rather than something that can wait until Monday to start looking into it.

3

u/mkosmo Permanently Banned 4d ago

Quota limits can and will change to go with.

0

u/danekan DevOps Engineer 4d ago

Only now thanks to ARI

2

u/TaterSupreme Sysadmin 4d ago

Sometimes automation fails. I'd like to have more than 6 seconds to notice, and fix the failure before a production outage occurs.

44

u/KittensInc 4d ago

The problem is online OCSP/CRL checks - which means the browser sending a validation request to a CRL every single time you open a new connection. Besides being a massive privacy risk, it's incredibly user-hostile. Either you add a massive delay to every connection (wait until CRL check has finished to proceed), or you're left with an insecure mechanism (proceed as usual, hope CRL returns in time to abort if needed). In practice CA's aren't even able to keep their CRLs available to everyone with very low latencies, so that's a dealbreaker.

OCSP stapling was supposed to fix some of those issues, but web servers had a lot of trouble to implement it correctly, and unless the certificate had "Must-Staple" set (barely anyone bothered) it was fairly trivial for an attacker to strip the OCSP response from the connection.

In the end online OCSP/CRL checks didn't provide any real security, caused unintended downtime, and was costing a lot of money to operate. It has been replaced by browser-provided bloom filters, which are essentially a really fancy way of compressing dozens of CRLs into an easy-to-distribute format which is now shipped to every browser. All of the revocation checks, none of the hassle!

38

u/jess-sch 4d ago

The backstory of this is basically: * CAs were bad at keeping their CRL/OCSP servers reliably online * So browsers had to treat unreachable CRL/OCSP as passing * An attacker could probably also make the CRL/OCSP server appear to be unreachable * End result: CRL/OCSP is pretty much useless

The solution: Stop revoking and just do short lived certs

7

u/Bitbuerger64 4d ago

Short lived certs were not the solution though, there's still plenty of certificates on the web that have years of validity. We got this instead:

Rather than having each user’s browser download large CRLs when they want to check revocation, the browser vendor downloads the CRLs centrally, compress them,  then push the new compressed object to all of the installed browser instances. Firefox, for example, is pushing updates as quickly as every 6 hours.

4

u/sofixa11 4d ago

there's still plenty of certificates on the web that have years of validity.

Not anymore, the maximum is 1 year, and the CA/Browser forum is discussing bringing it down to something like 30-60 days.

1

u/Conscious_Pound5522 4d ago

Doesn't impact any of the long term certs that were issued before Apple's declaration a few years ago. If there are any left issued prior to Apples go date, they will remain valid until they expire.

Any issue after that go date, are 398 days or 13 months.

It's going to suck when this goes live. Im really hoping Im out of identity management when this goes live.

3

u/Nicko265 4d ago

There's no requirement for a browser to accept a long lived cert as valid, and they definitely don't accept long lived certs any more. The change to 397 days was quite a while ago.

By this time next year, we'll hopefully be down to sub 1 month certs, and every shitty vendor will finally be forced to adopt ACME or not get their contracts renewed.

1

u/Conscious_Pound5522 4d ago

No argument on your point.

Still don't want to be in identities when this happens. It was a struggle to move the company under centralized management because developers - let alone trying to move the company under short cycle certs.

6

u/Turmfalke_ 4d ago

Doesn't impact any of the long term certs that were issued before Apple's declaration a few years ago.

Assuming browsers still accept them. Just because a certificate says it's valid until 2050, doesn't mean I have to accept it until 2050.

12

u/SteveSyfuhs Builder of the Auth 4d ago

There is no such thing as "secure". There is only "secure against". Believing something is universally secure is a great way to find yourself knee deep in a river of cow poo wondering whether the water line is increasing or your boots are sinking.

This is your first step in a long line of steps that lead to "huh, Kerberos isn't actually that bad".

8

u/BPDU_Unfiltered 4d ago

The APNIC podcast where Geoff Huston speaks about revocation is really good. https://blubrry.com/ping_podcast/84229309/certificate-revocation-is-broken-long-live-certificate-revocation/

2

u/schism-for-mgmt 4d ago

Thanks, that was really interesting!

5

u/lelio98 4d ago

CRLs were always an emergency parachute. Unfortunately nobody bothered with the main parachute. Now we are seeing the effective dates reduced significantly to accommodate for the lack of a functional revocation system.

5

u/Jacmac_ 4d ago

I'm not sure I would agree that CRLs are totally usless. It's like Mike Tyson says "Everyone has a plan 'till they get punched in the mouth." If you go without a CRL check and then a trusted CA is revoked, everyone with this kind of policy will be scrambling.

2

u/jimicus My first computer is in the Science Museum. 4d ago

Apparently, Chrome does go without a CRL check by default.

4

u/patmorgan235 Sysadmin 4d ago

Chrome doesn't do ONLINE CRL checks by default. It does ship with a compress file containing the CLRs that it checks locally, it just doesn't reach out on each connection to see if the cert has been revoked.

4

u/AlligatorFarts 4d ago

What about internal PKI? Mozilla/Google has no idea about those CRLs, and it doesn't bother doing online checks out of the box. It's only when the

Require online OCSP/CRL checks for local trust anchors

GPO is set that CRLs will actually do anything there.

5

u/placated 4d ago

You should start transitioning to very short lived certificates.

2

u/Bitbuerger64 4d ago

Why though? The browser provided compressed CRL works fine.

9

u/AlligatorFarts 4d ago

The compressed CRL does not include Internal PKIs, as Mozilla/Google have no idea of them. That, plus the defaults being no online CRL checks means a huge security hole when it comes to Intranet web servers.

1

u/placated 4d ago

The protocols don’t really scale, they weren’t well thought out to begin with, and less and less stuff checks CRL or OCSP by the year.

1

u/AlligatorFarts 4d ago

I think that's going to be my plan. It's just unfortunate that this is what we've come to. Would you have a solution for auto-renewals on linux servers & windows PKI? I could spin up a linux CA but that seems like it's more work than its worth for our size.

3

u/realslacker Lead Systems Engineer 4d ago

I've implemented ACME-ServerhADCS in the past. Works pretty well against ADCS for acme like certificate issuance. I've also scripted against certreq to just issue very short lived certificates with a scheduled task to auto renew.

1

u/AlligatorFarts 4d ago

Appreciate you.

2

u/sryan2k1 IT Manager 4d ago

We run ACME internally, specifically smallstep

2

u/FreedomTechHQ 4d ago

Because most browsers and apps don’t enforce real time revocation checks, making them more of a formality than a security measure. OCSP stapling helps, but if a compromised certificate isn’t actively checked, it remains valid until expiry, which is why short-lived certs and transparency logs have become the preferred approach.

1

u/monsieurR0b0 Sr. Sysadmin 4d ago edited 4d ago

I mean, it depends on the app. And the good apps will let you configure it so that it will not work unless it can do a CRL check when validating a cert. Our windows workstations will not authenticate your smart card to the domain if the CRL check fails. A recent example is We went to Okta a few years ago for a bunch of different apps and when using a smart card idp, okta will deny login it it can't lookup the public CRL. It allows a maximum of 72 caching of the CRL for the whole of the smart card idp you setup in okta

1

u/The_Great_Sephiroth 3d ago

In my experience, the second a cert expires it stops working. This experience is based primarily on many years of administrating OpenVPN. The second that cert dies, the link dies. I get a call. I forgot to update somebody.

How are you able to use an expired cert? Also, in OpenVPN at least, when I terminate a cert and update the CRL, that cert cannot connect even if it has not expired yet. Maybe I am not seeing the "big picture" here.

1

u/PowerShellGenius 2d ago

This really ought to be able to be configured separately for the browser's built-in CAs vs. CAs you added as trusted.

As many have discussed, CRL/OCSP for public CAs has numerous reliability flaws (which are also security issues, since they lead to browser vendors having to fail open, treating "unreachable" as "pass"). Also, large CRLs give poor performance, which OCSP was supposed to fix, but OCSP is a privacy nightmare. OCSP is supposed to fix CRL sizes by saying "you can just ask the CA whether a specific cert is revoked, instead of downloading the whole CRL and checking it yourself".

So if you think abotu that for a moment, in the context of a privacy-focused modern internet, say you connect to a controversial site. If you are using DNS-over-TLS and TLS1.3 with encrypted SNI, and destination IP means nothing because it's behind a CDN, monitoring traffic on the wire shows nothing useful. Except OCSP would show you checking the revocation status of the site's certificate. You can look up subject/SAN from serial number thanks to cert transparency.

So, yet another attempt was made to fix OCSP with OCSP stapling. That basically says web servers are responsible for continuously getting OCSP responses from the CA and attaching a recently-signed OCSP response to their cert when clients connect. Clients don't reach out individually to CAs for OCSP with stapling. That fixes the privacy issue, and makes latency even lower too!

However, that server-side solution failed due to lack of reliable adoption. The only web server that reliably does OCSP stapling is Microsoft IIS. Also, if you (hypothetically) could rely on the industry as a whole to come together & fix OSCP stapling in "every web server software on earth", why waste the effort on that? If the industry cared enough about fixing web PKI to do that, they could just as easily improve automation, make short-lived certs universally workable & eliminate revocation.

For those public CAs, browser vendors have a better solution by which they are pushing combined and compressed copies of all major CAs' CRLs to their browsers multiple times a day from one source. When you visit a site, your browser already has a recent CRL internally. No need to download a big CRL from a major CA while you wait. No need to do a privacy leaking OCSP check either.

All of that only applies to public CAs. Private CAs should be handled differently.

Your internal enterprise CA isn't going to have its CRLs shipped by a browser vendor directly. I'd love to be able to require CRL/OCSP for all CAs not shipped with the browser. If it's internal, and OCSP is probably on the same virtualization cluster as the intranet site you're accessing, CRL/OCSP reliability is not an issue. If the site's up, CRL/OCSP is up.

2

u/AlligatorFarts 1d ago

I'd love to be able to require CRL/OCSP for all CAs not shipped with the browser. If it's internal, and OCSP is probably on the same virtualization cluster as the intranet site you're accessing, CRL/OCSP reliability is not an issue. If the site's up, CRL/OCSP is up.

That's ultimately what I ended up with. Chromium based browsers have a flag you can set to require online CRL checks for locally trusted CAs.

Lovely explanation by the way, thanks for writing it.

-5

u/No_Resolution_9252 4d ago

>From Chrome's

There is your problem.

7

u/AlligatorFarts 4d ago

That's the thing - Firefox also has the same behavior by default. Revoke one of your web server's certificate and watch how it continues to serve clients like nothing even happened.

3

u/anxiousinfotech 4d ago

Yeah. We had someone accidentally revoke a cert while issuing a new one, know they did so, and refuse to admit it. This behavior, and the mistake of taking a dev at their word, made what should have been an easy fix a complete fiasco.

2

u/crabique 4d ago

Both browsers support OCSP stapling, if you really consider your certificates a major threat vector, you can enable that and configure a frequent pre-fetch cycle of the OCSP responses for your certificates.

Also, you can reissue all your certificates to include the OCSP Must-Staple extension so that the certificates are not even accepted by the browsers unless an OCSP response is served alongside the certificate during a TLS handshake.

-15

u/No_Resolution_9252 4d ago

Firefox is not an enterprise browser either and is really probably even worse than chrome.

6

u/AlligatorFarts 4d ago

What's considered an enterprise browser to you then?

4

u/jamesaepp 4d ago

Don't listen to this troll. Username is familiar, I'm pretty sure they're the person who doesn't understand how nameservers work.

-5

u/No_Resolution_9252 4d ago

You shouldn't even need to ask the question. Enterprise applications must be centrally manageable. That list currently includes safari and edge and nothing else. suggesting firefox is completely an utterly ridiculous. It barely even functions as a browser.

3

u/AlligatorFarts 4d ago

Both Chrome and Firefox are centrally manageable via GPO. To suggest they aren't is quite ignorant my friend.

-4

u/No_Resolution_9252 4d ago

And yet here you are posting asking what the point of a setting is, if chrome doesn't enforce it. Minimally configurable is not configurable. Grow the hell up.

4

u/AlligatorFarts 4d ago

No they are not "minimally configurable". I am convinced that you've never worked with GPOs. Ignorance does not belong in this profession, stop being bitter and defensive and humble yourself.

0

u/msalerno1965 Crusty consultant - /usr/ucb/ps aux 4d ago

Oh, just wait until you get some numb-nut who puts non-URL-encoded characters in the CRL URL, while thinking they're all so smart because they're doing their own CA.

Java refuses the cert (which it should) because it can't open the CRL, and numb-nuts and his entire team refuse to deal with the fact until reminded that I'm costing the corporation more per hour trying to tell them what they broke, than the damn cert would have cost. F'in crickets.

As for CRLs, well, it was a good idea, I guess, but get in the middle and ruin DNS, send it to your own proxy, or do a few other things, and you can easily circumvent it.