This article discusses the lack of certificate checking done by ACMEv2 clients, as well as the lack of provision in the ACMEv2 protocol specification to encourage any checking. This article explores the implications of this, and demonstrate why we should probably being doing some additional checks in our ACMEv2 clients.

The project is called “Meep Meep”, because that’s the sound a roadrunner makes. The author couldn’t think of a cleverer name for something related to ACME. There’s already a Go client library called meepmeep, last worked on in 2018, proving that all ideas are derivative.

In brief

ACMEv2, best known for its use with Let’s Encrypt, is a protocol designed to make it relatively simple to get certificates. The protocol has a number of verification steps in it, and requirements for the server and client to meet. However, once a certificate issuance has been agreed, and a certificate downloaded by the client, the requirements run out.

Summarising what happens:

1. The client establishes trust in the service ✔️
2. The service establishes trust in the client ✔️
3. The service establishes trust in the identities claimed by the client ✔️
4. The client does not establish trust in the certificates its given ❌

To see the evidence, you can jump straight to this article’s survey of clients. To see how to deal with it, here’s a proposed enhancement to one such client.

So what?

This means that the certificates we get from the likes of Let’s Encrypt aren’t checked to see if they contain what we expect. That means, when we come to use them in our web servers etc, they might not work, or might not be as secure as we asked for. Some quick examples of potential abuses:

• The certifcate we get has domains in its SAN that are different to what we requested
• We want a certificate for a TLS Server, but the certificate we get also has code signing in its Key Usage
• The signature type and length doesn’t match what we were expecting
• We asked for the OSCP Must Staple extension, but didn’t get it
  • Which has already come to some people’s attention

Who’s problem is that?

There’s nothing stopping us from checking these things ourselves, and, quite possibly, this was left intentionally out of the scope of ACMEv2. However, the author believes that some best-practices should be established for verifying certificates, something that perhaps all clients should perform, and something that might be worth considering as forming part of any future versions of the ACME spec.

This isn’t just about checking that issued certificates are valid, but also ensuring that they contain what was requested.

What do I do?

There’s a few things:

1. Carry on as you are, it’s probably not that big of a deal.
2. Have a conversation in your DevSecOps team about how you check that the certificates you get contain what you expect, and how you might want to strengthen that capability.
3. Read the rest of document and think about what you can do to effect a change to close the loop of “trust, but verify”.

In depth

There are several ways to get certificates from a Public Key Infrastucture (PKI). Manually setting up your own PKI by using openssl, or even using it to make self-signed certificates, is possible, but not usually very convenient or useful. Copying and pasting CSRs into services that will then give you a downloadable certificate was once the most common way of getting legitimate certificates for the web. Then there’s enrollment protocols like SCEP and EST, used on some networking and mobile devices. But by far the most prolific enrollment protocol is now ACMEv2, thanks to Let’s Encrypt. ACMEv2 is not Let’s Encrypt, but Let’s Encrypt had a big role to play in shaping ACMEv2.

Let’s Encrypt’s purpose is to give a free and easy way for people to make websites safer by providing certificates to site owners. Then, there’s no reason to use HTTP… most of the time.

The process

The basic steps for ACMEv2 are:

1. Create an account with the service provider (usually Let’s Encrypt).
2. Generate an order for some certificates with the service provider
3. Give the service provider a way to verify that the identities you wish to use in the certificates are owned and controlled by you.
4. Request and obtain the certificates

A sub-set of this process is used for renewals of certificates, and revocation of certificates is also possible. AJ ONeal has written a step by step node.js example of how to interact with Let’s Encrypt using ACMEv2.

The sticking point: Getting certs

It’s the last step in the above simplified break-down of ACMEv2 where things could go wrong. A client has proven who they are and that they control the identities (domains) they want to represent, and has generated a Certificate Signing Request (CSR) for the very same. The issuer will check this CSR to make sure it contains what was agreed, and then issue a certificate. Clients download that issued certificate and that’s it.

CSRs vs certs

CSRs are very similar to certificates. They’re an ASN.1 encoded data structure with a bunch of fields, one of which is a public key, and other is a signature. Assuming a chain of trust can be established with that public key, then the origin of the ASN.1 data structure, and its integrity, are verifiable.

A CSR is signed by its creator, and the private key used to sign it is owned by the creator. There’s no chain at this point - CSRs are self-signed. The main things that can be known by verifying a CSR signature are that:

1. The CSR wasn’t modified by anyone who isn’t the key-holder
2. Some cryptographic effort was used to generate a key and sign it
3. The public key in the CSR is correctly paired with the private key used to sign the CSR.

That’s why ACMEv2 ties CSRs to accounts, as well as orders with verified challenges. More than the CSR alone is required to determine if the contents of a CSR are something that should be put into a certificate and issued to the requester.

So what about a certificate? In a nutshell, when creating a certificate, some fields are copied from a CSR into a new ASN.1 structure, which has some additional fields in it as well. So a verified certificate proves that:

1. The certificate issuer accepted some of the fields from a CSR into the certificate.
2. The issuer has provided their details in the certificate (creating a chain)
3. The public key from the CSR is presented in the certificate
4. The certificate could only have been created and signed by the issuer, and hasn’t been modified by anybody else.

If a chain of trust stretches back to the issuer, and the issuer’s identity information and public key is available, then there is trust that they chose to issue this cert for the requester. Therefore, there is trust in the identify of the holder of the certificate. How that’s done is not an exercise for this article, but you can have some fun with openssl if you want.

So a certificate is very similar to a CSR, except it’s got a few extra and different fields, and is usually signed by a different key.

The big reveal

A question came up in my workplace recently: “Can you modify a CSR without holding the private key?” Of course not. Not without invalidating the signature, at least. But that question was meant to take us towards a more important one: “Can I use my CSR to control what goes into the certificate I get?”

The answer to that is… not really. At least, not with any guarantee.

The fact is, requesters are at the mercy of the issuer to take a CSR and use the information in it to build an appropriate certificate. If they don’t, it could be for one of several reasons:

1. The requester wanted something we’re not allowed to have
   • The validity period is not something we can choose
2. The requested capabilitity isn’t part of the product
   • Wildcard domains, code signing capabilities, etc, aren’t always included in the same product
3. The capabiltiies exceed those required by the certificate
   • A web server shouldn’t be able to sign code. If an ACME server provided such capabilities unexpectedly, the web server could be next on the list to compromise, thanks to its excessive privileges.
4. The issuer got compromised and somebody is mangling certs for nefarious purposes
   • A DoS on servers by giving out garbage certs, for example

The last two are quite exciting… and unlikely. But does that mean it won’t ever happen? Further, by doing better checking of issued certs, DevSecOps teams can squash issues relating to misunderstandings of what is being requested/purchased from a provider.

Most of these issues can be mitigated in other ways. For example, code signing certs probably shouldn’t share the same root of trust as web servers, and different trust stores should be used for each. But, the principle of defence-in-depth tells us that we shouldn’t rely on a single mitigation.

A closer look

If you’re interested in going deeper still, here’s a comparison of some (but not all) of the components in taking a CSR and creating a signed certificate from it.

You can generate similar outputs to these with openssl (req|x509) -in <infile> -noout -text.

Sections

CSRs and certificates have different headers, obviously, which are shown in grey here.

In blue is the public key of the CSR or certificate owner.

In orange, is the identity information of the certificate, or the subject. This is the “who”, and for ACMEv2 we have proven ownership of these identifying subjects, to the CA, through one of ACMEv2’s challenges. This inforamtion appears in two places: “Subject” and “Subject Alernative Name”. The latter is newer, and now required for modern browsers and TLS libraries to verify certs.

In yellow are usage extensions. In a CSR, these take the form of requested extensions. These denote capabilities the requester wants encoded into the certificate. In other words, these are the activities the requester wants to perform, with the identity and key information provided. For a certificate, these are the activites allowed to be performed. It’s up to clients interpreting certificates to enforce these.

Then, in pinkish-purple, is the signature. For a CSR, this is self-signed by the owner. For the certificate, this is signed by the issuing CA, the public key for which can be used to verify the certificate, but must be found elsewhere.

Finally, in green, and only in the certificate, is extra information added by the CA, such as the validity period of the certificate, identifying information for the issuer’s key, how to perform revocation lookups, and so on.

Copied/modified data from CSR to cert

So there’s a lot more in a certificate than a request, but some of the request makes it into the certificate. Arrowed lines indicate some of that transfer from CSR to cert.

The solid line indicates an essential cryptographic part. If the public key in the certificate does not match the public key we used in the request, then the certificate is completely useless, as the owner will not possess the correct private key to do any cryptographic activities backed by the cert.

The dashed lines indicate identifying information. These are necessary in order to use a certificate to identify the requester in the way they want. However, there may be some deviations. For example, some CAs will add an extra SAN entry for the www. prefix of a domain, even if one wasn’t included in the request. This is not the case for Let’s Encrypt, however.

The dotted line indicates that the usage extensions and constraints that were requested are, hopefully, reflected in the certificate, but additional constraints, policy information, and other information may be added. This is probably fine, but if the certificate gets unexpected capabilities, that might pose a problem for the owner.

If ACMEv2 clients are responsible for generating the CSRs (which they usually are), then it seems logical that they also check that the issued certificate respects the CSR to a reasonable extent. For externally generated CSRs, it becomes more of a grey area.

This has nothing to with ACMEv2

The issue detailed in this article is actually related to requesters and Certificate Authorities. This issue can exist regardless of protocol: ACME, SCEP, EST, some in-house solution or even manual CSR submissions. But, because ACMEv2 is so prolific, and has a well defined and extensible set of verification challenges, it seems right to look to it to see how we could do a little bit more on the client side.

Ultimately, only the application that takes the certificate, or the clients connecting to the application, can know if the certificate is suitable. But if there was a way of specifying to the ACMEv2 client exactly what is required, then the client could provide some assurance that those requirements had been met. The parent application could then do further checks, if needed.

A note on Certificate Transparency

Approaches such as Certificate Transparency go some way towards providing checks against misbehaving CAs. However, the main focus in that, so far, is to prevent a rogue or compromised CA from issuing certificates to people or businesses who don’t own the domains they’re claiming to.

However, there are other deficiencies or deviations that could find their way into certificates without immediately alerting those watching the CT logs. Further, ACMEv2 clients aren’t doing anything to check these anyway (at least not the ones looked at in this article). While it’s great that browsers can conduct extra checks like this, it seems like a good idea of a certificate enrollment client to do some checks of its own, too.

A survey

The ACMEv2 specification, RFC8555 doesn’t mention any need to verify the certificate that is returned. So, if clients don’t check the cert, they’re still following the spec. But maybe they should check the cert. Below is a sample of ACMEv2 clients and an assessment of what certificate verification they perform:

There are many more clients, so PRs to this table are welcome.

So, using GitLab’s validation of certificates obtained via acme-client for Pages as an example of an application that does some due diligence on the certs it receives, the ACMEv2 client itself leaves the job of checking the certificate to the parent application. That separation of duties is what really comes into question in this article.

What next?

In the brief section of this article, I listed three things that we could do, so let’s expand on those here.

Carry on as you are, it’s probably not that big of a deal.

The truth is, a web server will probably complain if the certificate loaded into it is completely nonsensical. And, hopefully, you already have decent healthchecks in place that you can stage and then deploy a cert after proving it doesn’t break things.

Not breaking things doesn’t mean everything is fine, but if the thing you care about most if people being able to securely connect to your website, then so long as their browser trusts the cert you’ve got, you’re good in that regard.

Have a conversation within your DevSecOps team

Certificate issuance and renewal is, hopefully, part of your deployment process, and updates are scheduled somehow.

It makes sense to ensure that you check your certificates before putting them to use. Just because you were issued one, doesn’t mean it will work. Ensure there’s enough coupling between the certificate issuance/renewal process, and the services that rely on them. That way, if one day Let’s Encrypt has a crazy moment, you’ll catch it, and keep using your old cert until they fix the problem. If certificate enrollment clients helped with this, that would be great.

Strengthen the specification

There are a few ways in which the specification could be strengthened.

Additional operational considerations

RFC8555 section 11.4 addresses potentially malformed certificate chains, and how the contents should be verified to only be a bundle of PEM-encoded certificates, to mitigate possible private key replacement attacks.

Section 11 may benefit from an additional sub-section that encourages the certificate chain to be verified in full, and the certificate contents and capabilities to be compared to those expected by the client, or based on the CSR.

Additional expectations

A future protocol version could include an indication of expected properties of the certificate. For example, certificate duration, exact key usage requirements, etc, could be stated. If the server cannot, or will not, abide by these, then it can reject the order, rather than signing a certificate that would ultimately be rejected by the client.

Similarly, definition of these expectations would form the basis for exactly what checks a client would need to perform.

This could also be addressed in the specification of CSRs, and could then apply regardless of protocol, although implementations of enrollment protocols would at least need to indicate whether they support these CSR extensions. This is similar in nature to the identification and handling of critical extensions in X.509 certificates.

Strengthen the clients

Standards be damned, there’s nothing stopping clients from offering “hardened” modes where some of these checks are performed now. In fact, the author has forked acme.sh to provide an example of such hardening. The modification performs the following:

• Verifies the issued certificate and chain against the system’s trust store, or a specified root of trust.
  • No specific consideration for alt chains is given, or how the root is obtained if it’s not in the trust store.
• Verifies the certificate has the purpose “TLS Server” set in the certificate
  • This may be excessive, or need parameterising, because while Let’s Encrypt and similar services are expected to provide certs for TLS servers, not all uses of ACMEv2 may require it.
• Checks that the common name in the cert matches the main requested domain
  • This fails to account for certificates with only a SAN and no subject/DN, which is possible
• Checks that the list of requested domains exactly matches the list of names in the SAN
  • To ensure none are missing, modified, or unexpected entries added
• Performs an OCSP query based on the certificates responder URL
  • Assumes one exists, and will fail if it doesn’t, which may be excessive
• If OCSP must-staple was specified when forming the CSR, checks that status_request is present in the certificate’s TLS features list.

It’s around 50 addional lines of ash compatible shell scripting, or less than 1% additional code. In terms of compatibility, it relies only on openssl, basic grep and basic sed for its checks. Clients built around more sophisticated languages could do more robust and sophisticated checking more easily.

A good potential next step from this is, after some validation by the communitity, formulating some updates to the Let’s Encrypt Integration Guide, so that more clients might adopt such enhancements.

Conclusions

Certificate enrollment protocols have made the adoption of PKI easier, but there are still gaps in the way we trust the interactions between clients and CAs. This article showed how one enrollment protocol - ACMEv2 - misses out on an opportunity to ensure clients get the certificates they expect, and as a result, clients supporting ACMEv2 happily accept whatever they’re given, without extra checks.

At this stage, we have a talking point more than we have a critical security issue. A broken or misbehaving CA could take advantage of these lack of checks, but then, there are much worse things that could happen in such a scenario. Nevertheless, having stricter checks, and standardising them, might reveal such issues faster, and lead to fewer downtime notifications.

It’s quite possible that there’s another RFC, standard, or best practices document out there that addresses this issue, but has been overlooked by the author. In which case, get in touch, so that this article can be updated to give those materials the spotlight.

Acknowledgements

The author would like to thank MICROSEC colleagues Ahnaf Siddiqi, Ragavan Kalatharan and Shazina Zaini for being part of the discussions that lead to this exploration.

Also thanks to Phil from ISRG/Let’s Encrypt for answering my security support query and recommending to engage with the client developer community as well as taking a look at the integration guide.