19. Securing the Web with Let's Encrypt

Craig Ingram: Great. And I'm terrible for using an acronym, but without defining it, can you tell me what the ISRG is?

Josh Aas: So Let's Encrypt is the name of a service, it's not the name of a company. The public benefit, non-profit that runs Let's Encrypt is called Internet Security Research Group. And we just say ISRG.

Craig Ingram: So what prompted you, while you're doing your work at Mozilla, what prompted you to get started on ISRG and Let's Encrypt?

Josh Aas: Well, I was working in strategy there, one of the things that we were thinking a lot about is how to make HTTPS more ubiquitous. At that time HTTPS percentage of page loads was something like 35%. So, every time you load a page, 35% of the time it would be HTTPS, which is just too low. So we're looking at how to make that better. And there wasn't too much that could be done on the browser side of things if servers don't turn it on, the browser can't really do anything.

Josh Aas: So we looked at why more servers don't turn it on. It had seemed like the answer was that it was very hard to get and manage the SSL or TLS certificates that you need to turn on HTTPS. So we're looking at how to make it easier to get and manage those certificates. And the only solution that we really came up with that was viable that would get us to more HTTPS on a relatively quick timeline, was to have a new certificate authority that gave away certificates for free, all around the globe, in a really easy to use way. So that's what we did.

Craig Ingram: That's fantastic. One of the reasons I'm real excited about talking to you, and talking about Let's Encrypt is that it's really been a change in making the internet a safer place for more people by having a process and method that's allowed even hobbyist developers for any app they're working on to start with a more secure state with TLS certificates, taking care of not worrying about self-signed certificates or no certificate at all. So, that's great.

Craig Ingram: So what was the process to get started with Let's Encrypt? And how did it work to start issuing certificates?

Josh Aas: It was a big learning process for me. When I started, I did not know too much about how one might run or build a CA. I did a lot of research. Being a CA is all about being trusted, both in the minds of the people on the internet, but also technically. So, if you want browsers to trust the certificates that you issue, you need browsers and operating systems to add your root certificate, so every CA has at least one root certificate. And all the certificates that are issued for websites, are issued under that root certificate. So that root's got to be trusted by all the browsers, and the operating systems out there, cell phones, game consoles, desktop computers, everywhere.

Josh Aas: That takes a long time to get one of those roots trusted. I think a pretty reasonable estimate is somewhere between six and 10 years, depending on what kind of propagation you want, how widely you want that root to be trusted.

Josh Aas: So, if you want to start a CA from scratch, you really have two choices. You can either wait that six to 10 years and during that whole time, you're going to have to have your root audited, and keep it up and running and functional. So that's a long time to operate a CA without being able to issue certificates that are trusted. So what most new CAs do is they go out and make a deal with an existing certificate authority, where the existing certificate authority cross-signs the new certificate authority. Meaning, they use their root certificate to cryptographically vouch for the new CA.

Josh Aas: So, we didn't want to wait around six to 10 years to be able to start moving more of the web to HTTPS, so we found a great partner in a certificate authority called IdenTrust, and they cross-signed us. So we became trusted through IdenTrust in 2015, I believe it was. And then since then, we've been working on getting our own root trusted so that we don't need this cross-sign forever. It's not that we don't enjoy working with IdenTrust, but we'd like to be a fully independent CA.

Josh Aas: So, we've been working on getting our own root trusted, trusted by all the major programs right now, which are Apple, Google, Microsoft, Oracle, and Mozilla. But it takes some time for that trust to propagate through the ecosystem. So for example, a year or two ago, Google started trusting Let's Encrypt in Android, directly. But it takes a long time for all Android devices to update to a version of the operating system that trusts us. So we became trusted for example, on Android 7.1.1, something like that, but we need to wait til all devices are past that OS checkpoint. And that takes awhile. So in the meantime, while people are out there updating their devices over time, we're still trusted by IdenTrust, and therefore, trusted in browsers and phones around the world.

Craig Ingram: That's great. What else is going on with Let's Encrypt? What are some other, some news from the Let's Encrypt product side?

Josh Aas: So we're a pretty focused organization. Our goal is to get as many websites to use certificates as possible. So we're not a particularly feature-driven organization. We're not constantly coming out with new features, we're really just trying to refine things and make it that much easier so that the next round of websites can move to HTTPS. So, for us, it's mostly about refinement and continuing to get certificates to more and more websites.

Josh Aas: Yeah, we're not a very feature-oriented organization. We really focus on making sure that we're stable as possible, that we perform well, that the software ecosystem out in the world that people use, interact with Let's Encrypt, works really well. And we prepare for growth, so we serve between 160, 170 million websites today. And our engineers are spending their time making sure that we're ready to serve 300 million.

Craig Ingram: One feature that I know a lot of people were excited about was the support for wildcard certificates from Let's Encrypt. Can you tell me a little bit more about that? And why that was an important thing?

Josh Aas: Yeah. Wildcards were an interesting feature internally. So when we started Let's Encrypt, we made a pretty conscious decision not to issue wildcard certificates, and we were not planning to ever issue wildcard certificates at the time. But over the years, we got a lot of feedback, talking to people about what they need, and what needs to be done to spread HTTPS even further. It became clear to us, maybe three years in, that we were going to need to offer wildcard certificates if we're going to fit all of the use cases that people have, and take HTTPS to the unencrypted part of the web that remains.

Josh Aas: So, for a long time, it wasn't even really on our radar, we just weren't going to do it until there came a point where we just realized this is something that some people really do need. And it's not too hard for us to offer it. It's not like a technically difficult thing on our side. The big question on our side is, is us doing this good for the web ecosystem? And the deal with wildcards that makes them complicated is you end up using the same public-private key pair for a lot of subdomains. So, you might have a single domain example.com with a thousand subdomains under it, and you use a wildcard for it. And a wildcard does make it really easy to manage the certificate for those thousand subdomains. But now you've got a thousand subdomains sharing the same cryptographic keys.

Josh Aas: So in general, we like to encourage people to use separate certificates for different websites, including subdomains. So that was our hesitance about wildcards is, are we going to promote a practice that we don't really want to promote? Are we going to get too many websites sharing the same private keys? But I think there are cases where that really does make sense. And so, we decided to go forward with wildcards. I don't think our fears have turned out to be truth, it's still pretty common. People do the right thing most of the time. But, when people want to use a wildcard, it's there.

Craig Ingram: Yeah, I have to wonder if the changes in how certificates are issued through Let's Encrypt, and making it easier makes it, the concerns around wildcard certificates can not happen as often because it's easier just to get a certificate for the subdomains to begin with, and not just start with a wildcard and have it cover everything.

Josh Aas: Yeah, and that's our strategy for everything, is make it really easy to do the right thing.

Craig Ingram: So you mentioned working on refinement and more so than new product features. One of the things I was reading about was, something that might be considered a refinement is having the ACME protocol become an IETF standard. Can you tell me about why that process was started, and the importance around having it become an official standard?

Josh Aas: Yeah, that was an exciting moment for us. So when we started Let's Encrypt, there were not too many people who knew about the project, and we needed to make a protocol quickly. And we didn't have the years that it takes to produce its true internet standards so we built a protocol which we now refer to as ACME v1, that would work well for Let's Encrypt. But it was mainly built by Let's Encrypt people for Let's Encrypt, although in theory it could be used somewhere else.

Josh Aas: And, that's what we needed to do to get up and running. But in principle, we really want to operate on open web standards. So not too long after we launched, we started the process in the ITF to get ACME standardized, or a new version of ACME standardized. In some ways it was a pain for us, like the original ACME wasn't bad enough that we would have thrown it out, it worked quite well. It still works quite well today. But, this is one of those cases where it's worth doing the work and taking some lumps to make sure you're on a real standard.

Josh Aas: And it was also a chance to improve things. So there are some things that are better in ACME v2, so we decided to replace our original protocol with ACME v2. And like I said, it was a tough decision because we don't want to break compatibility for people, right? Many tens of millions of websites were using Let's Encrypt on ACME v1, and still do today. So moving to a whole new protocol is a big deal, right? It's normally not something we would ever do, but being on a standard protocol that many other web stakeholders have had a chance to have input on is just that important to us, so we decided to do it.

Craig Ingram: So I have background in web security and application security, and trying to keep up to date with the latest trends and security and threats and things like that. From your perspective, what do you see as the future of web security and other things that we can be doing to make the web safer?

Josh Aas: When people ask me what do I think is the biggest weak point on web security today, the answer for me is quick and easy, and it's BGP. So, for people who don't know, BGP or Border Gateway Protocol is essentially the protocol that's used to decide how to route information through the internet. And if you can mess with BGP, you can mess with the way that information is routed through the internet. And BGP is essentially completely not secure. There's no authentication, it's a big problem. And, it wasn't so bad when people didn't pay attention to BGP, but people are really starting to pay attention and see that it is a very weak ecosystem.

Josh Aas: And the way I like to think about it is like this. You know, it used to be that everything was not secure on the web. And then we started doing a much better job of securing the application layer, these days we're doing much better job of securing the transport layer with TLS. And, as we secure the stack from the top-down, it pushes attackers further and further down the stack, right? They're looking for the next weak link on down. And BGP is really the weak underbelly of the web.

Josh Aas: It's where we are starting to push attackers because that's where we have not invested in security. So you'll sometimes see big internet outages as a result of BGP problems, which are either innocent misconfigurations, or something malicious. And I think we're going to see a lot more of that. And I think there're probably targeted attacks being carried out with BGP, that are just really hard to detect. So it's hard to say how wide-spread issues with that are. But, yeah, my answer is I think we need to do something about the security of BGP next.

Craig Ingram: Very interesting. So, is BGP hijacking ... You know, I'm familiar with it where I've seen a BGP hijacking where suddenly Facebook traffic is redirected to some other site that it shouldn't be. Is this a concern for a CA, from a certificate trust standpoint, around BGP hijacking?

Josh Aas: Yeah, if you can hijack BGP in the right ways, you can mess with the certificate issuance process. So we're very much paying attention to this. It's difficult and somewhat impractical for most people to pull this off today. But there have been demonstrated attacks on CAs using BGP hijacking. So there's a research team from Princeton, did a great job demonstrating this, and one of the most shocking moments of my time writing Let's Encrypt was sitting in a room and watching them pull this off, watching them get a certificate from a website they shouldn't be able to get it to by just redirecting traffic around the globe at a flip of a switch. It's terrifying.

Josh Aas: Luckily most people are not able to do that. But we were really impressed with what we saw, so we teamed up with this team from Princeton to do more research on this, and figure out how CAs can mitigate the problem. So we're working on some exciting mitigations for this. And I think most likely, what we're going to end up doing is doing certificate validation from multiple perspectives at the same time. Right now when you validate, we tell you, you need to do something on your website in order to get a certificate, you do it, and then we check it. And we typically check from only one vantage point, we just say, did you put this file in the right place? Or did you fix this DNS record in the right way?

Josh Aas: In the future, we're probably going to need to do it from multiple vantage points at the same time so that if you want to try to pull off a BGP hijack attack, you're going to have to pull off a successful attack on multiple different routes at exactly the same time. And, I think that will just make it very impractical. But those are the kinds of mitigations we're working on.

Craig Ingram: Yeah, if you're able to do a BGP hijack for most of the internet, then that's a much trickier thing to prevent, and definitely sounds a lot less practical than what we see now.

Josh Aas: It's worth mentioning a safeguard that's been introduced for the certificate ecosystem in the past years that's really important, and it's called certificate transparency. Or we often just say CT. So certificate transparency is built on certificate transparency logs. So a number of organizations run certificate transparency logs. Google runs them, some of the other CAs run them. And what happens with certificate transparency logs is that every time a CA issues a certificate, they submit the certificate to these logs, usually very quickly after the certificate was issued. And in some cases, even before the certificate is issued. I won't get into the details there, the point is, certificates get issued, they go into these logs. And these logs are set up so they can't be tampered with, so you can't modify the history of what was put in the log. They use this algorithm called a Merkle tree to make sure that you can't modify a log, or that will be detected.

Josh Aas: So anyway, we got these logs that keep track of all the certificates that are issued, and those are searchable, they're auditable, things like that. So if somebody does get a certificate that they shouldn't be able to get, at the very least, it's going to become public relatively quickly. Because it will be in these logs. And if it's not in these logs, then the certificates aren't trusted. So browsers have started making sure that the certificates they trust are in these logs, so that if somebody does manage to issue a certificate that they shouldn't have issued, they will most likely get caught, because it will be in these logs. And if it's not in these logs, the certificate won't be trusted. So certificate transparency is a really important safeguard for the CA ecosystem, it helps with a lot of different potential issues.

Craig Ingram: Now, is this something that companies could, or should be monitoring to see if I own Heroku.com and I see in certificate transparency logs that a certificate was issued when it shouldn't have been? Is that a good safeguard the company should be doing?

Josh Aas: Totally. I would definitely recommend that companies, people, monitor CT logs and make sure that the certificates being issued for your domain are the certificates that you expect. We sometimes get people emailing us and say hey, we saw the certificate for our domain from Let's Encrypt, but we don't use Let's Encrypt. And so far, none of these have turned out to be malicious. Usually, what it is, is a learning opportunity for the people running the domain. Sometimes they have contracted out subdomain, like for an event or something to another provider, and that provider uses Let's Encrypt, and we point this out to them.

Josh Aas: Or, in some cases, Let's Encrypt certificates are really easy to get. And you might not even know that your software is doing it. So you can get certificates for a domain, and it just happens basically in the background, and it's not something you think about, and you see it in a CT log, and you think, yeah, I didn't do that, when really you did. Usually it's a learning opportunity for the people running those domains, so it's valuable in that way. We've not had a case where somebody has reported a certificate that actually wasn't issued by them.

Craig Ingram: So what are some of the things that you've learned in the years running Let's Encrypt and being a CA, that would be helpful to pass on, if somebody else wanted to start a CA? Or also, help make the web a better place?

Josh Aas: I think that the most important perspective for me is that trust means doing the right thing in a lot of different contexts. So, for Let's Encrypt to be trusted, it's not just about making sure that we issue certificates correctly. It's about, you know, when you're one of the biggest CAs in the world, you're going to make mistakes for example, and when you do, you need to be forthcoming about them, you need to disclose them quickly. Be really honest with people about what happens there. When you communicate about anything, you want to take that same approach, be honest, give people every reason to trust you.

Josh Aas: So, doing the right thing in almost every context is important. It all ties together, right? So whenever we make a decision about anything at Let's Encrypt, we really think about how does this affect how people might trust us.

Craig Ingram: Well Josh, I want to thank you for your time today. I've really learned a lot about the history of Let's Encrypt. I'm very excited to see the things that come around certificate transparency and tackling the next big things like BGP hijacking. And excited to hear that we're pushing attackers further down the stack and making their lives a little bit more difficult. So thank you very much for your time, and thanks for talking with me.

Josh Aas: Thank you.