69. Designing a Better 2FA Mobile App

Chris: So before we jump into formerly Toopher, now Salesforce Authenticator, can you share or tell us what is two factor authentication and why is it important in this day and age with our use of technology?

Evan: Most people, by this point, have had at least some interaction with this, very typically through their banks. But the idea is to try to allow websites or various online properties to have more confidence that the person on the other end of a transaction is actually you.

Evan: Historically it's just been a password between anyone online and what you want to protect with a password, and more and more that's falling down. And so the industries across all sorts of, whether it's banking, education, or Facebook, social networks, Heroku, or Salesforce, those kinds of things, are looking for ways to increase the confidence.

Evan: It breaks down into three categories, typically. Something you know, that's the traditional password base, something you have, that's where Salesforce Authenticator comes in, and then something you are, which is biometrics. And the idea behind multifactor authentication is to have something that crosses between at least two of those categories. All three is even better, which now that mobile phones have biometric capabilities like face ID or touch ID, we're getting into the realm where we can do all three.

Evan: The trick is that most people end up not liking it because it's getting in the way of you accessing the things you want to access online and it's an annoying thing to have to do every time you log in.

Evan: It really improves your security stance because now someone can't just troll long lists of usernames and passwords and try them on different accounts and get in because they have to have this additional factor to let you in.

Evan: So the idea, a lot of what Toopher was focused on was trying to find a way to find that balance between security and usability.

Evan: And what we did was, as I said, the prompt was kind of vague, but it said do something with location. The idea was that Toopher would learn your typical usage behavior, and instead of you having to pull your phone out and hitting the allow button, it would say, Hey, it's the same person from the same browser, they're at home. It's the same time of day. All of this kind of context information to be smart about I'll hit that allow button for you so that you got this nice experience.

Evan: I still use it on a daily basis and it makes me smile. It's like, yep, I just did strong authentication but I didn't have to pull something out of my pocket. I didn't have to push a button on a USB key or touch a biometric endpoint. The system said does this look typical? Cool, let him in. And then the idea is would be it only bugs, the actual person that's trying to get in if something's unusual or important.

Chris: That's cool. That's interesting. I remember, maybe I'm dating myself, but 15 years ago, aa little bit more maybe starting my first job and I remember being given this, I think it was like an RSA token that was this little, I don't know, a couple inch long plastic thing that they said to put on my key chain and it had a monochrome LCD screen that just had a six character numeric code on it that cycled through and I was always like, ah, I'd have to carry this like big bulky thing around. It's on my key chain and then you know when I need to use it, because my keys are bulky and I left them, on a table somewhere, my front table, so I have to get up and go get them.

Evan: You're not alone. I mean, that technology has literally been around in one form or another since the 70s. And with the advent of smartphones, it really did need another look and say, How can we bring additional capabilities into this so that you're not having to carry around? And the thing is people that had to access multiple different things would have to carry around one of those tokens for each. And so a smartphone definitely made that a better idea for innovating there.

Chris: I often hear from friends that work in security. They say often that like there's always this trade off between security and usability. And it seems like one of the things that you were trying to approach was breaking down that that being a pure trade off and see how we can maintain the same level of security but make it more usable.

Evan: Absolutely. And when explaining what I did to friends and family and potential investors, it often was put into the bin of security, but it very much was trying to straddle the usability, the design thinking, as well. And that was the important distinction of what we were doing, and it really was just a reflection of my internal struggle between paranoia and laziness.

Evan: I was paranoid that there were so many things online that if someone just knew my username and password, they could get in. But I didn't want to be slowed down to try to get in. I just wanted to thwart anyone who wasn't me from getting into it, and so it definitely was pulling that thread that I just couldn't put down, why it wasn't just a semester long project. It was like, Oh, I want this. I want this everywhere.

Chris: One of the reasons why I joined Heroku was because of Heroku's is focus on improving usability, making the tools we use every day easier to use. So what about the original name of the tool or the company was Toopher, is that correct? And then somehow Salesforce found you?

Evan: Yes. So the name, just as a quick note, comes from, we talked about multifactor authentication. Sometimes it's known as two factor authentication. And naming things is something that is hard. But if you just say two factor authentication really quickly, you sort of get Toopher. And so that's where the name came from, and we had built out the API behind to enable this that we'd built out the mobile front ends. We'd gotten venture funding and were trying to go to market and were out talking to people.

Evan: And Salesforce is, I think from its inception, it's one of its number one values, if not the number one throughout that whole period, was trust. And so they were a logical person to go to and try to sell Toopher.

Evan: And I wasn't involved. It was more on the business side. I was leading the engineering team, but the feedback I got back from our sales team and CEO was, Yeah, we were talking to them. They really loved the idea. They basically just said, I don't think we're ever going to buy this as a product, but we might buy the company.

Evan: And the CEO is like, Yes, tell me more. Let's talk about that.

Chris: Okay. Okay.

Evan: And it really was from the executive leadership within Salesforce who saw something that they wanted to integrate deep into the DNA of their login stack, as opposed to just being a third party product. They wanted to own it and provide it to all of the Salesforce customers. It's not something we charge for. It is just built in because we, in general, try not to charge people to be more secure.

Chris: Right. No, agreed.

Evan: That's just a good principle for companies to live by. And I definitely cringe sometimes because in a world where companies are trying to offer what they have to people that are secure innovations, often I'll see services that it's free, but if you want to turn on strong authentication, you have to pay more.

Evan: And that makes me cringe. It's like, aah. You're basically saying, What's the value if we secure our product, and I understand it, because it's hard to come up with a business model that can actually reach people.

Evan: And so that's a way to differentiate your enterprise customers from just people who want to use it on a daily basis.

Chris: Sure.

Evan: And I'm biased because I've spent so long working in the authentication space, but I see that and it makes me cringe as like, Oh, I don't want that to be what differentiates your commercial customers from your normal customers.

Chris: Right.

Evan: Because we want everyone to be using strong authentication.

Chris: Yeah, totally. It's kind of like SSL, TLS certificates, which before Let's Encrypt, right, used to cost lots of money and even be more complex to implement and figure out, get working on your website. But now it's becoming the default or the expected norm that every website is served over HTTPS.

Evan: Yeah. And that was so painful for so long that the Let's Encrypt experience now just seems like magic. It's like how could it be that simple? And it's through a lot of hard work from those teams to build something that really does take the difficulty out of something that everybody should be doing.

Chris: Yeah, yeah, totally. Heroku uses them for ACM, which is automated certificate management, and I actually just got an email from Let's Encrypt yesterday saying we are one billion in just four years.

Evan: Wow.

Chris: They've issued one billion TLS certificates to do that. So it's pretty cool. The reality is sometimes adding additional security does cost more money to the business in people cost or implementation or R and D costs. But more and more it's getting easier in many different ways.

Chris: I guess maybe it's also getting more important, so it's becoming more expected. If I were to log into my bank or something like that. If they see that I'm somewhere else or if they see that I'm coming from a different browser, they'll send me an email and say, Hey, please click on this link first before we'll let you in. We just want to make sure that it's actually you. And sometimes they use SMS. And I know we could probably go down a whole rabbit hole of what's the problem with SMS and 2FA.

Chris: But in the interest of time, I'm curious to hear more about the technical path of Salesforce Authenticator. So you were running on Google Cloud, and you said that after that acquisition, you decided to move to Heroku.

Chris: So curious about, Google Cloud is great, has lots of cool features, and a lot more flexibility and power, maybe, than Heroku, but what prompted that decision for you to move Salesforce Authenticator from Google Cloud to Heroku?

Evan: Yeah. I want to make sure I say up front in case I forget to say it, but we were really happy on the Google Cloud platform. When we got acquired, we did make the decision of basically rebranding and starting fresh with our API. And so it was a logical time to evaluate architectural changes we wanted to make.

Evan: And that was what drove us to reevaluate our platform as well because we had been using, so this was very early on, it was before GCP was even an initialism, I think. It was basically just App Engine backed up by Bigtable.

Evan: And so we were using that, and it was great in a lot of ways because we weren't having to worry about managing our data store. A lot of the things that you get with Heroku, but it was a NoSQL database that had eventual consistency, and that often that became our go-to, well what's this bug? Was it an eventual inconsistency?

Evan: And so that's an anti-pattern and an engineering org of allowing for you to have sort of a catchall explanation for any bug.

Chris: Yeah, that's probably means we should think about fixing the root cause.

Evan: Yes, exactly. And so this was an opportunity to do that and we looked at the scale of what we needed to serve and it was clear that we didn't have to be on an individually consistent data store.

Evan: And I think looking back at making decisions about how it was architected originally, if I had advice to give to others, it would be, you don't necessarily have to build for that kind of scale from the beginning. You can scale very far on a Heroku Postgres instance.

Evan: Those things that are beasts and they can really serve a lot of the needs, and even projecting forward with adoption that's been growing year over year significantly and projecting pretty far down the road, we don't see a problem with what we're trying to do within a SQL store. And so at that point it became, okay.

Evan: And then you combine that with the practicality. We're being acquired by Salesforce and Heroku is a Salesforce company. If I look at all the options out there that a Salesforce developer has on what they want to build on something internally, hands down, it's Heroku.

Evan: That's what I want to build on. It just takes care of so many things that allow us to focus on the app we're trying to build as opposed to managing a data store. It is just, it's magical. I think it saves us easily two or three engineers over if we were trying to do this on AWS ourself, or on any of the first party data centers that Salesforce has.

Evan: And it is solid, and so much of the site reliability stuff falls on Heroku. We'll have an incident from time to time that that is an AWS instance that our Postgres database was running on will go down. But by the time we're paged or notified by it, it's already resolved itself, because it was on the other side of the fence and Heroku takes care of that for us.

Evan: And that's just, I cannot overstate how beautiful that is, how much that has enabled us to quickly, well do something like re-platform from Google Cloud to Heroku. We had an app that was built, it's a match made in heaven for Heroku, 12 Factor philosophy, and it was just super easy to take the code we had, but we were using an ORM that could do both, NoSQL and SQL based, and so since we weren't trying to migrate that data, it was super simple. We just swapped out the back end and started running.

Chris: Yeah. So I was going to ask, can you lay out a little bit of the architecture of Salesforce Authenticator or what are the pieces you use behind the scenes to run it?

Evan: Well, as I was saying, it really does fit in well with what a typical Heroku app would look like. We've got-

Chris: The 12 Factor app.

Evan: Yeah, a 12 Factor app. We've got the runtime sitting in front that we can scale on dynos connecting to Postgres for persistence. We've using Heroku Redis as well, for caching and just various kind of quick store doesn't need to be persisted data.

Evan: And underneath the covers, it's a Django app. It's using Django ORM. So we're using the Python buildpack, and it looks a lot like if you just started with a Heroku tutorial on run Django on Heroku, that's what it looks like. But it has grown and scaled with very little engineering resources needing to be devoted to the things that are just taken care of by Heroku.

Chris: Right. And then, so is it correct to say there's an API then that is used by the mobile app that I use every day when I'm doing 2FA into Salesforce?

Evan: Yes, exactly. So we do have, it's a fairly small team at acquisition. We're six engineers. We've grown a bit over the years since we were acquired. And so that that team has people that can play in a lot of different arenas, which they like to do.

Evan: So we've got Android expertise, that's Java and Kotlin. We've got iOS expertise that's Objective C and more and more Swift, and then the Python API that is running Heroku backs all that up so that when you log into a Salesforce org and your admins have turned on Salesforce authenticator for that organization, there's an API call made by the Salesforce tech stack that says, Hey, I want to know if this is actually that end-user. It reaches out to your phone, you hit allow, that gets back to them. You can turn on the contextual stuff that I was talking about where it'll learn your typical usage and then what strong auth looks like, you just provide a password. There's a moment where it checks with our back end and then you're in.

Chris: Salesforce Authenticator was my first experience with a push notification 2FA or multifactor authentication service or tool. So even that alone is a step up from, say, like having to unlock my phone, open an app, or find the app, open the app, scroll through all the other services I have 2FA set up with, find a number, hopefully read it correctly. Type it into the browser, tap approve. It's super simple.

Evan: Yeah, and we're certainly not the only ones doing the push authentication, but it is surprisingly rare for you to have that even as an option on a service that you're logging into.

Evan: And so we're definitely looking to increase the surface area and evangelize just for what strong authentication can be.

Evan: Like I said, we're not the only ones that do the push authentication, but that experience is such light years ahead of that RSA dongle that has the code.

Evan: I mean, because you're being proactively, like your phone buzzes, you don't even have to unlock it. You just tap the notification, your phone does a biometric check before it shows you the details even. And then you get an approved button and then you go on that.

Evan: I mean, it's a wonderful confluence of technology that allows the usability to really make a quite the leapfrog of what we had before all of those technologies could come together.

Chris: Earlier we had talked about this concept of, I mean, maybe it's overblown in tech circles these days, but the hashtag boring tech, don't over plan or don't like over plan for capacity or scale or things like that before you know you actually will need it because that generates lots of other things you have to worry about, things you have to implement and then things you have to maintain going forward. If you don't actually have a need for that scale, it's wasted time and effort.

Chris: So yeah. What were some examples of how you have approached not over-planning or planning enough and implementing enough to make the app work but not planning for huge amounts of scale before you actually need it?

Evan: Yeah, so I think it does come down to just keep it simple. It's easy to chase the new shiny, like you said, it's becoming cliche of that being the advice.

Evan: But there's a lot of promise behind some of those technologies, but often you do, it's the hype curve. You get really excited about it, and then you get to the practicalities of, okay, well there are some trade offs here, and it may appear to solve your problems upfront, but there's a reason some of those other technologies have stuck around for a while. It's because they're rock solid at what they do.

Evan: And I think that often it's hard, as humans, to you see one benefit, but it's hard to really evaluate what trade offs come with that. And it's easy to, if you're working with one technology to quickly find what those downsides are.

Evan: And so the technologies that have been around for a while, have had more momentum of gripes about what those may be, and I think they get more-

Chris: Yeah. They're more battle tested maybe.

Evan: Yeah. But the truth of it is like I don't want to think about my data store more than I have to. I want to think about the schema.

Evan: I don't want to think about managing that. I just want to figure out how to persist that data and make sure that it's going to be there when I need it, and that it's going to be consistent when I want it to access it. And I think there are applications, absolutely. There are applications that cannot scale on a single SQL database, but there are a lot of applications that can, particularly if you're talking about a microservice that is doing one thing and doing it well.

Evan: Really, really evaluate at what point will this break down using something that's more battle tested? And I think it is important to plan ahead about, okay, well what happens when we do reach the end of scalability on this? Am I going to move to a sharding strategy? Do I have a path where I can move to a NoSQL data store? But that's more future planning than just hop on it from day one.

Chris: Yeah. You were talking about Postgres earlier, and yes, for sure, Heroku Postgres is great, but I also think that like the Postgres, the open source project has just improved so much over the past five, seven, eight years, maybe 10 years. It seems to be on this, this breakneck pace, actually. Breakneck, but also measured. It's very high quality, everything that gets shipped from the Postgres project, or those that are contributing to the Postgres open source project.

Evan: In my experience, it is an exemplary project of what open source can do and just boggles the mind. I mean, how have they managed to do what they've done?

Chris: Right.

Evan: It's really amazing. And Heroku seemed to have gotten on the bandwagon with Postgres very early.

Chris: Yeah, I mean the project itself is like this silent freight train. It just keeps moving and it does well and it does what you ask it to do and it doesn't get a lot of press, it seems like, or it's not in lights a lot, but it does it job and does it well.

Evan: One of my favorite things, because we were users of Heroku before we were acquired for, like our dev portal was on Heroku. And one of the things that I really enjoyed as part of the acquisition was getting to put faces to the people behind the stuff we'd been using and to talk to Heroku's department of data and have them lay out the scale at which they are operating Postgres databases. But just kind of boggled my mind. I pretty sure they stand alone in the scale of operating both the count and size, and just resiliency availability. I was impressed with Heroku before and that just only made me more impressed.

Chris: Yeah, it's been pretty amazing. I'm continuously shocked by how few people can do so much, both in new R and D and also in maintaining and operating the service.

Evan: Yeah, and they've really embraced the automation and it's not something that people should be hands on with on a day to day basis. Figure out what could happen in the system and, and what the reaction to it be should be and then automate it. And they've really made it an art and a science out of that.

Chris: Yeah, yeah, that's true. Heroku was also one of my first exposures to the engineers who build the software also operate the service. In pre 2010 maybe and even earlier than that, when I started getting into software development, the standard was that there were software engineers and then there was operations people, and the software engineers built the stuff and then they threw it over the fence and the ops people ran it on the servers.

Chris: And I don't know if it was because of that, but when you, when you have that setup, which is how Heroku runs all of its services, when the software engineer who wrote the code is getting paged at 2:00 AM, that's pretty good motivation to the next day or the next week to fix the issue or automate the response to whatever that issue was.

Chris: And so that has been always part of the engineering culture of Heroku, which was cool for me to learn, to see that and learn it and continue it, I guess, or become part of it.

Chris: So what is, as much as you can share, what's next for Salesforce Authenticator, whether it's maybe there's those two competing forces. It's like increasing security and trust or maintaining that while also improving usability?

Evan: We're really focused on adoption, which I love. We're getting a lot of executive alignment. For Salesforce products, we need to be pushing strong authentication. We should be targeting 100% adoption, which if you know anything about typical adoption of strong authentication technologies, particularly things like this, we'll put it this way. The people that run it are thrilled if they get in the double digit percent-wise adoption of it.

Evan: And so it is great to have executive leadership saying, No, we want to push this well beyond that, and our goal is going to be 100% and to try to figure out how we're going to do that.

Chris: And this is for Salesforce customers, right? Because corporations such as Salesforce internally can say it's required for you to use to 2FA. But for instance, if I'm a Salesforce customer, I have the option to use multifactor authentication or not when I log in. That's correct.

Evan: Yeah. Yeah, exactly. And so obviously all Salesforce employees long ago were forced to start using Salesforce Authenticator, which is its own cool thing, to work on a project that all of your peers use every day, which can cut both ways.

Evan: Oh, you're the one that's doing this thing that I don't like. You definitely get good feedback when everyone in the company is dog fooding it.

Chris: Right.

Evan: But yes, we want to take this to our customers and get them to drive for their employees that are using our products, that they're using strong auth for all their logins.

Evan: And the strategy we're taking for that is, so historically we've just been the API and the apps, and as Salesforce continues to grow through acquisitions or through new stacks, there are all these different verticals who are at different stages of adoption of strong authentication or using different technologies. And so we're trying to bring that together as a more globally run service, and to aid in the adoption throughout all of our tech stacks, it became clear that we don't want to ask all of them.

Evan: So we've got Marketing Cloud, we've got Commerce Cloud, we've got Crux, Tableau's joining us. There's all these different tech stacks. We don't want each of them to have to build UI around it. And so we are, and we did a little bit of this back because we had the same barriers to adoption back in the startup days, but we're returning to it, of actually putting in a front end that anyone who wants to use this can use, that will just redirect through us. They tell us who the user is, we'll manage both enrolling and doing the verification and then kick them back out to where they're coming from.

Evan: And so that's what we're working on. The interesting thing about that is it is a different muscle to flex for us, because the API we were running could get away with running in a single region because pretty much everything we were doing operates at the human timescale.

Evan: So if I'm sending a message out to a phone and they have to reply, it doesn't matter all that much if that has a longer travel time, because I'm in Sydney connecting to our Heroku region back in North America.

Chris: Right.

Evan: But once we're moving into the UI, it's clear, okay, we're going to have to start doing some multi-region stuff. So we're exploring that. Heroku for a while has been in multi-region, but we had been focused pretty much on a single one. And so now, with putting the UI in front, we've got to be fast everywhere. And so we're splitting off a front end and running it geographically distributed throughout the world. And that's a new thing to operate an app, what looks like one app, in multiple regions.

Chris: Right. That brings up, seems like it would bring up lots of the questions of eventual consistency again.

Evan: Yes. Yes. And so earlier, I was all about, no, you can do it with just one database, but now that we're actually in the position where we've got to be quick everywhere, we are seeing, okay, we're going to have multiple databases. How do we do that in a way that is quick, that doesn't suffer from eventual consistency? And it's been a real, I'd say, fun engineering challenge to come up with disaster response types of scenarios. Okay.

Evan: And then testing those out, like, okay, what happens if our Virginia based space just goes away? And how can we keep the service up and running? And, to some extent, there's, Heroku has been helpful. Some of the stuff that Heroku, like cross-region followers, a database that we could just switch over to a different database, but we're also starting to have to realize, okay, we want data locality to be close to those that are using it.

Evan: So it'd be good to move some of the data we need for quick responses to Sydney, but also do it in a way that if we lose Sydney, that things keep going. And so yeah, it's a whole new world. It adds a lot of complexity. There's some things that we'd love to see for Heroku do to help us with that.

Evan: Yeah. I think it's going to be a fun process to push this out.

Chris: Yeah, that sounds cool. Well, and luckily you have semi easy access to Heroku product managers and engineers to chat about not only the best way to do this stuff, but also share your request.

Evan: Yes and that is great. And the good thing is in talking to the product's ownership, we're not the only ones that are interested in this, so it's definitely on their radar. It is, like I said, being able to dog food something in a company helps deliver things that any customer would be interested in.

Chris: Cool. Well thanks for joining us on Code[ish], Evan. I hope your continued use of Heroku is good and excited to see where Salesforce Authenticator goes in the future. So yeah. Thanks for sharing your story with us.

Evan: Oh, it's my pleasure.