Special Episode: When Giving Back Saves 1000s of Jobs

Greg: Oh, that's pretty cool. So of course, the topic on top of mind is COVID and how that's changed the banking and the credit markets. What was your guys' response and how did you help out with that?

Brett: As COVID happened, as unemployment rates started going up and we were looking at what the Fed was doing with interest rates, and they acted very quickly to take them down to zero. And we were looking at what the economic impact of this situation was going to be. The government came out and said, "Look, we really want to try to keep people employed and we know that as businesses get shut down because of this virus and the situation, we want to do something to help those businesses retain their employees. We want to have somewhat of a safety net for these small business owners because they're really the backbone of the economy." And so they decided to do this paycheck protection program with the idea that they want to protect paychecks. And so as we got into the situation, we started hearing about the rollout of this paycheck protection program from the financial institutions that we work with.

Brett: And because of the popularity of the program, banks and credit unions were, they're getting inundated with applications to apply for this small business loan, these PPP funds. And really I think most people were applying as you know now because there was a part of this PPP loan that was going to be forgivable and maybe even all of it forgivable. And so, assuming that you were to use the funds primarily for payroll to protect paychecks like was the intent, you could also use it for rent and some utilities, but it was really there to keep your people working and keep them having a paycheck.

Brett: And so we saw the problem that was existing in the market and in the United States, then we started hearing about it from our financial institutions and we thought, "You know what? This is a good opportunity for us to really fulfill our mission as a company to empower the world to be financially strong. Having a paycheck and protecting paychecks I think is really important. And then also helping those small business owners." So it met the mission and then two, our clients are primarily banks and credit unions, so we could also help them at the same time.

Garrett: Yeah. We have a little over 2000 banks and credit unions on our platform today and anything we can do to help them, we are definitely game to jump in on. And it was interesting because when the SBA first launched the PPP program, it was all manual data entry. There's no sense of automation. And that's funny when you think about the millions of Americans who needed immediate relief for their businesses which have typically at most 20 days of cash in the bank. Of our pool of institutions who are asking us questions about if we're going to do anything, they said, "Our initial problem is that we have tens of thousands of applications for our customers and we have three employees at the bank who can submit these loans for approval."

Garrett: And so it just became this overburden on their loan team to get these approved in a timely manner. And on top of that, the SBA was constantly changing the applications. So what was on paper was not what actually reflected on the website which meant a lot of phone calls at strange hours calling these people who are applying, these applicants for these loans saying, "Hey, we need this other additional information. How fast can you get it to me?" And so it was 30 minutes plus to get this loan from paper on the desk of a loan officer to approved by the SBA, it was very painful. So they reached out to us to see what we could do to help automate that process.

Greg: How did you guys choose Heroku for this and what value do you think it brought to the table?

Brett: Early on when we decided we were looking at the problem, it was actually a Friday afternoon when we got on the phone with a particular financial institution that was talking about this problem. And they have this backlog of applications that they needed to submit. And the word in the back room if you will, was that these PPP funds were going to run out extremely quick.

Greg: Mm-hmm (affirmative).

Brett: So we're talking to this financial institution, we understand the magnitude of the problem. And we know we basically have a weekend to get a solution up. There's a handful of us on a Friday afternoon and we're like, "Okay, well, we're working all night for the weekend and we need to get something up Monday morning so we can start processing these loans in some type of automated fashion and really help these people before the funds are gone." And so when you start looking at what you need to do to innovate rapidly, and we start choosing technology stacks, both on the software side and the hardware side, you can build a Rails app, push it to Heroku in a matter of minutes, and you can scale it the way that you need to scale it pretty quickly.

Brett: And with the whole marketplace of add-ons, it became really easy when we need to do asynchronous processing. It's just like we can add a Redis server, right? We can scale up databases and there's a lot of stuff there that ultimately when we decided, "Hey, do we need to do this from roll our own or if we could look to a platform, this platform as a service concept that Heroku has." It was really for quick iteration, quick innovation and getting something to market very quickly that is absolutely critical, seemed like a no brainer decision to me.

Garrett: Yeah. At MX, we don't typically use a cloud solution because we'll have PII data and we have to essentially make our entire production environment a PCI zone. So in order to push new features, it has to meet all these security requirements. And so we were used to looking at products, anything we're going to use outside of our ecosystem with those security eyeballs. And so that was one of the first concerns with looking for a place to push it. We're like, "Okay, we know we can probably make AWS work because there's plenty of startups who've been able to achieve that." And then honestly, we looked at doing due diligence. We saw all of the compliance that Heroku had and we said, "Is this enough of a security story to get this product live?" And we found out it was and were able to conduct audits.

Garrett: Some things like encryption at rest with Postgres was very valuable. And then on top of that, any engineer who has to work with Terraform plus AWS knows you're going to have 15+ files of boilerplate just to get a hello world across a load balancer. And for us just to say, "Hey, what's that? We're doing a Rails app. Also, we need ImageMagick and we need a Chrome Webdriver. Go."

Greg: So you guys are using Terraform plus Heroku. So you're using Terraform to manage Heroku?

Garrett: Yeah. So we're actually using Terraform to manage Heroku as we bring institutions on. It's just really easy to add another plugin on Heroku and share the code that provisions the environment. And it's only a matter of minutes to take a client from phone calls to live in production.

Brett: And I think that's important because you think of again, taking something to market quickly. And if we had to build this, there's a couple challenges that exist. One, the multi-tenant nature of the solution and having to build that in and manage that. And then we have isolation of data. And so we have one institution that doesn't want to share, mingle their data in a database with another institution. And so we were able to actually build it in a very simple way that kept everything... Each institution isolated with their own application servers and web servers as well as database servers, but using Terraform with Heroku, with Garret's work that he did. It was really amazing. A new institution wanted to come on, we added a couple configuration settings into Terraform and they were spun up and running extremely quick. And I don't think we could have done it as fast as we did and responded as quick as we did if we didn't have the solution that we chose with Terraform and Heroku and some of these other technologies.

Greg: Tell me about what Heroku products are you guys using? I'm assuming you're probably using Shield, Private Spaces, that sort of thing. Could you let me know what you're using? Is that safe to talk about?

Garrett: Yeah, so we were initially looking at Shield rolling that out, especially because you can get some added safety blankets with Shield. We ended up not needing it. The way we rolled this product out was we tried to treat it like a startup and what is the minimum path you need to get to production? So we spun up just on the standard Heroku plan and then we had a third-party security auditor come in and conduct a series of penetration tests against our instances to make sure we weren't doing anything silly. Once we got those out of spec, we presented them to our customers we're working with. And they passed it by their security and their legal team and they said, "This passes. This is exactly what we need to get up and running." That's what we went with. I think if we were doing this long-term, we would definitely evaluate something like Shield, but the nature of this product is spin it up quickly, be very effective. And then, when we're done and this COVID situation is resolving itself, shut it down.

Greg: Tell me a little bit about your open source contributions. I see here that you guys built out the Terraform to connect the SBA API directly then you open sourced it?

Garrett: Yeah. So one of the big things with this project was we didn't want to be a gateway where pay us money to get loans approved. That's not what we wanted to do at all. And so we said, "Look, the software is free and you can pay for hosting and support." And I thought that was such a brilliant way to build this, right? Because there are so many partner vendors who've reached out to us saying, "Look, we need something. We might not even deploy this code, but we need an example of how to work with the SBA's website." We said, "Great, look at our code. Please use it."

Garrett: And we've had individuals who said, "Look, Heroku, even though with something like Shield, it's just not going to pass our legal team where we still don't believe in cloud. That scares us, but we do have infrastructure. Can we please run it on our own?" We said, "Great, here's the source code." But we update it all the time. And that became a really common thread. So anyone who wanted access to our software has total free access to it. It's a private repo due to the nature of SBA documentation, but it's source that's available for anybody who wants a copy.

Brett: And I think that's important because you look at the problem and honestly before us, there were some companies, I probably can't name them, but they were going out and really taking advantage of the opportunity and saying, "Well, hey look, we're going to build something and we're going to charge you enterprise type pricing for this." In my opinion were gouging but if you look at it from a technology history, all of the wonderful technology that we get to use because people decided to open source the technology, right? And everything from the servers that it's running on, the databases that this is running, the web application stack. We used Rails and the Ruby programming language.

Brett: There's this whole beautiful history of open source software and so when we look at what we can do to help out in some way, we thought at a minimum, we should open source our solution and just try to help out with all these small businesses that need assistance. And really the banks are the primary people that need assistance with this. And just even having the technology to spin this up really quick, have them up and running in 30 minutes to an hour. If you were to look at the whole technology stack, we did a fraction of what it took. And so it's a huge thanks to the whole open source community for deciding to open source technology. And we're a huge believer in being able to do that. So it's been pretty special.

Greg: Same boat as you. Absolutely love the open source community. And I'm so thankful for all the work of all the people that do. And I love to hear people that give back to it because I mean, in my mind, that's really what open source is. It's not just using Linux and all of that. It's giving back to that community, it's giving back to the world itself and making the world a better place than you left it.

Garrett: One thing that goes along with open sources is we're often just giving our own free time to make software better and to enable companies stand on our shoulders and propel themselves forward. And that's a philosophy we adopt for this project because the first institutions that stepped forward and asked for help, we said, "Look, let's just do this together. We're not going to focus on... We'll do it for the sake of partnership and for the sake of... We are nerds and for the first time ever nerds get to be superheroes for a brief moment. So let's put on our capes and just knock this thing out." So we were working Friday night, Saturday night, Sunday night, just through the weekend long hours, trying to bust this thing out. And to the point where we're putting so many hours in that I don't care what the sales commission could ever be on a product full enterprise grade, it's just not worth it.

Garrett: What we're doing it for is those people who would need a paycheck to pay their bills and that's constantly what we're reminding ourselves of. Every meeting we said, "Look, why are we doing this? We're spending these long hours to help those who don't have right now, because this is the skill set that we as a nation need in order to lift one another up." That's the whole open source philosophy applied to this project. Why this project was free for anyone who wanted to use it? There's no product manager who can say, "Please." Worked that much time on this. We just wanted to do it for those who need it. And we wanted to make it available for those who need it.

Greg: If you had to estimate, how long do you think it took from idea to production?

Brett: Yeah. So I would say we started on a Friday afternoon, worked all weekend. The first cut of it, we put up on a Monday morning and as anyone who's written software knows you put up the first cut and then the work begins. We put that up and we started submitting these loans into the SBA system. And the first approach we actually had to take was using some robotic process automation. So we're logging into the website, we're imitating a user, we're filling out these forms, et cetera.

Greg: Mm-hmm (affirmative).

Brett: That first weekend was really critical. And then from there, it was just iterating on making sure that the software is submitting right. And then later we started working with the SBA. They heard about what we were doing, they saw the traffic that we were sending, the volume that were doing and they reached out and said, "Hey look, we were working on an API. We need to work with some partners like you that are technology-focused that can innovate rapidly, that can make changes. As we make changes, we see how fast you're able to get something up and running with this RPA style system." And so we started working with them on the API and got API access really quick. When we heard about it... You did the API integration so maybe you can talk about that.

Garrett: Yeah. Well, before we... Let's come back to that for a second because what was crazy is the people who don't really know, the PPP came in two waves. The first traunch of money ran out in about a little less than two weeks.

Greg: Mm-hmm (affirmative).

Garrett: And so you can imagine the traunch opens that week and then Friday, we're just hanging out and then it's like, "Boom, let's start building." We start building Monday, we release maybe by Wednesday or Thursday, worked very consistently submitting applications to the SBA. You can tell other people had a similar mindset to using a Chrome Webdriver to try submitting these applications. We start seeing the website more consistently fail, right? I mean, you hit these timeouts, these Oracle-

Greg: The website being the SBA's website-

Garrett: The SBA's website fail as more people started trying to write these web drivers and then the money ran out and they closed the doors in the middle of the night. And so we thought we didn't know what was going to happen next, but we took that time to just offer a little bit of the software and start asking phone calls with the SBA, with partners ask, "Hey, how many loans were you able to clear? What's left for you?" And we heard a number of them say, "Hey, we didn't even get our forms out to people until the day they closed."

Garrett: Right? I think we've had this week of we hear there's some rumble that there might be a second wave of finances coming and it's in this week that we were able to finally get on the phone with the SBA. And that's where they said, "Hey look, we want to green light you as fast as possible. We know that you've integrated with a number of financial institutions and you can help push their loans through." So they were amazing. They had some partners working with them. We met them on a Thursday. And if you know banking or you know government, a fast turnaround typically never happens.

Greg: Yeah.

Garrett: And here we have a meeting on a Thursday where we're honestly expecting, "Okay, we'll put you in the queue in three weeks. We'll send you some auto reports and we'll get you this paperwork to fill out and come back to us. That very slow turn is how these deals typically build. They said, "Okay. Unfortunately, our testing website's down, but we might be able to create users for you on our side. And later tonight, it's hopefully going to be back in working order. And tomorrow let's get on the phone again to try give you a test request to show you how to submit the loans through our API."

Greg: Wow.

Garrett: "And then we'll green light you. We'll give you production access." We're like, "Are you serious?" That's how fast they were. And because their test site was down and they had just opened up a second traunch, Congress had just approved it, we knew we had to act quickly. So we had a bunch of us just trying to reverse engineer the test site to figure out their SOAP API to get these loans submitted. And then we emailed them back in the middle of the night and said, "Hey, we think we got this figured out." And expecting maybe a little bit of like, "Hey, please don't do that." But also like, "Okay, we'll talk about it tomorrow." They just CC'd us directly with the SBA people said, "They're good for production. Thanks guys." They were amazing.

Garrett: But that gave us access to their test API and 12 hours later, people started low testing on their test environment. So things are going to be crazy when the floodgates open Monday morning. But as a result of the load testing, they shut off the whole API. So we had access, we have some documentation and then we had a weekend with no test environment. So that was pretty stressful, but we're able to code to it the best we could. And then Monday morning we thought, "Hey, we still have this webdriver that works really well. So we'll go live with that Monday."

Garrett: And then 10:00 AM Monday rolls around and the site is completely dead. I mean, everyone's hitting it as hard as they can. This is when the second traunch is finally live. Their site is completely unresponsive. And so we start switching back to our direct connect with the SBA and we're able to get maybe 1 in 1000 thousand loans through. It's just so slow. And then we found bugs and we're iterating on it, but it was an amazing experience to just work with the team and help iterate slowly and fix one bug, put up a pull request, merge it, push it up, and then hit the... The Heroku auto deploy runs out and get the next batch of errors popping up and stomp them. And then eventually the SBA said, "Hey, we see that a bunch of the traffic is coming from webdrivers and we're going to ban all webdrivers."

Garrett: But fortunately we had our direct connect from a couple of days prior, and we were able to stay live and continue processing loans. And that's when we really started kicking into gear and we were able to process thousands and thousands of loans for these institutions. And it was staying up until 5:00 AM, sleeping two hours and getting back up and going until you die, but that's what we needed to do to get it done and we did.

Greg: So what do you think the impact has been? Do you have any metrics you can share?

Brett: Yeah, so we're looking at the total volume. We look at the amount of money that's been able to get distributed out to the small businesses across the United States. And so we're approaching right at the $5 billion mark on what's been processed through our system and then that's just us directly. The open source portion of this has continued to extend out and other institutions have put this and they've rolled out their... They're doing their own version of our software and processing loans that way so we don't have a lot of insight into those numbers because they're private. But from what we've done, write up about $5 billion of loans processed and distributed out to small businesses.

Greg: That's amazing. That is really amazing. I'm sure that there were lots of jobs saved and businesses that were able to continue to stay afloat because of the hard work you guys put in.

Garrett: Yeah. And we can't really talk too much about the number of jobs but it was definitely in the 10,000 to 100,000 range of jobs. Again, that's the most satisfying thing to take away from my perspective. I'm just coding on this project and working on a Rails app. I was like, "Everyone at Hacker News says I could do that in a weekend." Well, here we are doing it in a weekend. And the impact is not just people, but it's people where it matters, right? It's in their paycheck, it's their lights in their home and it's their family food on the table. That is the most meaningful thing that I, as a software engineer, can give back to people. But it's funny to me still how just a little bit of Ruby code turns into a normal life for somebody. Maybe not totally normal in this strange time, but close enough.

Greg: Anything you would like to add before we close up?

Brett: There's one thing I've been thinking about as we've been talking and it's an idea that I originally heard from Steve Jobs and I'm going to paraphrase what he said. But it was this idea that your whole worldview changes as soon as you realize that the world was created by people that are no smarter than you are. They just went out and they decided to do something and they made it happen. And everything we see around us as a result of people who just believed in themselves and went out and made it happen. And I've been in situations where you hear stories or maybe for those that are listening to the podcast are really like, "Man, that's a great story and it's really cool how these people were able to help out or whatnot."

Brett: But really, I think as Garrett was saying early on in the podcast is that when you have the ability to create, we have the ability to as technologists altogether in this and you can write code and you can come up with ideas and you can create solutions that solve problems that exist in the world, it's really amazing. But I think a lot of it is just believing in yourself and finding when that opportunity comes, taking advantage of it.

Brett: You talk to anybody that worked on this project, I think you can hear it from Gary. It was a lot of hard work, but it's worth it and the team did an amazing job. The motivation the team had, there's just the intrinsic reward that the team had from doing all this was awesome. And so I think we can all in whatever way we can, pay it forward. And that can be a pull request on an open source project, right? It can be whatever, but just do something to give back. And that's just what was going through my head as we were thinking about this that I think everybody has the opportunity or can find the opportunity or look for an opportunity to help someone out and pay it forward.

Greg: Thank you so much for your time. Thank you for sharing this amazing story, Garrett and Brett, and thank you for what you guys have done to help out the world as a whole and our country and our community and the open source movement. We're super excited that you're able to build this on Heroku And I'm glad that we were able to have some small part in helping you get this project across the line.

Brett: Awesome. Thank you. Appreciate you having us on.