Special Episode: Scaling Businesses During a Pandemic

Greg: So this is a bit different topic this time, however, because of course the world looks different when we've got a global pandemic, climate change, all sorts of stuff going on. What prompted you to go ahead and looking at building a new app during all of this world changing events?

Ryan: So obviously, the retail sector has been hit pretty hard by this, and particularly in the UK, because we had quite a strict lockdown. It was something like a third of all retail became e-commerce in May and non-food online spend has almost doubled since January. So people are spending far more money online, and obviously they couldn't go to all the shops very easily. A lot of shops had to be shut.

Ryan: One of our biggest clients is a company called Matalan. They're a fashion retailer, and they employ something like 13,000 people. So quite a significant portion of the UK. They have something like 220ish stores that they sell with physical retail. And then they have their website as well, which is the side of things that we typically manage.

Ryan: Obviously, when all of their stores had to shut that meant a massive hit to their revenue. So we really had to scramble to help them get through COVID, and then obviously succeed on the other side of it as well. So that's kind of really where we really needed to step in.

Ryan: So one thing that they saw was that because this volume was increasing online so much, their distribution centers couldn't scope with the volume of orders anymore. Everyone had to buy online. And as people realize that they would have to be stuck indoors for a lot of the next kind of weeks and months, people wanted to kind of stock up on kind of those basic clothing items and things like that. So they needed a way to be able to continue selling as much as possible because there was that pent up demand there, but also they needed to offset all those stores that had to be shut.

Ryan: And then there was even more further concern at one point because their distribution centers, obviously they had to look at social distancing within them as well. So that was further impacting their ability to ship online orders and increasing that backlog again and again and again.

Ryan: And then the kind of final element that we had to look out for them was obviously with a retailer of that size, a lot of them in the UK don't just have enormous pots of cash sat around, kind of a massive rainy day fund. So we needed to help them raise funds as a safety net to get through and make sure that there's thousands and thousands of jobs there that we needed to protect.

Greg: And how were you able to protect those jobs? I could see a couple of ways that I'd think about it, maybe turning stores into distribution centers themselves perhaps. But how did you approach that?

Ryan: Yeah. You've hit the nail on the head right there. So what we did is we worked with them to get access to that RFID stock data. So it was actually a few years previous to the whole pandemic situation. Our CEO had actually, he sat on the board at Matalan and had initiated a project to install RFID tags into all of their products, rather than just having the traditional barcodes. And the idea behind that being that you have far greater stock file accuracy. So you can walk around the stores with a magic wand effectively, and it'll pick up all the items that you have there and be able to do that stock report really easily and really accurately.

Ryan: So the great thing is that that had already been in place. We just needed access to it. So that's where we kind of kicked things off and said, okay, well, if we know what stock is in each of these stores, obviously they don't have customers going in at the moment. So we can accurately calculate whether an online order could actually be fulfilled by one of those stores. So that's kind of how we kicked off the project is literally just doing like a proof of concept algorithm. We effectively double routed every online order for a period of time through both the traditional mechanism, which just kind of sent them off to the distribution centers, and into this algorithm that would detect whether it could've gone to a store or not. So we kind of simulated what would happen with real production data and with obviously causing no disruption to the actual, the real order flow.

Ryan: And then off the back of that, we could quickly see that we could offload masses of orders into the stores. So that's when things ramped up, and we had to make that reality really.

Greg: That's really cool. That's a fantastic idea. And you could probably then start to bring back some of those employees that had been furloughed from the store, so they could fulfill orders and ship them.

Ryan: Yeah, absolutely. In the early days we, we trialed it with, I think it was a set of 10 stores initially. It was very eerie going into the stores with no customers there and helping their staff. And we went in to look at how optimal can we get the flow of going around with a trolley and picking the items. Is it better to pick one order at a time? Or was it better to print off, say 10, 20 orders, and then walk around with your little trolley and go and pick the items off the shelves just as if you were a real customer, really.

Ryan: So we looked at that whole process and how they pack the orders, and then how they get collected and all that kind of stuff, just to see how streamlined we can get this and see it as more than just reacting to the pandemic, but actually a longterm solution to de-risking the business going forwards. Because you've always got the ability to scale up and down this. For whatever reason, we went into a second wave, and the stores got shut down again, this protects the business. It gives them another opportunity to keep fulfilling orders. So if the employees don't have customers to deal with anymore, because it's all shut down, that would obviously give them far more capacity to ship online orders. So they can finetune this as they go along, really.

Ryan: Since then, they've been able to roll this out as click and collect as well. Before they have this process where click and collect orders would actually go from a distribution center, be shipped to that store where the customer would pick them up. So because the stores now have the technology that we'd built, they could actually pick and pack those click and collect orders directly in the store because we know whether they have the items already there. It was a big waste of money, effectively, picking and packing them in a distribution center only to send them to a store where the store might already have the items. We're just moving around stock for no reason. And obviously there's costs and delays involved in that. You're having to pay couriers and having to wait for it all to get collected. Whereas this way, it's just a case of does that store already have that stock? Go and collect the items from there. So it actually opens the opportunity to come out of the other side of the pandemic far stronger than the side that they went into it.

Greg: So how long did it take from idea to trial to going live with this?

Ryan: So we built the initial algorithm in about a few days. It was a week at most. I mean, most of the time was actually tracking down where we could get this data from and getting it out of those systems. Once we'd got access to the data, it was relatively straightforward for us to implement these algorithms. We've built a platform that's completely API driven. It's got a whole bunch of things like webhooks in there. So it's very easy for us to say, okay, when an order comes in, actually routes it via this additional mechanism, which could analyze it and see whether it was applicable for the stores.

Ryan: So because we'd invested in that architecture and that flexible way of working, it meant that it was actually very easy for us to bolt on this service. So, yeah, it was about a week to run the proof of concepts. And then the whole process of making this an actual reality, actually allowing the stores to act as these miniature warehouses and getting the employees back and getting things like printers in there for labels and hooking up different couriers to come and collect the products and all that kind of stuff, that took about four to six weeks, if I remember rightly. So, it was fairly rapid.

Greg: Yeah, I'd say so. That's pretty impressive, being able to deliver this new functionality and change how a business does business in four or five, six weeks is pretty incredible. What sort of tools did you use to achieve this?

Ryan: I don't want to get into a sales pitch, but we relied on Heroku quite heavily. So even the proof of concepts, it was booted up as a Heroku app. Obviously, with having such short timeframes to work to we wouldn't be able to deliver fully kind of production ready application within a week if we had to do all the kind of DevOps in it, all that stuff behind the scenes. So the initial proof of concept was using Heroku Postgres. We use Redis for kind of analyzing things. We've since evolved that to use things like the Heroku Scheduler for polling, for stock files. We also use Kafka. So we attach that. We actually have it already as part of our ecosystem for our applications, for doing various bits of integration and kind of data analysis. So we could just bolt that straight into the new application, and it gets that access to that full stream of events and is able to contribute back into it. So as it evolved into this production application, we were able to just kind of attach the existing parts of our ecosystem very easily and focus on the actual building of the application rather than the. plumbing behind the scenes, which was great.

Greg: So yeah, that's kind of really how we kick things off really. Just dive straight into coding at that point, because it was the proof of concepts. We weren't really overly concerned about things like performance or scaling and things like that. We just needed to see it'd operate. Now we've optimized it to a point where, even though we're still using things like Ruby, we're not using anything like fancy particularly performant language or anything like that, and not doing any special optimizations. The API responds in something like 20 milliseconds. And a routing job takes less than a hundred milliseconds. So we can decide within a 100 milliseconds of receiving an order, whether a store is able to process that order or not out of hundreds and hundreds of stores. And that's with minimal performance optimization.

Ryan: So we've really taken this application from proof of concept through without having to do some major rewrite or anything like that. And Heroku's just been there to help us scale it as we go through really.

Greg: That's amazing. And as a Rubyist, an old school Rubyist, that makes my heart fond and happy. So I've seen lots and lots of customers, and I've advised lots and lots of customers about using Kafak as sort of a bus between microservices. Is that how you're using it in this use case?

Ryan: Yeah, I'd say so. So one of the main things is that once an order's been routed, that needs to make it into our order management system, which is obviously where it can get processed, people can ship the items, etc. And that process goes through Kafka. We wanted something that was incredibly robust, designed from the ground up to be rock solid. And Kafka is the perfect solution for that. It's a little more tricky than just running the likes of Postgres. So that's particularly why we'd like the Heroku solution where it's pretty much all managed for you. We've gotten nothing but good things to say about the service there. It's been phenomenal at the time. I don't think we've had any outages, any performance issues there at all. It's just kind of set and forget really, which is fantastic. So yeah, it allows us to tie together the whole variety of different services behind the scenes. So you get that benefit.

Ryan: And the great thing is that obviously if something goes down, or if something has an error, it just stops processing. And then you can address that. And the service can pick up on all the chronological messages from then on. So you get that kind of automated healing process, which has been useful a few times. We've had times when some bizarre data has made it through, and then an exception has got raised. The great thing is that the developers can jump on that. They can see the error. Nothing's getting processed in the meantime. Either then we can look, do we need to skip this message? Or do we need to address it? How do we need to respond to that? Fix it and carry on.

Ryan: So, yeah, it's the perfect use case for e-commerce, I think, where you've got these kind of disparate systems and different steps in the workflow of someone going from placing an order through to receiving it, and even beyond when it comes to things like returns and refunds and all that kind of stuff. So yeah, we rely on it quite heavily. Everything from triggering webhooks. We use it for things like keeping search indexes up to date, all sorts of stuff.

Greg: Can you talk about how you guys approach your security and trust stance, and how that works on the Heroku platform?

Ryan: Sure. One of the things from our perspective running an e-comm platform is that we have to care about things like PCI. Thankfully Heroku checks a lot of the boxes for you. So there's a lot of things that you don't really have to worry about. That could be a whole talk in itself is running PCI applications on Heroku, because there's so much that, there's a feature for this. So it's just like, when you're doing your audits and things like that, it makes it very easy for you. So that's things like the Private Spaces and things like that. You can lock things down with firewalls and segregate things off. There's all the things around logging, so you've got easy access to that stuff.

Ryan: It's very straightforward to be secure on Heroku. It's almost like you've got to go out of your way to make it less secure. So it's one of those things that it's really hard to get right. There's the likes of yourselves, and we deal with CDN partners, people like Fastly. Both of your companies have got massive teams of people who deal with security on a day in, day out basis. In my business, we can invest in that, but we're not going to have the scale of teams that you have. So it makes far more sense to outsource what you can to a partner who truly understands this stuff and can really innovate in that space even. It's one thing to just kind of clone a bunch of features that are necessary to check some boxes on PCI and make sure that you're safeguarding data and whatnot, but then there's a whole other industry in responding to upcoming threats and things like that. So having entire teams of people who are dedicated to working on those features and working on upcoming threats and how you address different things, that makes me be able to sleep at night.

Ryan: So obviously we are dealing with sensitive data here. It's critical that, you're investing in that area. At the smaller end of the scale, for anyone who's considering doing it themselves, it's a no brainer just to pay a little bit of money and get a partner to offload as much as you can onto. But beyond that, because this is a fairly, going back to the whole kind of older routine, COVID side of things, because this is a fairly straightforward app, we could live very easily within the Heroku ecosystem. It's absorbing a few web hooks and running some algorithms over some data. So the security aspects on this part of the application was quite straightforward.

Greg: Anything that you've taken away from this experience, or any advice to anybody who is in a situation like you were in, where they have an idea and they need to iterate on it quickly? How would you go about redoing this if you had to do it again?

Ryan: I think, in principle, before you even get to problems like this, the word agile is kind of thrown around. I think agility might be a better word to use here, but being ready to respond very quickly to changes in the market. It might not be the next pandemic. It might be one of your competitors suddenly has gone bust, and you need to capitalize on that. Or it might be that there's a new entrance to your industry, and some phenomenal new businesses come up and suddenly half the customers are looking at walking away. That kind of mentality of something can always happen, how do we ensure that we can respond to it as quickly as possible? You don't necessarily need to start making all the plans now, but you need to have the capability to respond quickly. So the more educated flexibility that you can build into any applications you're building and things like that, thinking about how you can extend them in future is the foundation of delivering on that agility, I think, as we start to get a bit more technical.

Greg: Yeah. One of the things that I like to talk about, and I like to think about, is a combination of simplification and abstraction. If you can abstract hard problems into simple solutions, and then you can interconnect those simple solutions in simple ways, you can keep, in your head, the whole system you're building. And that can give you some about agility sometimes to go ahead and rethink how can we use what we have and reconfigure it in a new way to respond to a different problem.

Ryan: Yeah, absolutely. I think having a collection of services that you're very used to using, you know how you can tie them together, you know that they are extensible, that's a great foundation of a business. Abstraction is one of those things that you have to be careful with it because you end up just adding more and more layers. If you try and get too flexible and too generic with a solution, that's where you get indirection and the sprawling massive layers. But as you say, that's why that simplification part is critical. You really need to be balancing how much can we abstract versus what adds more complexity. So the overall solution is that it's that systems thinking of what does the whole system look like together? What's my whole architecture like together? Not just, oh, what does this one application do? How can we build a service that can fit in here, do what it needs to do, be part of an ecosystem, and then also be extended and worked upon and built upon in the future?

Greg: Exactly. I think one of the things that I've seen from all of the people I talked to is the folks that are the most successful tend to build platforms, not products. And then out of those platforms, their products kind of evolve. And it sounds like that's what you've done. So you have that platform, and you were able to quickly build a new product or solution on top of that platform.

Ryan: Yeah, absolutely. I think businesses should always see themselves as miniature platforms. It's like, you need your own platform. It's not as a single product. But take a retailer, for example. You need to be thinking about, okay, how can I expose my inventory levels? How can I expose my product catalog? How can I get product imagery? Access to all these different things so that I can do things like go into different marketplaces. During this lockdown, Amazon's absolutely smashed out the park because a lot of retailers had to suddenly migrate their products into their warehouses. There's a lot of bricks and mortar retailers who literally didn't have a website or a mechanism to start selling products. And they suddenly had to scramble to be able to put those online.

Ryan: If those data sources were readily available, if you had all your product imagery, your catalog, any price books, inventory, all that kind of stuff, maybe even to the extent of customer data, maybe more of ingesting that rather than exporting it. But all those kinds of different facets of your business, if you've got access to those as a platform, and it's a case of any customer facing interaction you might have is using your business as that platform, it doesn't need to be necessarily technology. It can be more kind of real than that, but effectively, if that's the way that you see your business, you can respond to any changes going forward.

Greg: I totally agree with that sentiment that having that platform thinking is incredibly important to being able to respond to changes quickly and be successful. You can build a very brittle thing that will work in one use case, but then the world changes. We have a COVID, or we have climate change. All of that can destroy those brittle infrastructures that people build. But if they build more of a flexible platform, they can then pivot quickly, and then they can respond, and they can survive.

Ryan: Yeah, absolutely. I think most businesses out there should be doing a review of these kinds of systems. What's an absolute nightmare when you come to change it? What needs to go through change management boards and extensive QA processes? What hasn't been automated well enough? What has a kind of flaky test suite that kind of half covers the application and doesn't really give us the confidence to roll it out without everyone sat there biting their teeth? That kind of thing. You should be reviewing right now and say, okay, well, if the world changes, could we foresee this application or this service being something that could hold us back? If it's like four months to test something to get it out into production, then you need to be thinking, well, is that going to be quick enough for us to respond?

Ryan: So I think that's a great place to start for any business right now. If you're already established, thinking if tomorrow we had a whole different situation, like a pandemic, what would be holding you back. But hopefully COVID has raised some of these things to the forefront for you. Well, hopefully your business doesn't have any at all, which would be lovely. And you were able to pivot very, very quickly. But if you have had those, hopefully some of them have been brought to the forefront. And I guess it's just a case of checking, do you have any of those behind the scenes? What does the whole team get stressed about and complain about whenever they have to change it? That's the targets for me.

Greg: Thank you so much, Ryan. I really enjoyed talking with you on this episode of Code[ish]. Maybe we will have to go back and do yet another episode with you, talking about the security and PCI stance that you guys have and how Heroku makes that easier.

Ryan: Yeah, absolutely. My pleasure.