7. Application Performance and Building SaaS on PaaS

Chris: Right. I was just saying, or we were just talking about, I was just on a plane last night for a few hours and got on the airplane wifi, which of course is high latency and low bandwidth. And, I was waiting for websites to load. I will not name the sites, but certain big brand websites loading, and I see all these different domains that that assets, or different things, are being loaded from. And, we as developers tend to be spoiled because we have fast internet connections, generally. I have, for instance, gigabit fiber at home just because I like it and I want fast internet. I don't need a thousand-

Ryan Townsend: If you can get it, why not?

Chris: ... megabits. Yeah, exactly. And, I pay extra for that. But, it's great. And, I'm a nerd. But, anyway. But, it's easy for me to ignore performance when I'm building front ends. So, anyway, it kind of hit home last night on the plane when things were taking forever to load. And then, you go to a super, super simple website, like, I don't know. Just go to example.com, for instance. It's just the canonical blank website, or basic website, and it's just text, and it's great. Just HTML.

Ryan Townsend: I think performance is one of those interesting things because a lot of people think it's a problem for emerging markets where you don't have the latest iPhones, and you've not got 4G, all this kind of stuff. But, there's plenty of circumstances where you're out in the sticks on a train or something, or you're up in the air and on wifi. And, everyone experiences this. It could just be your cell provider saying, hey, you've got 4G. I've experienced this up in Redwood. It was saying, oh good 4G signal. But, the bandwidth on it was terrible. There was really bad latency. Everything was a nightmare to load. So, everyone experiences this and there's a lot of revenue to be made. If you've got a company that makes money online, a lot of my clients are in e-commerce, there's a lot of money to be made by optimizing for those circumstances, and kind of reducing the amount of bandwidth, reducing the amount of requests, and all these are the factors that make websites slow.

Chris: Yeah. Makes sense. And, I forgot to allow you to introduce yourself. You just said you do e-commerce, or Shift Commerce, does e-commerce. So, give us a quick intro of who you are and what you do and where you work.

Ryan Townsend: Sure. So, I'm the CTO of Shift Commerce. We're a company based out of Leeds in North England, and a lot of people have heard of the Leeds United Football Club. And, what we do is we build an e-commerce platform. It's designed for kind of ambitious retailers who want to really grow and had hit limitations of other platforms. They don't want to maintain their own thing, have a copy of a well known PHP project running, or they've kind of hit the limits of a lot of the hosted solutions. And, we want to provide a lot more flexibility and a lot more scalability to grow as a business on our platform. So, that's kind of the target market for us, is kind of established businesses that are looking to grow up to enterprise scale, or people that are just outright ambitious. They're backed by someone who's putting a lot of cash in, and they just want to really scale very, very quickly, and they don't want to have to replatform in the middle of that process.

Chris: Yeah, makes sense. They don't want to have to replatform. They don't want to have to worry about that layer and below.

Ryan Townsend: Exactly. Yeah. Where they provide value is in the products that they sell, in the customer service that they provide. And it's not in building out technology to just to support the normal kind of e-commerce stuff. So, we just take away all of that headache and we provide an API based platform so people can kind of integrate with it in a whole variety of ways. It's all hosted on Heroku so [crosstalk 00:05:25].

Chris: Yeah, that makes sense. Let's talk about that. It's interesting because you're building and providing a SaaS, right? Software as a service, on top of Heroku, which is a PaaS, platform as a service. And so, Heroku thinks about lots of multi-tenant things for our customers. There are a lot of things we think about like security and noisy neighbor and performance and things like that. But, how does Shift Commerce, or how have you thought about providing a multi-tenant SaaS with the tools that Heroku provides?

Ryan Townsend: Sure. So, there's kind of two layers that we've built. As I mentioned, we're an API based platform. So, effectively each client has their own frontend code and is deployed separate to that. So, we've kind of wrapped up your API in the background, allowing people to just throw a zip file or code at us, the latest version of-

Chris: The Heroku platform API?

Ryan Townsend: Yes.

Chris: So, you have proxy API that's a subset and secure functionality, or secure layer between your customer and the Heroku API?

Ryan Townsend: Yes. So, what we do is a client obviously has an account on our platform, and they can load it with all their products and everything like that. And, we manage the APIs that distribute that data and allow people to do things like add things to cart, remove from cart, check out, all the kind of usual e-comm stuff. And, that's all one kind of shared application. And, we have a lot of safeguards in place for things like performance and noisy neighbors you mentioned Heroku have to deal with. We have the same situation so we have loads of safeguards around that. Things like rate limiting, all that kind of stuff. We've got to care about the security of that data. So, we've done some things with Postgres to help kind of segregate that data, and make sure that obviously it's not going to cross pollinate.

Ryan Townsend: But, ultimately one of the best kind of decisions that we appear to have made is separating out that for an end. So, because it's just a platform that we kind of keep as one application, people can upload their own frontend code. That gets deployed as its own Heroku application.

Chris: Oh, okay.

Ryan Townsend: So, that can be independently scaled. Obviously, a portion like if someone gets a ton of traffic through, that's going to put a ton of traffic through to our API as well. But, the frontend bears the brunt of that and it can be scaled completely independently of other clients of our platform, and we avoid that noisy neighbor situation because Heroku have already taken care of that for us.

Chris: If I were a customer, can I upload anything from a PHP frontend to my static react frontend or something like that.

Ryan Townsend: Anything that Heroku supports.

Chris: Okay.

Ryan Townsend: So, it's pretty much that you guys support pretty much everything but .net, right?

Chris: Right. Totally. Anything that can run on a Linux box.

Ryan Townsend: Exactly. Yeah. So, we can support any of those and it's just throws a zip file or integrate with a GitHub repo, and then you can do all the review apps, and all that kind of stuff that you guys offer. So, we have that full workflow built inside our platform in terms of deploying a prerelease environment, and then you can push that into staging. And then, when you're happy with it, you've QA-ed. We set up a temporary domain for all of these environments and secure them down so that way people can't just access them from the public. And then, when you're happy with it, you promote it to production, and it's just effectively that kind of view layer in front of your data. So, that way when you're testing in those prerelease environments, you can test with the real data.

Ryan Townsend: So, it's not like the traditional setup where your prerelease environment, whether you call it a staging or UAT, or whatever it may be, gets completely out of sync with production. And you're like, oh, we tested all of this, but the data was very different.

Chris: Yeah. Old data or, yeah.

Ryan Townsend: Yeah, you get into production, and suddenly it's all crashing. We wanted to try and prevent that so that way you're reading from your production data set, and you can create orders, but they're just flagged as test orders and things like that. But, you can use that single data source to actually test new frontends because they're effectively stateless. It's just an application that is reading from the API.

Chris: That sounds great.

Ryan Townsend: It doesn't have a database or anything.

Chris: It seems like, I mean, this is you are specifically serving e-commerce customers, but it seems like a lot of the stuff that you've created could be extended, maybe changed slightly, but it could be extended to non-e-commerce or other industries. Yeah, in some ways, it's not just a SaaS you've created, but it's almost like you've created a kind of e-commerce specific platform on top of a more generic platform, which is Heroku.

Ryan Townsend: Yeah, absolutely. I think it can be applied to anyone who's building any form of custom frontend. And, that could be an actual website, something public, or it could be an application that's just internal. If you're building a web application, if you want to provide that generic service, where you add value because you've got the network effect of building a feature once, then every client gets it. But, at the same time, you want to add that degree of freedom and flexibility and having the kind of dual layer approach of you have this generic platform, you have one copy of it, you're hosting it once, got one thing to care about, and then you have individual little front end applications off the side of it. That provides that kind of ultimate freedom for clients to do whatever they want.

Ryan Townsend: The great thing that we can offer because they are so isolated, is that clients can actually, if there's a limitation with our platform, we don't ever want someone to avoid choosing us because they're like, oh, we love everything you do, but there's this one thing that you don't do that we need. So, in that circumstance, we can just say, hey, we'll help find you another partner. Or, if you've already got one, and you can just integrate that into your frontend. So, those are just API calls going out to another service. Or, it could be a third party JavaScript thing, or whatever it may be. You have full freedom to do whatever you want and we know that you're not going to have security issues because it's not our platform executing some sort of third-party code or anything like that. It's isolated to that single frontend app, and only that customer can see that stuff. So, that means that people can go far beyond us as well. If they accelerate faster than our roadmap in one area, they can swap out one of our APIs for another API and keep going.

Chris: Would an example of one of those be an email service or image manipulation service or-

Ryan Townsend: Absolutely, yeah.

Chris: Yeah. Sending text messages or something like that. Is that-

Ryan Townsend: Absolutely. Yeah. We don't have any native function in our platform to send text messages. So, if you wanted to do that, there'd be numerous ways that you can do that. You could do that the integration layer. So, if it's we have lots of webhooks that get spat out of our platforms, you'd could hook into that.

Chris: You could hook into Twilio really easily or something like that-

Ryan Townsend: Exactly.

Chris: ... if you want to.

Ryan Townsend: But yeah, so you could do that either in the integration layer, or you could do that and your frontend. So, if it's just a case of you click a button and it sends a text message for whatever reason, then you could have that.

Chris: Order updates or something.

Ryan Townsend: Exactly.

Chris: I want to know when a flash sale starts or something like that.

Ryan Townsend: Yeah.

Chris: Actually, so speaking of flash sale, my perception, let me know if this is wrong, but my perception of retail e-commerce is that the traffic is very spiky. It's normal for the traffic to be very spiky. Have you guys had to do anything different, or think about that? Or, do you just kind of like, Heroku absorbs that, and I just scale up when we need to? Or, how has that impacted your architecture?

Ryan Townsend: So, the challenge that we've had with that is the database layer. We can throw so many application dynos at it. Heroku will just keep scaling as far as we need it to. We just keep paying a little bit more money. And, the great thing is that you're really paying for when you're using it, right? So, that means that we can keep our costs really low. Black Friday comes around and everyone's hammering the web, and we don't really see any issue there, which is great. In fact, in the UK, someone had done ... it was an independent company that produced a dashboard of all the top retailers. And, some of our clients were on there, and they even had the likes of Apple on there, Amazon on there. It wasn't just purely UK retailers. It was global businesses that had a UK arm as well. So, they were pinging all of these sites and checking the performance of them all. The clients that we had on there were the top performing sites through Black Friday. They had zero downtime. You can see all these other massive retailers just toppling like dominoes.

Chris: Yeah, totally. Spiky traffic in general is a hard problem to handle for developers, I feel like. Because it's you're either wasting resources if you're just keeping this huge slack pool, or this huge capacity around all the time, or your customers are getting 500 errors or something like that if you don't have enough capacity, or rate limited errors or whatever it is, if you don't have enough capacity. So yeah, that's interesting. I mean, I don't know how to deal with that very well, or architect for that very well, but that's pretty cool. Your passion for performance, it's clearly rubbed off in Shift Commerce also.

Ryan Townsend: Yeah, it's one of those things that is kind of easy as a nerdy guy to get into because it's very technical. But then, as a kind of business owner, there's the kind of empathetic or sympathetic side where it's the effect that you have on that performance actually affects how people interact at the end of the day with you all your products. And, if things are nice and fast, everyone's happy. Obviously, when things get slow or stop working altogether, people get angry. So, you can connect to that human emotion as well. It's not just a pure numbers game where you're like, oh, we got 10 milliseconds faster last week. Great.

Chris: Great. Totally. Yeah. Last night, I had lots of anger and lots of, Chris, it's okay, just take a deep breath. That's how the Internet is on planes these days. It even affected my head space and my ability to focus because it was so slow sometimes, I had to open up two tabs so that I had two tasks running in parallel, effectively. I was doing one thing on a hotel booking or travel booking website, and then doing something else on Google Docs or something like that. And, I would just switch back and forth between them, and it was just kind of fatiguing after awhile.

Ryan Townsend: If you have to switch context, you're going to lose your train of thought and your focus.

Chris: Yeah, it just left me frustrated. Let's see. We've discussed scaling and performance a little bit. What about now reliability, durability, having B2B customers, or having businesses that are your customers, they are losing money if their site goes down or their shopping cart service goes down or their inventory service goes down. What are the kind of the architectures or the things you think about when building in a way that kind of requires high availability?

Ryan Townsend: So, this is a really important thing for me, not just for my clients, but it's--I don't want to be woken up in the middle of the night.

Chris: Right. Yeah.

Ryan Townsend: So, yeah. So, I mean, the main thing for us is that we use kind of very mature technology. We're not kind of archaic, but we're not throwing around bleeding edge technologies because it's interesting and fun and everyone wants to use the latest, greatest thing. Just evaluating things carefully. Obviously, this going to be the best tool for the job.

Chris: Yeah. What's an example? So, do you use GraphQL? Is that the new hotness still, or is that more mature now, would you say?

Ryan Townsend: I'd say that's coming into maturity. There's a few big companies who've implemented it. So, that's something that we're now considering and looking forward to using hopefully. But, we're not using that today. We could have seen the original announcement and been like, yeah, let's switch everything over.

Chris: Switch everything from REST, JSON based HTTP to GraphQL.

Ryan Townsend: Yeah, exactly.

Chris: But, you didn't jump on that bandwagon right away.

Ryan Townsend: No. But similarly, we're not using SOAP or XML or anything like that. We're not kind of using really old school stuff. You've got to be kind of practical with it. It's kind of like using mature technology on top of somewhat more boring architecture, more straightforward system architecture. Because if you start using kind of crazy things then that's where it all starts to fall apart. You know? And, the nice thing is that we can, because we've built on top of a platform-as-a-service unit, Heroku, we can kind of abstract that from ourselves so we don't have to worry about the underlying ... whether it's things like Docker and Kubernetes and all the other kinds of things. We don't have to worry about any of that. So, when the latest, greatest thing comes along, it's not even a temptation to say, oh we should go and use that. And, it's more of a kind of application side of things that we focus on. Ultimately, relying on a service that has, regardless of what platform-as-a-service you choose, the idea is that they're going to be far brighter and have far more resource in that specific area around managing infrastructure then you'd have internally.

Chris: Right. Yeah. They've specialized in providing this service, whether it's Heroku, Postgres or a runtime for thousands of customers.

Ryan Townsend: Exactly.

Chris: Whereas, hopefully you can hire great people and you do have great people, but they may not have the same experience that Heroku may in their runtime, managing their runtime, or their databases.

Ryan Townsend: Yeah, exactly. And, it happens up and down the stack. We're relying on you guys for your expertise. And then, the idea is that, if you have some sort of problem, we know that you guys are jumping on it and there's potentially a wider impact. So, you're going to throw loads of resource at it. More resource than we would have to deal with those infrastructure problems. If we had an internal team of a handful of people, it's a fraction of what Heroku could throw at the infrastructure. So, it's the same thing for our clients. They are relying on us, if we have a problem that's widespread, they know that we're going to throw our whole team at solving that problem and fixing it so we'll get it done quicker and more efficiently than if they had an internal team that will be a fraction of the size, potentially less skilled and so on. So, you're kind of relying on the kind of third party being the expert in the area, and they're focus on that means that ultimately, they're just going to have more resource. So, it makes complete sentence.

Chris: Yeah. So, for multi-tenant, at the platform layer and below, there's durability and kind of "high availability" because I know that has a real definition that you have to think about in a multi-tenant environment. Well, let's talk about monitoring, for instance. How do you think about, or handle, monitoring so that when customers are growing or scaling or going up and down, what do you do for monitoring, or how do you manage monitoring? And then, how do you use that also to think about durability and reliability?

Ryan Townsend: We have a load of logs coming out, things like New Relic, the loaded monitoring on that front, and effectively, it's just a case of looking at the overall load on the platform, knowing how far you can go. The wonderful thing is these days a lot of things are autoscaled, so that's kind of somewhat taken care of for you. But, there are hard limits in terms of things like databases because you have a finite number of connections so you can only scale your service so far before your database is going to be restricted there, or the IO within that database.

Ryan Townsend: So, you can add more and more replicas, but you can't just infinitely scale that without thinking about it very carefully. It's not something you can do on the fly.

Chris: Yeah, So, do you know what the indicators are, that are used for databases? Is it IO, or-

Ryan Townsend: Yeah, it's primarily IO, but then we monitor at other levels as well in terms of the volumes that each client's putting through and things like that. So, we have rate limiting in place and things like that. But, that's more of catching the problem once it's already kind of ... for that given client, they might be-

Chris: Catching the problem right as it's happening as opposed to planning for it ahead of time.

Ryan Townsend: Exactly. Yeah. What we look for is spikes across the board on top of that as well. So, if we get a kind of perfect storm scenario where all the clients are getting hammered, then we need to be aware of that. So, we look at overall volumes as well as per client.

Chris: Yep. What about on compute? For your software, running on a CPU, what sort of indicators do you use to understand load and understand what is the capacity? What's the high water line, I guess, for your software and when do you start being like, okay, do we need to scale this even more horizontally?

Ryan Townsend: Compute is somewhere we can autoscale. So, effectively, we are monitoring the CPU usage within each of those nodes or dynos. And, we effectively scale up when we need to on that. We've designed our platform to be able to do 10 times what the total of our clients' revenue could be. And, that's taking measurements out of Black Friday, peak hour, so it's a pretty big number. So, we know that the high end of our ultimate scalability is way more than any client could ever achieve even if all of them coalesced together.

Chris: Isn't there some stat, I don't know if it was from Google or Amazon, that was every second our site is down, we're losing like $10 million or something like that? I think it was Amazon. Do you remember that or did you ever hear that?

Ryan Townsend: That must be an enormous volume of cash for Amazon, for sure.

Chris: Maybe there were two different ones. There was one that was every second our site is down, we're losing X number of dollars, but then there was also a latency number. So, maybe this is getting back to performance or things, but there's like every hundred milliseconds you lose 10 percent of your users or something like that. Do you remember what the number or a statistic was there?

Ryan Townsend: I believe the figure was for every 100 milliseconds of latency on a webpage, they saw conversion drop by one percent of their overall conversion rates. So, if their conversion rate was five percent, it wasn't going down to four, it was one percent off that five. Ultimately, that's a huge volume. I mean, I think that case study's 10 years old now, but there's far more that have come out since. There's a fantastic website called wpostats.com, or Web Performance Optimization Stats.com, if you count the algorithm. Algorithm?

Chris: Acronym?

Ryan Townsend: Yeah, acronym. Yes. So yeah, on this website you can see loads of case studies on this, on the facts of latency and things like that affecting conversion, bounce rate, all these different metrics.

Chris: Oh, interesting.

Ryan Townsend: So, there's a big one by Google recently, probably the past two, three years, where they were doing some analysis through the DoubleClick ad platform. And, they found that if a mobile website takes more than three seconds to load, over 50 percent of mobile users would bounce away. So, a lot of websites do take longer than that. I did some analysis in the UK recently of different retailers and, just on a normal day, this wasn't under stress. It wasn't Black Friday or anything like that. And, I think the figure came out as something like 40 percent of these massive retailers, their websites, just on a daily basis, took longer than three seconds or so.

Chris: Oh, wow.

Ryan Townsend: So, we're talking-

Chris: On a mobile device?

Ryan Townsend: Yeah. On a mobile device on. I was looking at, I think it was 3G or 4G. So, we're talking they're throwing away so much revenue. It's a big hidden problem. And, a lot the problem comes because there'll be tracking things like bounce rate through their analytic suites. But, if you can't get to that analytics suite in the first place, that needs to trigger and record the fact that someone was there before they bounced away. If you're missing that data because the analytic script never loaded, then you don't know how high that bounce rate is.

Chris: Right. Your number is going to be artificially deflated because they're not going to even be counted because they didn't even show up.

Ryan Townsend: So, I think web performance is a far bigger problem than people may count. And, it's a combination of frontend concerns and server side as well. Obviously, nothing can be loaded on a webpage until you get the HTML for it. Right?

Chris: Right.

Ryan Townsend: So, there's a lot of optimization that can happen after that. In fact, something like 90 percent of the load is after that HTML response. So, there's a lot of low hanging fruit there. But, if you do have a slow website, in terms of the server side and generating the HTML, then that has a huge impact because nothing can load until that's happened.

Chris: That's interesting. It's like when a page begins to load, or when the first little bit of something appears on a page, if it takes, just anecdotally, if it takes a long time for the rest of it to show up, I get frustrated. But, it seems it's almost a different frustration than if it's spinning and nothing has showed up because the server is still doing something and hasn't responded with the full HTML yet. That's a different frustration for me for some reason. I don't know why. It's like you've tempted me with a little bit of text or a little bit of HTML or something has shown up, but now you're not giving me any more. So, I'm waiting for it. But, when nothing has shown up, I'm kind of like, all right, it's just spinning. I'll wait a little bit longer. I don't know if that is a normal reaction or just me because I'm thinking about, oh is that on the server or is that on the browser or is that because it's going out to Google Analytics and Segment and all these other services to do tracking.

Chris: Yeah. But, it seems like the worlds of kind of performance and durability and reliability all kind of overlap somewhere. In the middle of where performance doesn't matter if your servers can't handle the spike or the peak of traffic that's coming from some flash sale or Black Friday or whatever it is.

Ryan Townsend: Absolutely. Yeah. I mean, they were all related because it ultimately comes back down to your customer experience. That's-

Chris: That makes sense.

Ryan Townsend: ... the nice thing about these kinds of metrics and things that we have to deal with is that it, whilst they're nerdy and you're talking about milliseconds here and optimizing this there and whatever it may be, ultimately all of that stuff has an impact on the customers at the end of the day. So, that's what you can bring it back into the real world.

Chris: Right. Yeah, that's a good way to think about it. Sometimes it's easy, like you said, to get too focused on all these little metrics or the names for different ways of measuring performance or durability or reliability. But, really the thing that matters, the only thing that matters at the end, is user experience or customer experience. That is if you are making a service where the the human or the customer is the end point, or the end user of it, and not a robot, I guess.

Ryan Townsend: There's a talk at the conference that I just spoke out by a guy called Nathan Bower of the Zillow group. And, he was showing how, in their company, they've mapped performance metrics and kind of the response time. Two different kinds of feelings. So, rather than it just being like, the median customer gets, the response time is X milliseconds and things like that, they've changed it to, I think it's happy and frustrated, or delighted, I can't remember the average one, and then angry.

Chris: Delighted, content, and angry, or something like that.

Ryan Townsend: Yeah. Exactly. So, that way you're bringing it back to that human level, and everyone buys into that like, oh, what do you mean, we're making five percent of our customers angry? Because it's all about that long tail. You can have a great median response time or uptime or whatever it may be that you're monitoring. But, for the long tail of customers, that's what you got to think about, is what are they experiencing? So, if they're all angry, then it's not great.

Chris: Right. That almost seems like a more inspiring and motivating thing for me, say I was a developer building this software for the customer. If you tell me that it takes 1500 milliseconds to load or 3,000 or whatever it is, my response is, well, what does that mean? Is that good? Is it bad? Relative to what? How do I measure that? But, if you say to the developer, oh, we have data that says that 30 percent of our users are frustrated or angry after they use our website, it puts you into action. It still would be useful to have numbers as goals to aim for it to be like, okay, I know they are frustrated when they get a 3,000 milliseconds response time or something like that.

Ryan Townsend: Yeah, it's got to be mapped back to something that you can actually action.

Chris: Right. So, then I can, yeah exactly, figure out the work I need to do to get it down to less than 1,000 or something like that. Whatever my goal is.

Ryan Townsend: Yeah. But, this relates to the whole kind of perception of time by human beings. We're not very good at this at all. This is why, again, there was a talk at the conference by two women, can't remember their last names, but they were Heather and Gemma from Mozilla. And, they worked on the new Firefox Quantum release of their browser. There was supposed to be a big speed up and a modernization of Firefox.

Chris: Right. That's my primary work browser. Now, I've-

Ryan Townsend: Oh, cool.

Chris: ... dumped Chrome, unless it's a Google thing where I need to use Chrome. Maybe I'll go back to it. I've been hearing Google Chrome is getting better at memory use and CPU use, but Firefox has been great.

Ryan Townsend: So, they talked about the kind of perceived performance that people ... you see things like loading spinners and it kind of makes it a lot less worse than just a blank screen. In your example of seeing something is good, but then you were frustrated that the rest of the webpage took so long to load. So, it's how do all of these things fit together to give you that perception of a fast and reliable experience. As soon as things start to slow down or don't work, you just start eroding that confidence from the end customer. So, it's not just the impact on that one time visit where they're trying to use your website, or your service. It's then, thereafter, every single experience is then going to be tarnished or they may never come back at all. I mean, you talked about being on your flight and those are the hotel website that you're using and that wasn't working great. Well, on your next flight, you might be like, oh, well I remember that that one doesn't load very well. I'm going to try another one, and then suddenly they've lost your business.

Chris: Right. Yeah, exactly.

Ryan Townsend: So, it's really important that that consistency is there.

Chris: Yeah. That makes sense. All right, well let's wrap up now. It seems like you're excited about performance and scalability these days. Would you have any, I don't know, a few tips that if you had to kind of simplify a message for developers to make sure they're creating performant, or a happy users, delighted users, and not content or angry users. What one or two tips would you share with them? Things to think about.

Ryan Townsend: Sure. I mean, if we're talking holistically and you're looking at, say, a webpage and you've got to care about everything from the server and client side as well. So, every image, every JavaScript file, every CSS file, all of that, all fonts, third-party things that you're including on top, all of that stuff comes together to create the user experience. So, that can all have an impact. The main thing is just caring really, not sitting in this bubble of-

Chris: Gigabit fiber.

Ryan Townsend: Exactly. You sat at home. You're on your quad core, or octa core these days. I can't keep up with these things. These really fast machines loaded with RAM, processing power, you're on this fast connection, you're having a very different experience than the majority of end users. Regardless of what you're building, it's really important that you look those different situations and analyze them. So, that could be, I know Facebook has this thing called 2G Tuesdays where they literally have a separate WiFi network that's throttled to 2G.

Chris: Yeah. I love that idea. I remember hearing about that. It's great. Just generating empathy, right?

Ryan Townsend: Exactly.

Chris: Helping your team, your developers, your company in general, better foster empathy for the end user is great and that seems like a great way to do it.

Ryan Townsend: Absolutely.

Chris: Yeah. Also, very frustrating. What do you mean you're turning off, not just my four, my three, but you're making me run at GPRS speeds or whatever it was. GSM speeds. Yeah.

Ryan Townsend: That must be painful.

Chris: Yeah. Cool. Well, thank you for joining us, Ryan. Thanks everyone listening to us online, for joining us for another episode of the Code[ish] podcast.