39. Evolving Alongside your Tech Stack

Chris: Okay. Yeah. It came out, and it was like this huge explosion on the scene, right? And got lots of excitement from users and also, I'm sure, VC funders or investors, and other things like that. I do remember. That was kind of like one of the first big audio and video-related mobile apps. Is that correct? Or maybe I just wasn't down with the cool stuff that early.

Tim: No, I would definitely say you're right with that. Dubsmash actually started very classically as kind of the follow up on previous video-related editing app that Jonas, my co-founder, and I worked on. And yeah, that didn't work out well, and then Dubsmash was the distillation of all those things that did work well, and all the simplicity in it. We really had a lot of initial track traction. We still have... Still going very strong. We are based out of New York these days, not out of Germany anymore. Definitely one of the first big video products next to Vine for example. And videos are-

Chris: Oh yeah, Vine-

Tim: That were out there. Yeah, Vine, exactly.

Chris: Yeah. That's cool. So that's one of the things I think that helped make Dubsmash so exciting and so popular, and the topic of our conversation today, was this focus on user experience, right? And I think one thing you like to talk about specifically is delighting the user, giving them something that is useful or fun, but also kind of delighting them with the interaction, their experience with the application. Is that accurate?

Tim: Yeah, I think these days actually, especially on mobile, the opportunities for us as engineers, and as engineering teams and companies, at the end of the day, are really pretty magnificent. If you compare that to the first iPhone for example, that came out back then, you couldn't literally own apps for it. It didn't support multitasking, it didn't support really any gestures in the way we know it these days, and we've really come a long way.

Tim: And what tech enables us these days, is not only to build very sophisticated and very broadly scaling systems of different forms, but it also enables us to utilize that technology, advancement to actually build great user experiences in the end. And I think these days the tools they are so commoditized for engineers that it's really easier and easier to build apps. But with that, of course, also comes a certain bar of quality that users expect. If you just look at the App Store over the last couple of years, it really has evolved a lot. In terms of not only the business models that apps use to monetize on, but also the way they're written and the way they're used by our users, at the end of the day.

Chris: Yeah, that's super interesting. So my background as a developer was not from the front end of development. I was always drawn more to how does the computer work? Or how do I build a system that allows these computers to work together, or these processes to work together? But your background, it sounds like you started as a mobile developer, and it seems like as a mobile developer, like mobile app developer, you have to be very aware, and keep in your mind, and have empathy for the end user from the beginning.

Chris: Where someone who is for example, in a lot of universities you learn C or C++ as your first language. Maybe you don't write a kernel driver as your first thing, but you're very low level and more focused on how does the computer work, how do CPU registers work? How does memory work? Memory allocation work? Instead of what does the user want from this, and how do they want to interact? And how does this irrational being interact with this very rational logical thing that is a computer.

Tim: Yeah, very well said, I think, actually.

Chris: Yeah, I think that's super interesting, you coming from that direction. I was just curious to hear more about your views on user experience and how that's important, or how that has manifested itself in your career as a software developer.

Tim: Yeah, sure. I think actually, at the end of the day for me, both sides, so both the practical knowledge and the user experience side, as well as the side of understanding the internals, and how certain things work, I think are really important these days. I think engineers, and the position of engineer, and the job description that comes with it, has changed quite a bit in the last couple of years, and probably mostly in the last decade or two. It has changed from something where historically, you were very siloed in your department, or in your cubicle, maybe literally, and your main responsibility was to write a certain amount of code a certain way.

Tim: So that was very straightforward, very heads down, most of the time. And these days, usually you are required to have a way more diverse skillset, and not only being able to write an app, or an API, or a custom search index, or something like that, but also being able to understand why you're doing that, and why that is important to your users. And I think that is something that mobile development, especially, actually, the first versions of Objective-C and the iOS SDKs back then actually have taught me. Because on one side, of course you're working on something UI related, so you're spending most of your time dogfooding your own work, because you use the same app, you run it on your phone a lot. You see if that button works, or how the transition is, or if that makes sense on a UX point of view.

Tim: And on the other side, you also have to deal with a lot of under the hood topics, like memory management for example, which is something I'm personally, of course, I'm very happy to not have to do any more these days, because we have on the iOS for example, we have automatic retain counting these days. But for the first, I think, four or five years of my mobile engineering, I did retain counting myself. And it led to a lot of bugs, it let to a lot of memory leaks, because oftentimes it was forgotten. But it also has taught me a lot about how the processor works under the hood, and how the compiler works under the hood.

Tim: So I think for engineers these days it's really important to find a good balance of both. I think you should still specialize in either, though. Or I don't think it's really feasible to be either really great at building a great UX, or being really great at understanding how our compiler works and what preprocessor macros are. But I think at least a baseline knowledge, is at least something everyone should have to really make sure that if you get to that point where you need to debug that super esoteric issue, that only happens on one out of a thousand devices, and you really can't reproduce it locally, sometimes that deep knowledge of how system works really can can help you out in that.

Chris: What about though, you have filled roles from individual contributor, individual developer, earlier in your career, and now you're the CTO, which at a startup can mean many things, right? It can be like-

Tim: Yes.

Chris: You write code, but you also design the architecture or the structure of a system, and also manage other engineers. It seems like as a CTO though, you probably need to, at some point, be okay going broad and not getting down on yourself, or requiring yourself to know every little new thing about iOS development. And then on the other side, maybe not knowing every little thing about how to make Postgres queries efficient, or something like that. Is that accurate, or are you still able to do both? Kind of go broad and deep?

Tim: I think it's mostly a thing of prioritization for me. I think one thing that I really had to learn over the years is how to let go every once in a while, and have problems that be dealt with in ways that might not necessarily be the ones you would choose yourself. But I think that's one thing I personally really enjoy about engineering. There's always a better and the worse, but there's very rarely only one option that actually truly works, and that is the only path forward.

Tim: And that is something that I usually end up deciding quite on the fly, when it's enough for me to be on a high level familiar with something, and when it's necessary to actually go deep and understand why a certain piece of code works a certain way, or why, as you mentioned, Postgres for example. What implications are higher traffic workload on your index strategy has-

Chris: Yeah.

Tim: And how that affects the physical tables on disk at the end of the day, and the operations operating system has to do. And I think that is something that has changed for me personally a lot in the last couple of years. I used to really be the person that is always going as deep as possible, and if necessary, spend days on figuring out certain solutions to a problem. But these days I'm much more... Of course working with my team on that, and making sure that I'm not necessarily the one that actually gets to answer or solve it in the end, but also at some point cut your losses.

Tim: Is a certain obstacle preventing a feature from progressing, or from being completed, or a certain kind of bottleneck and scalability prevents a feature from rolling out in the first place, then there is always two options, right? Either you can go in and fix it, whatever it might be, or you can decide, hey, okay, approach A works for some reason not as well as I would've thought. So let's instead of trying to make approach A better, let's just cut our losses and figure out what approach B could be. I think that is something that I really picked up over the years, that ability and that insight to be honest as well, to know when it's time to move on, and when it's worth it going deep.

Tim: That is I think really one of these things in engineering that make it really exciting, because there is always so many new technology, so many new paradigms or architectures or even frameworks released on a day by day basis. That on one side of course makes it impossible, really, to truly keep up with everything, and truly be done knowing or learning things at some point. But on the other side, it also implies that while one year a certain problem might've been too hard for you to solve based on the resources or the knowledge that you have, so the next year it might actually be a problem that is totally within reach of you and your team.

Tim: And I think a great example of that in the last couple of years has been TensorFlow, the machine learning-

Chris: Mm-hmm (affirmative).

Tim: Framework, let's call it, from Google, that has really commoditized the way we can build machine learning models. Two or three years ago, that would've been impossible for most of us due to resource constraints, teams being too small or that knowledge still being just very expert level, and by that being very expensive. And these days it's literally possible within five minutes to build an image classifier that tells you if an image is dog or hotdog.

Tim: And that is something that I know it's a funny comparison, but the HBO guys say they actually built that little sample app in C++ and OpenCB, and also TensorFlow, I think. And they incorporated that into the show and everyone had a good laugh about it. I personally like that episode a lot, but it also tells you a lot about what's possible these days with technology, and-

Chris: Right...

Tim: What we can do as engineers to take that knowledge, and take that possibility and unlock some form of user value for our very specific users, depending on the product we're working on.

Chris: Yeah. Yeah. That's cool. I wonder if... You know how there's the canonical app, at least for front end frameworks, there's the todo app, that-

Tim: Yeah.

Chris: Is a web app, that if you build a new front end framework, so when React came out, or Vue.js came out, or Angular, people try to show the code to build a simple todo tracking app. List of todos. I wonder if hot dog or not hot dog will become the new canonical app, or maybe the canonical app for machine learning that you build, and that's how you compare different machine learning frameworks, or things like that.

Tim: Yeah. I think at least for machine learnings and image classification in particular, have actually seen that quite a bit already. But what I find interesting, very personally, about the todo list example that is probably the most well known "Hello, world" app these days. 10 or 15 years ago I learned... Was it Visual Basic ? That programming language where you made the turtle walk on the screen and draw stuff. I-

Chris: Yeah, I don't remember what language it was, but there was, I remember the cursor, and it's like-

Tim: Yeah...

Chris: A triangle-

Tim: Exactly.

Chris: It was called a turtle, and it would draw where-

Tim: Yeah.

Chris: Ever a path that it would take, until you could learn for loops-

Tim: Exactly. And that was back then the Hello, world app. That was the first "Hey, here, look I can program", and then over years, with React and Backbone.js, and all the likes, it turned into that to do list app. And these days even a simple todo list is too simple for a Hello, world App. People all of a sudden are like "Oh yeah". We used to be able to do, todo lists as the MVP for new engineers, that go into a coding boot camp or something like that. And these days we're going to raise the bar to be able, within 10 minutes, to be able to distinguish pictures, which is really crazy how that has accelerated over the last couple of years.

Tim: It's actually interesting, I think, in what it teaches people, because on one side, of course, it teaches people way quicker to utilize the tooling that is available in whatever stack that might be, and that they might be interested in, and how to use the most common libraries, which I think is great because most of the things we build today, we don't actually have to build ourselves anymore. It's more about stitching them together, and being smart about which components, whether that be software, or hardware, or infrastructure be to use. But on the other side, it is also dangerous for people, I think, because it abstracts away a lot of the under the hood things that are happening in terms of memory management, in terms of import structure.

Tim: You should have seen me for example, struggle a couple of years ago for first time setting up a React app, because there was like 5 million different options, and bootstrap tools, and tutorials out there on the internet, and I picked a bunch but then they didn't work out really. And I needed to learn first how Webpack, for example, actually works, and how it imports modules in order to overcome that. And that wasn't really part of any of that. And that was a very frustrating experience for me personally, which I'm very happy to see that more recent tutorials on Kubernetes or TensorFlow or any of those, they do a little bit better. And finding the balance between "Hey, I want a display on how easy I am to use and to integrate", as a sales argument. And on the other side, also making sure that the engineers that end up using those frameworks and libraries truly understand the implications and the under-the-hood workings and the architecture of it as a whole.

Chris: Yeah. It seems like it's a common theme in this very, very fast evolving world of software development in general. It's kind of the idea of abstractions, right? I as a software developer want to be able to use higher and higher level abstractions, so that I can produce more, faster. Like you said, stitch things together, produce more faster.

Chris: But then yeah, there's pitfalls that you can not understand, or not know about. When I say stitch something together, with these very high level abstractions, and then push that thing to production, or release it to customers or users in some way, and it breaks. What do I do? How do I... Do I know how to debug it? Do I know how to read the stack trace that comes out of that? Does it even produce a stack trace that I can use?

Tim: Yeah, it sounds, sounds like a funny question. Something that doesn't produce a stack trace these days anymore, but that actually exists, right? And I think that is probably one of the most interesting inflection points for young tech startups especially, I think, because in the beginning of course, it's all about being as quick as possible, and going as broad as possible, and covering a lot of ground. And validating your product idea of whatever sorts.

Tim: But as you grow, and not only within your team of course, but also hopefully with users, you will get to that point where the full-stack generalist approach that you used to take in the past, probably doesn't scale and doesn't work as well with your current usage anymore. Because you do have more and more issues to debug that are rather scarce, and that are really hard to find and to track down. And at that point you need to be able to either advance from your very broad skillset into a T-shape, in some shape or form basically, or you need to bring talent on-board that has that deep fundamental knowledge and actually is able to diagnose memory issues for example. That is something that we-

Chris: Yeah...

Tim: Historically always have found as the most troublesome to implement or to debug and to fix, because there is usually no stack trace because there is no exception being thrown. And on the other side, it also something that is very impactful for your users, and for your infrastructure, because either your machine just crashes, or if you run on something managed, like on the AWS, Google, then your machines just get terminated due to over utilization of resources, right?

Tim: So you have no trace of anything going wrong except for your machine to all of a sudden restarting, you know, and you dropping traffic as a result. And at that point you need to be very familiar with your stack and with the tooling and with the frameworks you're using. And you need to understand how they work internally, and to a degree where you need to understand how Python for example allocates memory and how the global interpretable lock works for it. And that is something that I think takes years of experience, and most importantly years of a failure to see, and years failure in production as well.

Chris: One thing I think about, that's always spinning in my mind is the industry that we work in, there's a lot changing in it, and it's exploding, and on one hand it's like technology is becoming core to what people do at work every day. It's becoming, it's growing and kind of has tentacles into everything that happens in the world today. Whether it's like social interactions with your friends or business in general. It's like our industry has become this, I don't want to say foundation, but it's just like seeped into everything.

Chris: But along with that then comes, people, companies trying to create services, right? Trying to create the shovels and the picks to allow developers to build bigger and better things. So examples are the service catalogs from AWS, or GCP, or Azure, the Kubernetes, not just Kubernetes the project, but then the ecosystem of other open source projects that have come up, or exist all around it. And they're all at kind of different levels of maturity. And then to what you work on.

Chris: There's new startups coming every day. Some are more consumer focused, but some are also very creating services for developers. This is Heroku back in 2008, 2009, 2010. How do you as a CTO deal with that chaos without going crazy or feel that you are doing the right... Doing what you can, or doing the right job, or doing the right thing for your business, for the product that you're building, making the right decisions, but also knowing that a human can't keep track of all of these things that are available, and know everything about all of these things.

Tim: Yeah.

Chris: How do you think about that?

Tim: Yeah. I think at the end of the day you need to be really invested into this space, the stack, the, the ecosystem that you're working in, and follow it quite a bit. I personally follow a bunch of publications, stack shares, only something I can recommend. I read a lot on Medium, and I think that is usually something I tend to do on one off occasions, where I stumble over a certain service being released, or someone posting on a HackerNews about having an experiences with a certain AWS service, or something like that.

Tim: And that's simply consume, and the way I use those picks and then axes that you mentioned earlier, I see that as my mental tool shack, you know, and I know exactly-

Chris: Yeah.

Tim: What kind of tools I have in there. Some are a little bit trusty, some a little bit more shiny, for sure. Some are great for excavating some bedrock. Others are probably more for getting rid of a couple of flowers in the yard. But it's important to know what you have at your disposal. And that is not the only technology that is in the first place kind of like available to you by either an open source framework, or a library or a hosted service on AWS, or GCP, or Azure, or whatever other cloud you're in.

Tim: And once you have that mental image, and once you have that little tool shed off yours, I think it's very important to A, be excited about any new tools to it, because that's the only way you will evolve and continue learning as an engineer. But on the other side, also being very realistic and very honest with yourself, and to be frank also with your team on which tool is actually ready to go into the shed, and which tool is more still like a blueprint, and not truly ready for use.

Tim: And that's always that question on how quickly do I want to adopt a certain new paradigm, or a certain new technology. While we here at Dubsmash usually try to be very, very quick and very decisive about things like that as well, we do usually wait at least a couple of months to hear the first waves off complaints and of stories basically, about what worked out well and what didn't work out well, and also to have some time to talk to other people in the industry. And once we've identified something as a tool that we want to add, we usually commit fully to it.

Tim: Once that decision has been made, and we all feel comfortable with it, we usually go and start integrating it either on a smaller scale basis, or if it's like a Django upgrade for example, from one to two or something like that, we usually roll it out a little bit wider across the organization. But what you should avoid at any cost is jumping on board a ship that just left the harbor, and that is very likely to sink right in front of it.

Tim: I think one of the best stories, probably not, but one of the worst stories I think that I like to tell about that, is graph databases that we used a couple of years ago. Back then we had a Cassandra, a storage layer, which is great. I'm personally a big fan of that kind of technology. And then we had something called M Titan sitting on top of it, which was basically a GraphQL engine, back then it was open source, and then it was part of the Apache Incubator, which tells you a lot about already in what state that project is in, right? If it's still in the incubator... And then the company that was maintaining Titan as a framework got bought by DataStax back then, so they enterprised it. Made it a little bit more enterprisey, kind of stopped supporting the open source work of it, and that project kind of like ebbed out a little bit. We had bet a lot of our product strategy, a lot of our in-app recommendations and social connections on top of that.

Tim: And that ended up costing us I think two months of development time, because we started integrating it, we've tried to get Cassandra to scale to a level where we would want it to be, based on the usage off the graph database, which didn't work out. And then in the end we had to implement a completely different solution.

Tim: And that was something that I think we could have avoided, if we would have waited for another, not even much, two month, I think. And would have been a little bit more honest to ourselves that a project, for example, that still is in the incubating status of Apache is incubating for a reason. So while it might be great technology that's available, and while it might be a very great skill and tool to pick up in your spare time, or in a development capacity, it's probably not ready for big scale millions of users production using yet.

Chris: Right, right. So we got to wrap up now, but is there anything that you want to share out with people about like, is Dubsmash hiring? Do you want people to check out something new you're working on or releasing? Anything, anything new and exciting you personally, or you professionally would like to, wrap up with?

Tim: I mean, yeah, we're definitely hiring in New York. We're looking for a lead iOS engineer and a senior Android engineer right now. So we're staffing our mobile teams quite a bit more. Other than that, I think if there's probably one piece of advice I would want to give other engineers today, it's keep it simple. I think building simple solutions and building simple architectures, and choosing the simple way to solve problems first, at least, is always something that has done me personally very well in my career, and in the code that I've written at the end of the day. And I think if you always opt for the most simple, most elegant solution, that's probably a good default to base your decisions on.

Chris: Yeah, that's good. I think good wisdom from someone who's experienced lots of ups and downs, failures and successes. I think I agree. Yeah, I agree too. I think the older I get, the more I feel that, what you just said, that keeping it simple is so important. Cool. Well, thanks for joining us, Tim.

Tim: Thank you so much for having me.