56. Updating Legacy Code

Corey: Huh? So it's not necessarily negative?

Joe: No, definitely not. We've all written legacy code, and we all will continue to write code that will become legacy.

Corey: That's good to know. What is an example then of a legacy system? It seems pretty broad. What's one that you've seen that has screened legacy to you?

Joe: Well, that's true. It's a broad definition. Because we work in the services industry of software, we tend to look at legacy software as software that we didn't write. A legacy system is simply one that somebody else has built and has maintained or is maintaining, and they're now giving us the opportunity to jump in and look at it, add, improve, et cetera.

Corey: In your work with legacy software, do you find it's frequently within large non-technical companies or product companies or both?

Joe: It's definitely both. Legacy software is everywhere. To the extent that people attach some feelings to legacy software, sometimes though it can connote a large system or system that's been around for 10 plus years. I don't necessarily subscribe to that, because the fact is that software changes so fast that the tools and even the approach you use to constructing software could be made obsolete in a matter of months, in which case it's important to look at it as something that is evolving, and something that you need to constantly take care of and keep healthy.

Corey: In your role, you're often brought in to say improve a legacy system or address some problems with it. Let's play this out and say I work at a company and you're called in to help us out. How do we know when we should call you when the system needs help?

Joe: There's a couple of things. One is, it's really hard to ask for help in this industry. In some ways, that's just a business problem. One thing that's true about Def Method is that we are often, though not always, called in to provide something that sounds innocuous like, "Hey, we need some extra engineers to help us build a new feature or give us an extra engineers because we're scaling very quickly."

Joe: While all of that may be true, I think it's really hard from a CTO's perspective, or from a key stakeholder's perspective, to go to the business and advocate for dollars when the dollars are needed strictly for rehabilitating a system that may be unhealthy. Because there's a stigma there, it's all of a sudden you have to explain to nontechnical people some very technical things, and it's really easy for that conversation to get turned around into, "Why isn't it healthy? Right? It was your job to keep this thing up and running, and successful forever, and so why would we need to bring in anybody else?"

Joe: Which really is unfortunate, because every system needs help. No software program ever has been released without bugs. Sometimes those bugs get fixed really quick and dirty, and so there needs to be some love that goes into going back and rebuilding something. New business features are actually the biggest driver of legacy code that can be hard to maintain because it can sometimes be easier to bolt on a solution in a quick way rather than going back and planning and viewing the entire architecture and looking for the best way to add a new feature.

Joe: There are lots of reasons people get into the situation, but suffice it to say that when Def Method gets called, typically we're getting called not specifically for help with the code base, and so we have to look a little deeper to see, "Well, is our help needed on the code base or do they really just need staff augmentation?"

Corey: Say a company calls you, and says, "We need more bodies," like you were saying is often the initial ask. How do you get them from that to acknowledging the real problems that the software might have that they need help with?

Joe: Many times we do our diligence, so it starts there, right? We acknowledge that any system that has been up and running for some substantial amount of time has been successful, right? So developers have come in, engineers, architects, product managers, business people have come together to build a product that is successful, and has achieved some measure of success or else we wouldn't be there. There'd be no money, we wouldn't be able to go in and do anything.

Joe: So if we approach it with that amount of respect, then we're able to look at it and say, "Okay, how can we provide value?" Sometimes that starts with a simple code audit, if it's a small project. If it's a larger project, it may start with ... It could be one of our developers that is just sitting down, working with an engineer on the team to try to do something simple, build a new feature, resolve a bug.

Joe: A lot of times, we do this up front and we start to call some of the things that are really issues with the system. Honestly, we just have really frank, candid feedback for the people that are requesting our presence, to say, "Yes, we want to come in and help you. Yes, we think we can make positive progress. Here's what we think we need to do in order to make that positive progress."

Corey: You said sometimes one of the first steps is sitting alongside an engineer that's already on the team, looking through the software, et cetera. Maybe having some frank conversations about it. Say I'm an engineer on that team, how do I first meet you and what are our initial conversations like?

Joe: I think these days you wouldn't meet me, unfortunately.

Corey: Right. Of course, you're the CEO. I should say one of your team.

Joe: Right. So somebody on my team. But that's important for a couple of reasons. One is that, I don't write code every single day like I used to, and we really need to build trust with the team. The way to do that is by having one of our engineers demonstrate themselves to be capable of adding meaningfully to the system that you have been building and maintaining all along.

Joe: The type of interactions that happen, we start with easy questions about process, because processes are about people and people will always lead us to the technical problems. So typically, a conversation can start with, "How do you gather requirements? Who does acceptance? Where's the staging environment?" Right? Those kinds of things. Or even, “What's your cadence? How often do you release?”

Joe: A question like that is process-oriented at the surface level, but underneath has a lot of technical underpinnings. If they're releasing to production once a month, you can ask why. Or if you ask, "Okay, well, what's it like to release? You should start ..." Then you could say, "Well, what does it take to release?" The answer really should be, "Well, we run this command, right? There should be a single button or a single command that does a deploy."

Joe: If that's not the case, we can ask why, right? We can just start to dig into it a little bit. Again, this isn't an interrogation. Our company doesn't run perfectly and we don't expect other companies to run perfectly. We expect there to be issues. But we've seen a lot of different issues, and we've helped to fix them. A brief frank conversation in the beginning, and especially with an engineer or a PM that's on the team, that's low level, that's ... by low level, I mean building the software every day, not just talking about it.

Joe: Those folks typically are really honest with us, right? They'll tell us where the issues are because they want it to get better as much or more than we do.

Corey: So you're working with the existing team, you've established a level of trust, they know that your intentions are to help improve the software. How do you ultimately evaluate what needs to change and how do you communicate that?

Joe: You can always improve a code base no matter what. That doesn't mean that those changes are needed right now. A big part of this is understanding where the system is now and how safe we feel pushing something into production, right? How safe we feel pushing a new feature into production. The way we evaluate how safe we feel is seeing, are there any automated tests? If so, where are they? Is there a CI/CD pipeline? Where is it? How do we run it? How do we evaluate the feedback? Then is there a staging environment that exactly mirrors production? Right?

Joe: So if we have those things in place, then we can start to say, "Okay, this system, which is legacy and by definition is imperfect, we can make some reasonable assumptions that we can get code out the door in some manner and feel safe about it." If any of those things are missing, that's where we start digging in. It's not our engineer's job, by the way, to say, "Well, we can't do anything unless you fix this." That's really the engagement manager's job. It can be my job. On large project, it can be a product manager's job. That way, we're not pointing fingers at anybody, we're just saying, "In order for us to be effective in our work, we need to be able to have these things in place."

Corey: Say that you find a lot of important and necessary changes that you recommend, but the budget's not there or the amount of time you've been given is not there. How do you deal with that?

Joe: I think we're crossing into a really important topic, and my staff has come up with a way of delineating or separating these different kinds of projects. There's legacy software which can encompass almost anything, and we call that brownfield software. There's greenfield software, which most people are familiar with, that's brand new code on a new project or an existing project, but it's writing something from scratch.

Joe: Brownfield is anything that's legacy. Then we also designate a category called minefield, which is projects that really are struggling in some critical way, and we don't feel that we can make a change to the production environment without considerable risk that we will trigger a mine, right? So bring a server down, cause real serious disruption to users, et cetera. When we come across those things, we have to flag them, and we have to tell them, "Look, this is not a code base that we feel safe shipping software to just yet." And—

Corey: I'm sorry. I'm just really curious about minefield. It sounds a little scary. What's an example of something that would pretty much make software minefields?

Joe: So what makes offer a minefield is typically you can discover it in conversation a lot of times. You ask, "Is there continuous integration?" They say, "No." You ask, "How do you deliver software for production?" They say, "We ssh onto the production server." You say, "Are there any tests?" They say, "What are tests?" Right. But those kinds of questions come up. The good news is that most of the time minefield customers can diagnose themselves.

Joe: A lot of times, they're not just hearing about these problems. They've got a team or they had a team that was trying to build and deploy software, and things were taking way too long, right? Like a user feature that used to take a day, now takes two weeks, and they have no confidence in their releases. After they release, it's just all hands on deck because anything could go wrong. And by the way, this was a central part of my career as a software developer, was rescuing projects that needed it.

Joe: So I love it. I have a soft spot in my heart for projects like this. So the good news is they know that they need it. You did ask before, what if they don't have the budget for it? That's really hard, and it happens sometimes, especially when we're working in the startup space, which is relatively common for us in New York, where they say, "Hey, look, we had 200 grand that we spent on this team that said it was going to take six months. It's been a year and a half. We spent all the money and we don't have anything to show for it."

Joe: I really sympathize here. In those cases, the good news is that we really can make some really positive strides for them on a shoestring budget if they will ... don't want to say listen to our recommendations, but if they're willing to make some changes and commit some budget to improving things, we can make a really strong case. For example, like what I told you before, well, if you ask them, "How do you deploy?" They say, "Well, we ask ssh onto the server."

Joe: It's easy to say, "Well look, we can get you off of that. We can get you out of that. We can put CI/CD in place. We do use Platform as a Service, we do use Heroku all the time, and we can get it to a point where at least you can release with confidence, right? At least you can get to a point where your existing team can show you something on a staging environment before it goes out into the world, and you can look at it, and you could give the yes or no, and they accept or reject.

Joe: Those kinds of things, really fortunately, sometimes they just take a couple of weeks. To go back and rehabilitate the whole code base could take a long time, and you have to make a decision on those minefield projects, whether to continue with what you have or whether to scrap part of that and rewrite it. Those are harder questions an those can be more expensive. But you can make really meaningful progress quickly on even a minefield project.

Corey: I want to talk about some of the areas of a legacy application that you go in and fix, and I want to start with the code. What do you look for when you read code in an application?

Joe: If you asked my engineers that, you'll probably get a couple of different answers. I think some people start with the tests because we are a test-driven organization. Some people start with the database tables because they want to get a sense of the data architecture and perhaps the modeling. We work on a lot of different kinds of systems, and so each one, for working in Django might have a certain set of really common antipatterns.

Joe: If you're working in Rails, that might have it's really common antipatterns. You'll see a lot of things. AntiPatterns are something that you can look for. You can look for a God-class, as it's referred to, right? Where there's some class that controls 80% of the application, and things just keep getting bolted onto that. We also use tools both open source and paid for things, like Code Climate, things like Flog and Flay for Ruby projects.

Joe: There are all kinds of static analyzers for .NET and Java projects that can tell us what's the complexity of different parts of the system. Where is there the most churn? Meaning, the classes are changing really, really frequently. You can start to suss out in really just ... We do this often, so in really less than a week, an engineer can get a really good sense of where the bodies are buried so to speak, and what needs to be targeted first.

Corey: What about infrastructure? You use an extreme example of you ssh in to deploy, or maybe it's not that extreme. I don't know. Maybe it happens a lot. But more broadly, the hosting situation, the CD situation, what do you look for in infrastructure that might need improvement if it's not there?

Joe: It is extreme but it is also common.

Corey: Wow.

Joe: Again, I don't want to stigmatize it. If somebody is coming to us for services, right? Whatever they may be, it means they've done something meaningful, unless it's greenfield and maybe they're just a brand new startup, then they're trying to do something meaningful. But 90% of the cases we see, there's been meaningful progress and we just want to improve on it.

Joe: Beyond that extreme example, a really frequent occurrence is that staging does not mirror production. So things look good on staging, they're shown to business people on staging. They may be even demoed, and then we release to production and it's like, "Oh man, well, something went wrong," because the data is not the same on staging as it is on production, or the architecture is not the same, sometimes databases aren't the same. Sometimes databases aren't even the same type, right? They could be SQLite on one and Postgres on another, and sometimes they're just ... not the same size.

Joe: Plenty of cases where ... Well, with production, we throw some giant EC2 instance at it and in staging we don't. So you can start to see different issues. I guess, you can go on beyond that. Is there a CDN behind production? You can find some nitty gritty things that are happening. But to be honest with you, usually the issues that you see are more glaring, and they're just a product of either somebody not knowing or not thinking it was as important, and maybe it wasn't in the very beginning, and it is now.

Corey: You talk a lot about people in teams and process. When you find yourself making recommendations about how people work or how teams day-to-day execute software development, what do those recommendations look like?

Joe: I love agile software development, but I am of the opinion that that has become a really bloated term in our industry. Many people define it differently to the point that it's a little bit watered down. So we've moved away from saying, "Well, this is agile and this is not agile," and we've started just asking questions and looking into different people's processes. Because the fact is that, a process that works on a Fintech customer works totally differently in our government sector customer.

Joe: People and processes, it is huge. One thing that we want to understand is, is the stakeholder or the person that is requesting our presence or requesting our services, have they ever worked with a software development team before? Have they worked with an external software development team before? Start there. In the process side, we just want there to be a cadence, and so without a lot of ceremony, we want to know is there a specific time that software should be released? If not, let's try to pick one, and say that, "Maybe we can release any time, but we're definitely going to release every Tuesday at 2:00 PM," right?

Joe: Is there a time where we can do a retrospective or reflect on the work that we've done with the goal of getting better over the next period of time, usually one week or two weeks? Is there a planning meeting where we can look at, write, edit user stories, look at our velocity, which means how quickly are we delivering stories, and target how many stories we think we can deliver over the next amount of time. Just putting those things in place really helps to put everybody at ease.

Joe: You to say, business people all of a sudden start to get things regularly, salespeople that are asking for things, they don't get everything all at once, but they get a little bit at a time and they can start to count on it. The engineering team can count on an opportunity to refine and improve their practices in a forum that is free from a lot of finger-pointing and blaming at a retrospective, and product managers, designers can all feel like they are a meaningful part of the process by contributing to the planning and to the overall strategy.

Corey: If I'm on a team, Def Method makes some recommendations about how we work and we implement them. What, as an engineer on a team, might I get from this experience to help in my career or what I want to do?

Joe: It really depends on how open you are to doing things differently. Some people are really set in their ways and they don't want change. If you are looking to grow your career, then at the very least, if we come in and you see us doing things a little differently, first of all, we're not doing it in a vacuum, so we're including you and we're asking your opinion, and we respect your opinion because you've been there longer than us so you know the legacy software better than us.

Joe: So you're able to see how we work in a collaborative, transparent way. We can give some really honest assessments of the software and where we stand. You'll also probably ... Knowing our engineers, we are constantly learning. Everybody at Def Method has in common the fact that they are career learners. Even forgetting everything else, you might just learn where all the good Meetups are nearby because we're going to them, or where the conferences are, or where there are good talks online or books to read because we've got an arsenal of them and we love talking about them.

Corey: That's great. So making it last, I'm on a team, you come in, I've maybe grown a bit as an engineer in this process. How do I help make sure that the changes you recommended that we implemented together stay?

Joe: You really have to be an advocate in your own organization. If you like what we have to offer, and you feel that things have been better during our time together, then you have an opportunity to continue doing them. But you will inevitably face resistance, because as soon as we leave, there will be some amounts of the team that wants to go back to the way things were, or go in a different direction.

Joe: Or some business person that says, "Well, this worked great for the time, but now we really have to move faster and we can't do X or Y, or we have to compromise on Z." The best thing I can say is, one, you need to advocate for the changes to stick, and two, you have to build up within your own organization people that also believe in the same things and want to do things in a correct or a better way. That's your best chance for making things stick and making things last.

Corey: Rewinding a bit, Say I'm on a team that is building greenfield software, we're making a new product, and we don't want to reach a point where, no offense, we call you for help with some serious issues on our software. What can we do in advance to try to avoid some of the legacy software pitfalls that you've seen?

Joe: Right. Well, we don't want you to call us for some serious problem either. On greenfield projects, we want you to call us because six months later you say, "Man, we've got so many people using this thing that we need help building more capacity." Or, "We've got a whole new class of users that we didn't expect that want to use it." Those are the calls that we want.

Corey: Absolutely.

Joe: How do you make that happen? There are plenty of books out there on keeping your software clean, on adhering to software patterns, architectural patterns. Some of them have been around for decades, others are still being written. There are great authors out there. But I think the best thing you can do is to continue learning yourself. They say the mark of a good programmer is that they look back on the software they wrote six months ago and they say, "Oh my God, I can't believe how I wrote that." Right? You would do it so much better now. You would do it so much different. Right?

Corey: We’ve all been there.

Joe: Yeah, we have. I think the humility that comes with that is important, and if you can keep doing that, well then that just means that you continually improve. There's a lot of advice I could give here. Startups are really challenging. There's a real urge to move really quick, like that ridiculous expression, move fast and break things. That might be fine from a business standpoint.

Joe: From a software standpoint, there needs to be constant attention and care to your software. I can recommend some of those same tools, Code Climate and others to keep an eye on your software, to keep a check on your dependencies, a check on the complexity of your software. The biggest thing you have to do is continually make the time, advocate for the time, to put into your system to make it healthy and to keep it healthy.

Corey: So you wrote a book, The Well-Grounded Rubyist. I had a read before this recording, and I really like how you cover the philosophy of the Ruby language, why it is the way it is, in addition to how to use it. Would you talk a little bit about why you wrote a book?

Joe: Sure. I'm flattered that you read it. It's not a quick read.

Corey: Thorough.

Joe: But it is thorough. I like the word philosophy. It's a little bit like a textbook with a philosophical bent. This is how the Ruby language works, and this is why it works that way. The key for my involvement with this project is that I did not write the first edition of The Well-Grounded Rubyist, and I did not write the second. I co-wrote the third edition. So there is a gentleman, and Ruby star, by the name of David A. Black. He's a close friend of mine. He wrote the first edition of The Well-Grounded Rubyist.

Joe: It's still, to this day, one of the most influential books in Ruby. Everybody I knew read it, I read it, and the publishers came to him for a second edition. He did that, it was fantastic. They came to him to write a third edition and he said, "No, thanks. I've had enough. But I know this guy, Joe, and he might be interested." We started to collaborate. The things that I brought to the process, and the reason we decided we wanted to do another one was not only is the language changing because every language is changing, but the Ruby language is changing in such a way and moving toward a more functional style.

Joe: They are trying to but not yet at a point where they can achieve real concurrency across the language, which would be an enormous benefit for a language that is dynamically interpreted. There are real tectonic shifts in the language, and the Ruby maintainers approach language development. So we wanted to capture that, and so, in addition to updating the entire book to reflect the new language features and phase out features that aren't there anymore, I also wrote a new chapter that is just dedicated to functional programming in Ruby. That's been a lot of fun to learn and to talk about and to put on paper.

Corey: Listeners, you can get 40% off Joe Leo's book, The Well-Grounded Rubyist, by using code Podish 19, that's P-O-D-I-S-H 19, through Manning Press at manning.com. M-A-N-N-I-N-G.com. We've talked about a lot, Joe. I've really enjoyed this, I've learned a lot about legacy software and how to make positive change. Thank you so much for being on Code[ish].

Joe: Thank, Corey. It's really fun to talk about this stuff. That's why I do it for a living, and so happy to chat anytime.