80. Defining Operational Agility

Yotam: And so the same thing can be applied to operations. And so operational agility is treating each of your operations as something that is evolving. Something that also requires a Build-Measure-Learn loop, and making sure that the feedback comes from all of your organization and not just from one person's experience or opinions.

Rick: What are the goals when you think about operational agility and bringing it to an organization?

Yotam: I think there's a few things we can talk about. The goals of operational agility is to make sure that you're running the team in the best possible way in every aspect. And so, as you're analyzing, what is it that makes your team move? Your agile development process, your development workflow, your CI/CD, your architectural process design and implementation, the developer experience that your engineers are feeling as they code every day locally and through your environments, all these different aspects of the way you work have an operations to them. And we can define what an operation is in a minute, but having an operational agility means that you continue to improve on those processes. You continue to ask questions, gather new feedback and iterate on your process.

Yotam: And so if I can use a term that I ran into recently and a really good article by Jeremiah Lee, he is quoting a person named Joakim Sunden who was an Agile Coach in Spotify where Jeremiah used to work and they use the term minimum viable agility. So think about an MVP for an operation is telling your teams not something like, "Hey, where are the agile police? This is how you have to work. You can't move left and right. No freedom." Not at all. It's just saying, here's your minimum viable agility based on what we know so far. Here are the best practices. Here's a bunch of decisions we made based on the best feedback we can get. Why don't you go and implement that as a starting point and then teach us new things?

Yotam: I think that's our overall goal is to just operate as best we know as a team, learn from each other and continue to improve the process on all fronts. So, that's operational agility. Specifically, what is operation? What am I applying all this agility to? I think there too there's an interesting way to look at it. And for me, looking at an operation is at the well-rounded version of the term. And so what is an operation? It is the overall processes, practices, culture, tooling, and technology. All these things come together to solve a single process or a single aspect of the way you do engineering is an operation. And so building all these things as one, as a team, how are we going to do things? What do we want to follow? What is our value? What are the different milestones within this process we're defining? How are we going to communicate around it? Answering all those questions continuously, that's operational agility.

Rick: That's a great answer. And it's really inspiring to hear about that iterative approach and what you mentioned as far as an MVP for an operation, the minimum viable agility. Does that map to, as we think about a traditional sprint and hitting a retro, and then looking at experiments or things that we could improve and just kind of tweaking something at the team level? Does that analogy hold?

Yotam: It does. And I can tell you a little bit of how we do it now at Electric. So the way we start in building an operation, we call it a journey to build an operation. We create a set of journeys to map everything that our team does. And each such journey ends up with a mature operation we can then follow and iterate on. And so the initial journey usually takes a lot of interviews because as we said, it's not about what Yotam did in the last 10 years and thinks is the best. It's not about any single individual. It's about what's going on in the field, how the teams are doing, what works for them and what doesn't, and just the overall synergy of feedback and approach that we can bring.

Yotam: And so, as we define the operation initially, and I could take Agile, for example, itself, Agile development is itself an operation. It's funny not to mix the terms, but it is an agile operation defining Agile development. And so starting to build that was for us, a lot of groundwork, just going team by team and getting feedback and asking the same set of questions. How do you currently do Agile versus Scrum? How do you run your Scrum? What's special about each of your ceremonies? What works and doesn't? What cadence do you do? Whatever it is, you're grooming or planning or anything else, but what values do you follow? How do you measure your success? All those questions, we went and asked every team.

Yotam: Of course, we added a lot of research on what are the best teams doing out there in the world at the moment. We all had a bunch of years of experience, some opinions and thoughts and ideas on how to do Agile well, but things are moving, as we said. So learning everything we can from inside from our teams and then a little bit from outside to get some inspiration. And then at that point, we could put together some kind of a draft for our operation. So a draft for all those things I mentioned before the practice, which in Agile's case is a lot around the ceremonies and how to run them and the inner workings of a sprint for a given team. The measurements for us, for example, two things that we like to measure around Agile is delivering on commitments, sprint by sprint, and then for our roadmap as a whole, as well as a volatility, essentially the change in velocity. But those are just examples. More important is the concept of defining our measurements and our success.

Yotam: And then the last thing was assuming we put together this kind of an MVP for our Agile practice, how are we going to continue the loop and learn. And so that came in the form of setting up what we call a council. Our Agile Council is a group of people that are passionate in thinking about Agile. And that's a pretty democratic things. People can join or leave as they wish. And we have some leaders that stay long term until they too feel like it's time to move along and pick up a different practice. And then this group has a very lightweight operation that just defines how often it meets and what we do in those meetings. We look at the success rates from the previous month. We look at what works and doesn't. We see if people have sent feedback to the council since the last meeting. We look into ideas or topics that the council members want to introduce. And then the goal of the council meeting is to end up with some action items that help us iterate on the operation itself.

Rick: Right. And you mentioned that council looks at, or maybe this was pre-council or above the council, looks across the board at every process and procedure that you have going in your software development life cycle, including you mentioned current Agile practices. I think you mentioned at the architectural level?

Yotam: Yeah. There was different groups we have different leaders lead different operations and each of them sets up that cadence separately. Some operations are really big and require a lot of iteration, a lot of opinions and so we set up a council for them. And some don't and so we keep them as an operation as both a living document and a practice in the field. And we iterate a little more freely just by raising subjects in any of our forums, any of our leadership meetings or team meetings or anything else. And so separate operations have separate groups that focus on just owning them and giving them all the attention needed and making sure they stay agile.

Rick: What you're doing there is you're also not putting a business-wide or a large organization-wide kind of mandated in place, recognizing that each part of the organization might have different needs.

Yotam: Yes. So the idea of owning the operation is to have some centralized group that thinks about it, but also not have it in a way that is just the dictatorship. And so we have different operations are owned by different group of people. We try to have interests play a part, and then these things move along separately and with different cadence based on the subject. And so there might be a group within the architecture operation of folks that are thinking about our architecture, it's longterm. There's a lot of diagramming and mapping and thinking about the future and maybe a few months later rethinking and seeing what worked and didn't. That group has a certain independence to it. As I said, it takes ideas and feedback from really anyone that's interested, but it would be separate say from the group that takes care of our Agile practice. Separate interests, separate cadence requires different communication with different people.

Yotam: And so those are certainly groups that are managed separately and has a lot of freedom on how they operate. And I certainly don't have any intention or interest in trying to sort of be the thought leader of all of them or even any of them. And so I'll be sort of an advisor to every operation, but make sure there's a leader that's more in tune than I am to this particular subject and can over time gain the expertise and lead us to greater heights. And then we just make sure to rotate after a few quarters, depends on the subject. It is very likely that a leader on a certain operation will feel like they've done the best they could with it, and they'll move on to something else. And they'll find a new passionate person to hydrate the process and offer new ideas we didn't think about before.

Rick: That's a great cycle, finding leadership, finding that passion, getting the wisdom of the group and keeping it fresh and keeping it current with that kind of, you mentioned the council's individuals across the org I think I heard you say, joining and leaving as they wish, keeping that content fresh as opposed to solving the problem and putting that solution in place and leaving it for two years.

Yotam: Yeah. It's a great thing to remember. It doesn't matter how much experience you have. Someone started three months ago will have an idea that you haven't thought about in the last 15 years. And so I think experience brings some advantages, especially in knowing what questions to ask and how to listen to what's going on around you, but it doesn't necessarily give you agile solutions. And I think even the solutions need agility, some things great and true and perfect right now in three months, it won't be. And so I think that rotation and the open approach to those councils and the group mentality to building operational agility is a lot more useful than one or two folks deciding they know everything and dictating how every aspect of the work will go.

Rick: Yeah. You get the best of both worlds, making sure you're not ignoring experience and wisdom gained over lots of different experiences, but also those new individuals, three months in have a set of glasses that are very valuable because they only last for six, nine months. And then these new glasses are off.

Yotam: Yeah, for sure. I think you know when your experience is needed, when it makes a difference, you can feel it in the room. You'll ask a question and there's no answers. You see how the eyes are pointed at you or at some other leader who is asking or suggesting something. I think it's easy enough if your culture is good and safe to know when your input is needed and when to actually start with what you know, that too is okay. I'm certainly not suggesting that you should just forget everything you've learned in your career, but if you stay tuned to your team, I think you're able to identify some cases where you present an idea, a concept, even a detailed operation and the response is like, "Wow, we've never done that. Let's try it and see."

Yotam: And other times you'll offer your experience or your opinions or a concept, and immediately a very healthy discussion or argument will start. And then you can realize, "Okay, there's a lot of people here that might know a thing or two that I don't. Let's appoint some leaders, make sure they have the resources they need to work on this" and then step aside.

Rick: And it sounds like it's a great environment to have those discussions. And that trust almost sets the stage for success. With so many different departments and everybody learning both individually and at the departmental level, obviously the Agile Councils that you mentioned. What have been the biggest challenges you've experienced in implementing and seeing this grow?

Yotam: One thing, it takes time, it takes an investment. You have to be willing to really take the time to do the groundwork. You know, schedule interviews, decide what questions to ask, have some strategy on how you're taking in all those responses and turning them into decisions and action, and some written guidelines. I think it takes a certain commitment. I think some other teams or organizations may say, you know what, we're not going to invest in this. We're just going to do something and people will follow. And once in a while, maybe on a more improvised manner, they'll offer something and maybe we'll make a change. That too can work. But for me, for how I like to work is it requires a little more investment upfront to go through those journeys.

Yotam: You know, it could take, I don't know, three or four months, depending on what we do just to get to our draft of an MVP of an operation, which after that, we'll take a lot less that might surface for a year or two before it really needs an overall. But that initial investment in there is there, you need some experience leaders to do that. For me, I'm very, very lucky to have some incredible engineering directors that could really take those challenges and run with them and be able to multitask because that is a lot of multitasking.

Yotam: And so just taking the time, getting all the feedback in a way that's organized and thoughtful, turning that into action, reviewing the action, iterating a few times before that first draft is there. Once it's there sitting with the team separately and as a whole presenting, discussing, trying, and then running the first few build, measure and learn loops are quicker and more demanding. And you get feedback initially, a lot of it and you need to do it a little more often until things are stable. So I think it all boils down to just the commitment. Putting in the time and the effort, creating over time some template for this process of a journey towards operation so that the process itself is enjoyable and not burdensome. I'd say that is the main challenge.

Yotam: The other is, I mean, the price you pay for taking on lots of opinions is there's a lot of opinions. And so sometimes you have to make some decisions. You have to talk the group into trying one thing first. Sometimes when it's hard to say which of two options work better, you either allow each team to choose, which we often do and say, Hey, we can do things in three possible ways so far that we know. These three are all good in different situations, so pick your poison. But then other times it's messy. And so you make a decision and be like, Hey, let's just try this, see how it works for two, three weeks then just commit to looking at it after. And if it works, great. And if it doesn't, try the other option that you, you initially left on the shelf.

Rick: This is big stuff as I think about it kind of leading-edge stuff, as you apply elements of agility to operations, and the ask to the organization as a whole, as you mentioned, is a fairly big ask and may be a big change in culture because of the involvement of things that were previously mandated or given from the top, as far as this is how we work instead asking how should we work? One, was it a hill to climb of shifting that culture? It sounds like it only might've been a slight shift in your organization as it already sounds a bit enlightened if I could use that word, but have you seen the shift in the culture and maybe as a followup, is the shift in culture providing additional benefits?

Yotam: Yeah, that's a great question. I think those two are polar situations to be in both requiring that change in culture you're talking about. One is where everything's mandated from top, and the other where nothing is mandated and every team just does whatever works. And I think both of those, if such operational agility, if such approach is to work they both have to change culturally, almost meet in the middle. For Electric in particular, this was fairly painless. And I think it's just because the people were very motivated and very willing. The culture was already good as far as trying new things adopting to change, contributing to the process or the practice as a whole. And so, relative to maybe other chapters of my career with Electric, it was really painless and enjoyable. I think in other cases it could be tough, but then yes, there's a huge gain, as you said.

Yotam: Finding the balance, and it doesn't mean that my balance is the best one, but finding the balance between how much is centralized and dictated, how much is required from each team to follow the same MVP that we talked about, minimum viable agility versus how much is left for the teams to choose, the freedom for them to choose their tooling or to change the process, or to add a step as a required step, or to remove a step that they're not interested in, how much to allow for that freedom and versatility is a great question. Not one that I can answer for any engineering team on earth. I could just say, ask yourself the question, find the balance. And I think if even that answering those questions is some kind of a teamwork, some kind of a process of getting feedback, then you'll find the sweet spot.

Yotam: And I think it's surprising, like you'd think maybe more people would want more freedom and less definitions, but I think in practice, it's not always the case. It's often the opposite. Teams actually want the best practice to be defined ahead of time and have something to follow that is already thoughtful and not improvised or circumstantial. So yeah, I find that really a curious thing, but once you answer that questions as to how you're going to balance best practice with freedom, I think the culture changes for the positive and your operation is just a lot more efficient.

Rick: For those of us that haven't experimented with operational agility, kind of along the lines of finding your team or your organizational balance. Do you have any other advice for those that would be interested in trying this?

Yotam: Yes. My advice would be just like any other long-term effort, build a roadmap for it. We actually have a quarterly roadmap for our operation building. And so given it takes time that this process is long, especially for operations that are brand new and haven't been built before. We define our quarterly roadmap as if it was a product roadmap. We figure out exactly what our main roadmap items that we want to achieve for each operation in a given quarter. We are able to show the whole thing in one slide so it's easy to digest what it is that we're trying to accomplish from an operational standpoint, in a given quarter. And then as the quarter comes to a close, we iterate on that and build operational roadmap for the next quarter. And this will involve some line items for operations that we haven't had before. So brand new starting journey, and also some iterations from one that we've built and we just want to keep following, checking on how things are going and continuing the feedback loop.

Yotam: Another advice is think about what makes your team go, what are all the working pipelines that you have within your engineering organization and create those buckets first to figure out what operations you even need. And that too is not something that, one solution fits all. It really depends on the team. So take the time to do that. For us, it ended up with a nice list of sort of seven to 10 different operations that we wanted to build over time. And it's multiple quarters of investment to build all of them, but some of them I've already touched. I had Agile practice developer experience, which is the experience of each engineer on their local machine, coding, debugging deploying and so on. Observability was for us a crucial operation to define what does observability means for us? What are the values? How are we going to implement that per service? How are we going to measure it? What would be the processes around it?

Yotam: So I'm just putting these out there as an example for inspiration, but you really have to figure out what are those core things that define the way you work and are important to you. Architecture is a really interesting one because architecture is a technical thing. It's technology, it's a concept on how to build a platform and at every layer. And we could talk so much tech when we think about architecture, but there's also an operation to architecture. There are concepts there that are operational. How do you make architectural decisions? How do you document them? How do you vet them? How do you do cross-team projects that involve architectural decisions? How do you change architecture? What happens when you need migration? Like all these things, if you could sort of quickly think about you'll find that it requires an operation.

Yotam: So I think some teams might have an architect. Some teams might have an architectural council or a platform entity that's in charge. Other teams might say, no, we don't want any centralized architecture. We have each engineers and architects separately. Those answers vary, but at least for us to have an architectural operation was a huge thing because we want to make sure there's attention put into the future of our platform and that we are thoughtful in decisions we make, as far as platform design and the rollout of whatever decisions we make. And so we built an operation around that.

Yotam: You can build one for quality engineering or for release management. You could certainly think of things that I haven't thought about. You know, even your OKRs, the way you define them, right? Your goal-setting can be an operation. How do I do it quarterly? How do I do it sprint to sprint? How do I do it with individuals or with teams? This too is an operation. And so it's a separate thing to think about your actual objectives and key results from thinking about what is the operation around building, measuring and learning from OKRs.

Yotam: So write down your seven to 10 such buckets that you want to build operations to, and then find who within your engineering leadership team is interested and experienced. Or that's just curious and trying to build those operations and then set them out on the journey to do the groundwork, be there to support. A couple of quarters later you'll have an absolutely revolutionized team, I think, especially if it's something you haven't done before. If you are a small, maybe a small startup that just became not so small anymore, just never quite matured or never needed to mature up until now, think about every aspect, map it out to what operations make sense and just launch those journeys. It's really gratifying and it makes you incredibly more efficient. And I think happier too.

Rick: Happier, I am sure. That's a solid, actionable first step of just identifying those seven to 10 operations. I'm sure every leader can pull three off the top of their head right away and then investigate and find plenty of other opportunities. And it occurs to me, this is a really timely message with so many organizations having to shift to a remote culture if they have not had that experience before. And lots of studies popping up about the overall productivity and that decrease as so much of the stress of working at home and adjusting to a new cycle and a new way of doing things.

Rick: It seems like this is a fantastic solution to look at a new way of doing things. Now that I work from home 100%, I have to think about my dog and visitors coming in and out and different schedule for many people that might now have to teach their children at home. And this seems like a real timely and effective way to, well, how should we be doing this in light of new information? It seems like this would be a huge benefit to businesses and teams today.

Yotam: Yeah, I couldn't agree more. I was quite fortunate within this madness that we sort of started working on all this a good few months before this crisis began. And I think it's just a huge advantage if your team not only asked before, but is used to asking the question, what's the best way we currently know to do X? If that's part of your culture, then remote work becomes easier, both because you have a lot of those answers already formed in a thoughtful way and in a collaborative way. And also because you know how to change and how to bring further improvement.

Yotam: And so I think it's huge, especially from working from home, especially from a distributed team, remote team standpoint, it's huge to have a mature operation. I've been thinking about that quite a lot in these last two months that we are remote just to see how the team kept together, how strong the culture is and how clear our ways are. We know how we do things. We know how to communicate. We know how to make things better. We know how to create visibility to what isn't going well. And I think it's served us incredibly well. It's one of the reasons why I think as a company, definitely as an engineering team, in some way, we haven't missed a beat. Of course, it's been harder. It's hard for everyone out there, but altogether, I think we've operated really well during this time.

Rick: Yeah. A good message for engineering teams and not just engineering teams everywhere. Yeah. Like I mentioned, it's timely and it's something that I think everybody could benefit from, but whether or not the current situation continues for a month, six months, a year, if it goes away next week, those that start down this path, I think will see a benefit regardless. I know I'm taking notes as we talk through it. I personally, I've benefited a lot from hearing about your experience, your thoughts. Obviously, you've given a lot of thought to this and I am grateful that you took some time out of your day to kind of walk through how you're thinking about this, what this looks like for you and your org and sharing some of that wisdom. So thank you.

Yotam: My pleasure. It's been fun. I'm always happy to talk about this subject.