11. The Agony and Ecstasy of Maintaining Good Documentation

Stephen Barlow: Sure. Documentation, it's always valuable no matter what your product is. The type of product you have influences a great deal just how important that documentation is. A lot of end user products like iPhone other things like that you imagine that if you're making this product for anyone and everyone, and that you've made it well often times somebody can operate it without looking at the documentation ever because they open it up, they turn it on whatever it is, maybe it's a can opener, and it just sort of works because it does one thing, it does it really well, or the user interface is just generally pretty straight forward. There are some products that are sophisticated enough because of what they do that documentation is part of the product, because no one could be expected to use this product without reading something about it.

Stephen Barlow: That doesn't necessarily say something negative about the product, it just says something about the complexity of what the product is trying to do. The nature of some products is just that they need to have documentation to go with them, because no one could be expected to use the product without reading something about it. Heroku absolutely falls into the category of a product where the documentation is a crucial part of the experience. Certainly Heroku describes itself as an easy to get started with platform-as-a-service. Even the easiest platform as a service imaginable no developer could be expected to say, “Okay I'm going to get started on this platform”, and they immediately know how to deploy an application on it just because who knows how you deploy an application on any given platform until you read those three or four--at least--sentences that tell you exactly how to do it.

Stephen Barlow: Even in that simplest case for Heroku, that documentation is necessary, we sort of view all of our documentation as the product.

Charlie Gleason: Yeah I think from a design point of view, there's never anything more humbling or interesting than watching how someone uses a thing that you made. That's not even in a negative or in a critical way to think. Human beings are fascinating.

Stephen Barlow: Oh yeah, no, no it's-

Charlie Gleason: Fascinating ways of doing things.

Stephen Barlow: Yes, I've definitely in past roles I've run user tests on documentation sets that I've been responsible for and watched in horror as everything that I thought was obvious about how to navigate and find information you needed was not at all obvious. This is not a reflection of who was in the user test at all. Anyone who has run a user test knows that everything you think you know about how people use something is wrong.

Charlie Gleason: Yeah I like that. I mean if you put me in front of let's say a different operating system, I feel like the compass just melts.

Stephen Barlow: Oh sure.

Charlie Gleason: A big part of our industry generally is that idea of googling things and knowing what to google for. I think that even within areas you would assume an extreme level of knowledge or kind of automatic knowledge, that documentation is still so important because the I nternet is very big. You ultimately can't keep it all in your head anymore. I think I mean I guess the flip side of that is the impact of bad documentation or incomplete or outdated documentation.

Stephen Barlow: Right. An adage within technical writing circles is often that incorrect documentation can be worse than no documentation, depending on how incorrect we're talking. If the instructions for something are so wrong or outdated that they tell the user to do something they can't do, or they tell the user to do something that is actively detrimental, then absolutely, that documentation should just be deleted. To simply have a vacuum that confuses the user because they don't know what to do is better than having a situation where they think they know what to do and do something bad.

Charlie Gleason: I think one of the environments that I've noticed that most is probably open source software, purely because it's often someone working in their personal time. If you are writing an API, it's very easy for you to see the blueprint of that in your head obviously because you wrote it, you wrote the API. I think as well it's hard to find people who are naturally good an enthusiastic about writing documentation. At least, I feel like I've noticed more and more in open source software that people are ... If you really want to get into open source software, help with the docs. It feels like there's such an opportunity there for people to become more involved in that scene just purely by virtue of how hard it is to find people who are willing or have the time or have the energy or have the inclination to really clearly and succinctly document things, and how valuable that is for any kind of software project, right?

Stephen Barlow: Honestly, that's in the open source community I believe recommended as maybe a first pull request you could send on a project is simply to improve the readme or the wiki pages on that GitHub project just because it shows one that you are interested in working on the project, two it's easier to maybe correct or improve English text than it is to improve source code if you're just getting started with the project because maybe you need to familiarize yourself with it. A big part of familiarizing yourself with it is onboarding by reading that documentation in the first place.

Charlie Gleason: Sure.

Stephen Barlow: Some projects have wonderful documentation. I think documentation for open source projects across the board would be all the more wonderful if as every person is coming up to speed by reading what they need to, they submit at least one pull request that improves what they just read.

Charlie Gleason: What are some of the common challenges in documentation coming out of the value of why it's good, the impact of when it's bad? What are the challenges in creating and maintaining good documentation?

Stephen Barlow: Yeah there's a few of these that you hear time and again if you talk to a tech writer at pretty much any organization, or the user of any documentation. I would say that the most frequent of all of them is simply not knowing whether or not a particular document is up to date. Even if you do know whether it's up to date, often times you know definitively that it isn't. Keeping documentation fresh is a real challenge that has no easy answer. I've seen many different strategies for maintaining it in different organizations. I think it's important to employ strategies no matter what they happen to be. I also think it's important to acknowledge to yourself you're never going to keep every single document in your collection perfectly up to date no matter how hard you try, just because there's always going to be more to document than you have people to fully document it.

Stephen Barlow: There's always going to be a priority within the organization about which documentation serves the largest percentage of your users. Those documents are just naturally going to get more love.

Charlie Gleason: Yeah, and I think that's a challenge as well when you move further out into like Heroku has a set number of natively supported languages, but then we open up buildpacks as a way to kind of access other languages. I've been doing a lot of Rust stuff lately. I was looking at the documentation for that, it just doesn't have the level of potentially interest maybe or comparative to something like let's say create-react-app in Node, which is I think one of the most starred projects in GitHub. I guess by virtue of the number of people who are involved in a project, and I'm not in any way comparing a single build pack to one of the largest GitHub, most popular GitHub repos by any stretch. I just think it's an interesting thing to try and get enough of a critical mass on a project to get that kind of passion or enthusiasm or even just sheer numbers of people using it and commenting on it and reacting to it and trying to make it better or feeling like they're a part of it.

Charlie Gleason: I think documentation is a really good way to do that, but it's like, “How do you get people to write it?”

Stephen Barlow: The challenge of actually getting people to write documentation is much less of a challenge if you're in an organization that has a dedicated team of technical writers, because their whole job is to actually write it. This one is more about situations where you're either talking about open source as we just described. We can ultimately talk about the internal docs certainly where your engineering department has written a collection of services that are serving the internal customers of your employees, and you've gotten those services up and running. Now the question is, do you have a readme in the repo for each of those internal apps to describe how they work, what the navigation of the code is and all of this which would help to onboard of course a new engineer on the project, and also probably help an SRE team to understand how that project works in the event that something about it happens to break while they're on call?

Stephen Barlow: There is a talk that I saw at a conference a few years ago, the conference was called Write the Docs. It was a talk by a senior technical writer at another organization who created a pipeline to get engineers to write docs, which after realizing that asking people to write documentation as a completely separate part of their job that had nothing to do with their existing work flow, it was having low returns. Adding something to somebody's task list that's orthogonal to what they're usually doing just has an innate resistance to it, I mean for anyone. I would for sure. What they did was they sort of added some sort of pre ... Part of reviewing code, part of the core cycle of reviewing code included a step that said, “Hey, this change to this code base is larger than X. There are no changes to any readme files in this repo. I'm spitting out this warning.”

Stephen Barlow: “We will let you merge this, but you're violating what is now a policy at this organization, which is because you've added this much or changed this many lines of code, you've probably changed something pretty significant about what this app does or how it works. Please update at least one Markdown file in this repository.”

Charlie Gleason: I love it because I always leave the oven on. It's really frustrating to keep doing it. I did it this afternoon, and last time I left the stove on as well. I feel like for some reason, I'm not sure how that's going to help with making commits, but I feel like if there was some way I could be warned that it's like, “Charlie, you've had the oven on for like four hours now, and I don't think you're cooking anything anymore” similarly would be great. I feel like it's awesome that our tools are starting to prompt these conversations. In a lot of ways, documentation is as much about that human experience of making things accessible and making things understandable and making things functional as anything. I don't know, there's something very cool about those kind of tools stepping into help along the way.

Stephen Barlow: Maybe we should get you an Internet connected stove that can send you an email with its current temperature every 30 minutes.

Charlie Gleason: I would not say no. Yeah I would not say no. Another common challenge I think that we've come across you and I have discussed in the past is finding things, the findability of documentation. Do you have any thoughts on that one?

Stephen Barlow: Yes. This is I'd say mostly comes up internally at organizations because almost anything external is findable because of Google. Now, even then sometimes it's important to maintain SEO and make sure that a document that is public facing uses the right terms I guess to make sure that if people are searching for it, that they can find it. Findability is most challenging. I hear the most complaints about it internally because inevitably when your organization reaches a certain size, you're putting documents into 50 different places depending on which team you're on and what you're doing it for. When I first joined a previous company, one of the first things I was asked to do was develop the guidelines for where to put documents moving forward. The idea I think initially was come up with the canonical, single place to put documents moving forward. We had a Wiki instance, and we had Google Docs, and we had Markdown files in GitHub.

Stephen Barlow: People were saying, “Well I don't know where this team documents this, so I go searching in the Wiki and I don't find it. Google Drive is harder to search than the Wiki. GitHub, I don't even know which repo this would be in. It's just a mess.” I interviewed several people at the company to try and suss out, “Okay, let's say we switched to putting everything in one place. Where should we put it?” I would interview people on one team and they would say, “Oh I have to put things in Google Docs because I have real time collaboration. My whole job is around working together with people in real time.” I said, “Oh that's a really good reason to use Google Docs.” Then another person said, “Well I put everything in the Wiki because I need people to be able ... the Wiki has the best search capability of anything that we have. I want people to be able to find, search for this.” I said, “Oh that's another good reason. Okay well if I'm doing this whole project because of findability, maybe the Wiki is the answer.”

Stephen Barlow: Then I would talk to the SRE team, and they'd say, “Absolutely not. We're not putting our run books on the Wiki.” I never worked directly with an SRE team before.” I said, “That's a very strong opinion. Why?” I honestly, I didn't understand why it would be such a problem. They said, “Well in the event of an outage, there's nothing to say the Wiki isn't going to be out.”

Charlie Gleason: Right. A run book is for anyone that might not-

Stephen Barlow: Right. A run book is a set of recipes for a given service that your company maintains that explains how to do common tasks for it, whether that's deploy it, bring it down, upgrade it.

Charlie Gleason: The interesting--

Stephen Barlow: Right, run a migration, things that an SRE team that doesn't directly maintain the code but is responsible for maintaining uptime across the entire company could consult in order to better understand how to resolve any issues that arise for that service. In the event of a huge outage for the company like a severe event like that, who is to say that the Wiki isn't also down? They kept everything in Markdown in GitHub--well specifically in repos that they regularly sync so that the documentation is replicated across every SRE laptop. These are great reasons to have documentation in different places. It does reduce findability because you have to look in X different places instead of one. Those use cases are so real that in the end we didn't say, “Stop using X, Y or Z.” We said, “Yeah use this if you need this. Use this if you need this, et cetera.”

Charlie Gleason: Which makes sense right? I mean also the best tools are the ones that people are using and by the sound of it, it seems infeasible to find something that can do everything for everyone. That's kind of almost fail safe of like having things across different platforms and services gives more flexibility almost. I don't know if that's the right word, but yeah.

Stephen Barlow: I will say for anyone that's listening that's like, “Well that doesn't answer the question. I need help in my organization”, I will say this. Here is one example of something we did. I created a page on our Wiki called The Registry. The Registry was a list of every service that ran internally at the organization that links out to the individual GitHub repos for those services. The Registry wasn't always 100% up to date, but as one example, at least this allowed somebody to quickly search the Wiki for the name of a service, find its registry page, find out where its repo lived, who maintains it according to the Wiki, and who to contact in the event of an outage, which at least made it more findable than N different repos on GitHub. Sometimes having a hub documentation source like your Wiki that links out to everything else at least creates like a canonical place to start. If you can't find something there, then you can start your breadth first search.

Charlie Gleason: Okay so that's some common challenges, and some ways of overcoming them or at least better understanding them. Do you have a philosophy for how you approach what you document, what you don't document, how you express changes to documentation, all those kind of things in your day to day life?

Stephen Barlow: Yeah totally. Our focus is on reflecting the platform and what it does. Historically, we haven't focused as much on documenting, “Okay, here's how to take all the different parts of the platform and put them together.” We've more recently put out some articles about mixing and matching components of the platform to achieve a certain architecture. Historically, Dev Center has been much more of a site that says, “Here is everything that the platform does. Here are all the components of it. Here is what dynos are. Here is what a stack is”, and allowing you to understand each of those components in isolation, except of course for getting started tutorials which hopefully walk you through enough of the platform landscape to actually do everything you need to do to get an app up. Our emphasis is always on the reference component of Dev Center.

Stephen Barlow: It's only more recently that we're expanding more into combining, mixing and matching. That's mostly just because the focus is always on first be the canonical truth of the platform, and then second, be the application of the platform. We also maintain something called Changelog. Changelog is hosted on Dev Center. It is a sequence of short posts that reflect public facing changes to the Heroku platform whether that means updating the versions of Node that we support, whether that means a new version of Postgres, a new feature going live always has at least one Changelog entry associated with it, or a deprecation of a particular version of X, Y or Z. We treat Changelog entries as moments in time. The only reason we would edit a Changelog entry after the fact somewhere down the line is we find a typo in it, occasionally to fix a broken link, that because something else changed. Usually, we don't even fix those because we want Changelog to reflect what happened then, and not so much something to repeatedly consult.

Charlie Gleason: Sure. It's like a history book.

Stephen Barlow: Yeah, exactly so.

Charlie Gleason: I don't know if that's the best analogy, but anyway, I'm going to go with that.

Stephen Barlow: Yeah, yeah. It's nice that you can sort of scroll through it if you haven't before just to get an idea of like the cadence of how we change things and what we change, and maybe even get an idea for how quickly we'll update to a new version of a programming language after that version became generally available, the largely accepted new standard, which spoiler alert is fast.

Charlie Gleason: One thing I think that could potentially be interesting as well is what we don't consider important to document, and what the rationale behind that would be.

Stephen Barlow: As I mentioned, we always do prioritize reflecting the truth of the platform first. We don't make a conscious choice to avoid documenting how to put things together. As I said, we're doing that more and more. Reference is the most important thing to get right and to have available. We always focus on that first. In terms of what we don't document, Heroku considers itself to be a very easy platform to get started with in the Cloud platform as a service space. We pride ourselves on that. We pride ourselves on being certainly the first platform that a lot of people get started using at all. Despite that, we don't consider ourselves responsible for teaching people how to code or program or use for example Git. Most of Heroku's deployment methods use Git. If you aren't at least a little familiar with Git, then a lot of Heroku's potential is lost.

Stephen Barlow: We don't focus on teaching those concepts mostly because there are a lot of great resources out there already to teach you version control, to teach you any number of languages that we support. We make an assumption in order just to limit our own scope of responsibility and keep our heads above water in terms of what we write. We make assumptions about who you are reading our docs. We assume that you know what Git is. We certainly direct you to install Git in a lot of our getting started tutorials in the case that you don't have it installed. We assume that you know what that is. We assume that you are capable of programming in at least one of our supported languages. We're not in the business of teaching you Ruby. We're in the business of helping you get your Ruby app that you know how to write into the Cloud with as little friction as possible.

Charlie Gleason: Absolutely. I think there are like you said, there are really good resources. If you are getting started, I know that in the past, you've always recommended that people look at tools like Stack Overflow, which has a dedicated Heroku section or tag I think it is or a label which can help. Certainly, when I was starting out, I think that checking the logs to understand why that isn't working feels infinitely frustrating, but also like wonderful when you get to the bottom of the problem. I think I definitely hear you on there being resources that have just so many more people that can really dedicate their time to helping you learn as well. I think that makes sense.

Stephen Barlow: Yeah. I think it's not our core competency. There are so many people out there that are doing such a good job, that we just sort of say, “All right, you all got this.” A lot of those, actually a lot of coding schools and stuff when they get to the part of, “Okay, you've learned how to code. You've learned what these variables do. You've learned how to structure an app. Okay now it's time to get it in the Cloud”, a lot of these coding things use Heroku as the first platform to get them deployed to again because it's pretty straight forward, and because it's so closely aligned with Git, which is probably the version control system they yse. We happen to be part of curricula, is that the plural-

Charlie Gleason: I don't know.

Stephen Barlow: ... that teach coding, even though we ourselves don't really do it.

Charlie Gleason: I guess I'd be curious if there's anything interesting that you think about how we manage our documentation, or how we do things that might be different or people find interesting as part of your role.

Stephen Barlow: Yeah. Heroku does one thing very differently from anywhere else I've worked in terms of its public facing documentation, which is that well I've worked for a lot of big companies. Heroku is a part of Salesforce, which is another big company. Heroku itself is pretty small. Heroku is the first company I've worked for where I as a Dev Center person I own, own meaning am the primary person responsible for a few articles on Dev Center, but not that many. We don't have a team of technical writers dedicated to writing all the stuff that's on Dev Center. Instead, our product managers and our language owners each of those folks owns ... The language owners each own the articles that pertain to their language, and the product managers all own the documentation that pertains to their product service area. Again, yeah we don't have dedicated technical writers, and instead, we have a process where they're all very good writers.

Stephen Barlow: In the lead up to a launch of a new product or the release of a change to one of our languages, if the accompanying documentation changes for that update are sufficiently complex, I simply act as some sort of copy editor and structural editor to changes that are being made by somebody who is closer to the subject matter expert than they are to a dedicated writer. This model has certain benefits and certain drawbacks. I'm not saying that it's perfect, and that teams should switch to this ownership model, but it is an interesting change of pace. I think it's pretty cool for a product that has a pretty reasonable total documentation service area. Salesforce proper has one might argue an unreasonably large product service area in that Salesforce offers everything under the sun. Of course, it needs a team of technical writers.

Stephen Barlow: Heroku as an individual product is a complex product, but it has a pretty well defined scope of what it does, and as such it can more reasonably be documented by the people who own the components of the platform. It doesn't need to be distributed to an army of writers instead.

Charlie Gleason: Yeah absolutely. I must admit, when I joined Heroku I was surprised. It is not like enormous, which I kind of-

Stephen Barlow: Yeah isn't that weird?

Charlie Gleason: Yeah it's super weird. I think one thing that I've noticed personally is that even when I joined, I was encouraged to make changes if I saw something, or to suggest or to be involved in our documentation if we see something to change. I know that you and I have talked about it in the past that that results in maybe deviations in style or in voice.

Stephen Barlow: Yeah so you're absolutely right. Any Heroku employee who wants access to edit Dev Center immediately gets it, and is welcome to make changes. We sort of assume that nobody is going to go in there and just delete all the text from an article, and even if they did, we have a back up. It's not like that would be the end of the world. Again, updated documentation I would say is much more delightful to a reader than perfectly consistent styled documentation, like, “Oh my God, this sentence is so exactly the same in style and tone as the sentence before it. That totally makes up for the fact that it's wrong.” No, that's not true. Have that sentence be a completely different style of sentence that the other writer would never have written, but at least have it give you the right information. That's why we sort of have very democratic, like if you see something, edit something sort of style.

Charlie Gleason: Well I think that's pretty much everything. I mean thank you for taking the time to chat. This sounds quite forward, but you are one of my favorite human beings that I work with so.

Stephen Barlow: Well we have to maintain some air or professionalism in this podcast environment.

Charlie Gleason: I know. Secretly outside of it though, it's all jellybeans and parties. I guess one way I thought would be a good way to finish up would be maybe some piece of advice that you would want to impart to our dear listeners before we wrap it up.

Stephen Barlow: Yeah sure. I would say as with any change to an organization, if you're thinking about getting more rigorous documentation processes in your organization, change is resisted in all aspects of life, right? It's hard to change humans. As much as you can, if you want documentation to be more rigorously practiced, the most effective way you can do it is to build it into the workflow that already exists for the people you want to do the documenting. As much as you can get those two things to align, I think you have the best odds of actually getting people to do it. I would also say like whenever you have a new employee, a new employee is an unbelievable resource for documentation because they have to get up to speed, they're going to be reading the documentation anyway. You might as well formalize their onboarding to include a part where they basically have to--because it's now their job--to write down and note anything that they experienced during their onboarding for reading the product documentation for the product they're now helping build that was wrong, where they could have been more clear.

Stephen Barlow: They're coming at it with fresh eyes, and you can make them do it.

Charlie Gleason: That's the secret. I like that it comes back to the common challenge of actually getting people to write is to like make it part of the job.

Stephen Barlow: Yeah there you go, right? Fold it on in there. People won't even realize they're doing it.

Charlie Gleason: All right, well again thank you so much for your time. Dear listeners, you've been listening to Code[ish]. I'm Charlie Gleason. My guest has been Stephen Barlow. Thank you for joining me.

Stephen Barlow: Thank you for having me.