21. Building APIs that Integrators Want To Use

Idan: What were those tools that helped make them successful?

Matte: Some of it is operational in nature. Some of it is just the system in general. The whole add-on system was essentially a set of APIs and webhook systems to enable to web services to developers' tools to interact with one another. Part of it was making sure that experience of interacting with the APIs and the tools and the things we've built for partners were easy and the best possible thing for them. The other was making sure they had enough data and information and insight into the actual operational aspects of things. When installations go wrong or provisioning goes wrong or webhooks aren't delivered, making sure they have the tools to figure that out and debug quickly and course correct.

Idan: How has that translated into the work that you're doing at Sentry now? You're still on the ecosystem team there just like you were at Heroku.

Matte: Yes, but I lead the ecosystem engineering team here. The overarching goal here is the idea that developer tools work better when they work together and developers use many tools. The goal is to create systems and platforms that let the folks who are developing those different developer tools integrate with one another so that developers have an easier time doing the things they need to do and that their workflows just fit into one another regardless of what tool they're using at any given moment.

Idan: It's like it's stitching together these different tools that developers like to use. What does that look like from Sentry's point of view? We haven't even actually touched on what Sentry does overall.

Matte: Sentry overall tracks information, mainly I'd say about errors and events that happen in your software so that you can figure out what's going wrong and get to a resolution as quickly as possible. That's one piece of the overall what a developer does, and so what it looks like for us is taking that information or those concepts and figuring out how it fits into other workflows.

Matte: A good example of that for us specifically is most engineering teams use project management tool. Fixing issues and fixing bugs and making improvements is all part of how you plan for an engineering team. Making it really easy to go from Sentry when you notice an error or an exception or something, fitting it into your backlog through an integration from Sentry to whatever project management tool you're using is really valuable because then previously, it's this process where you have to go from one tool. You have to interpret it. You have to figure out what data makes sense to go into the other tool. Somebody actually has to manage that. It's just a manual process that takes more effort than it should.

Matte: Instead of doing that, the two systems can be integrated in a way where we generally know what information you want from an error, in a story, in your project management tool. If we make it really easy for you to just click a button or select a few things and sync that data between the two systems, it cuts out a bunch of effort that's really unnecessary a lot of the time, but still enables you and your workflow in a way that's useful.

Idan: Nice. If I have some error report that shows up in Sentry that surfaced from my front end or my backend, then where am I going to be copying that data over? It's creating issues and GitHub for those things, that thing or-

Matte: Yeah. Going along with the project management idea or whatever. Say you're using something like Clubhouse or whatever. You're going to go from Sentry, you're going to pull information about the exception, the error class that popped up the message. Sentry exposes a whole bunch of information about it and expose breadcrumbs, which are essentially the steps that your user took or the things that happened that led up to this. There's a whole bunch of context and information about the error that you would have to go from Sentry, copy and paste it into a story in Clubhouse in whatever way you think makes sense.

Matte: The other side of it too is the synchronization of data between the two systems. Imagine you go into your code base, you fix a problem, you deploy it, you resolve something in Sentry, you resolve the error because it's no longer an issue. You also have to go into these other systems like Clubhouse in this example and move that story along into in progress and then done and then deployed or whatever. Cutting out all that manual work is one of the benefits of a system like this.

Idan: You touched a little bit about the best practices that you picked up in your time at Heroku on the ecosystem team here that's going into this new work that you're doing to automate the kinds of workflows that developers are looking to accomplish, centered around their issue tracking and their error reporting and all of that jazz.

Matte: I think there are two categories of things I learned from Heroku. I think one is how you approach this problem or maybe philosophically is a better way of describing it. The other is a technical set of things that I learned. The first philosophical or how you look at a system like this or a team like this or whatever, one of the things that became really, really clear pretty early on is that a team like this has two customers. They have the end user of your product, and then they have the set of developers or companies who are building the integrations for this end user. They're two different groups of people. They're incentivized two different ways and they want two different things often. Sometimes they overlap and that's really great.

Matte: One of the things I learned that I think is really important when you're doing ecosystem integration work is you need to actually view these as two different groups and you need to really understand them individually. That was the nice thing about how Heroku's ecosystem team was structured, was there was someone like me who was maybe more heavily geared towards the person building these tools. I was more on that end of the spectrum. I was more involved in the partner relationship side of things from an engineering perspective. We had other folks who were maybe a little more focused on the end user side of it because all these features that we were building for partners and all the features partners were building are ultimately for people, in this case, using Heroku in this example. Having people who focus on those two sides of it but are also on the same team or work really well together is really, really important. That was one thing.

Matte: The other philosophical thing that I learned and got a greater appreciation for at Heroku was this idea that an API is a user interface in a lot of ways or an interface at least just like a UI, like a visual UI is. One of the things I really, really loved about Heroku was how intentional they were and are about designing APIs and the use of that API, and it's not just necessarily a way to enable a front-end application. I think Heroku always did a good job of thinking about it as an interface to the primitives of their system. I think that's really important. It's really useful for a whole lot of reasons, but it also makes it easy to reason with a system like this when it's done in a very intentional way. I'm trying to pull that as much as possible into everything I do.

Idan: What do great API identity, the I in API is for an interface, and like you pointed out, it's an interface like any other, whether or not it's a seal higher, something visual. What are the hallmarks? A lot of people talk about what makes for great user interfaces and great user experiences using those interfaces. What do good experiences look like? What do great APIs look like? What are the qualities that really matter to have there in order to reach that level of polish and cohesiveness? What in your experience has been the things that are hallmarks of goodness there?

Matte: I think broadly speaking, the API gives you a general sense of the system as a whole and all of the entities within that system, how they're related to one another and what those relationships mean. That's a hallmark of a good API to me. A lot of it I think is just reasoning with it. You expose resources in a way where you can understand what each of them mean in isolation but then also understand how they interact with one another. That's really important. Tangibly, I think what that means is ... To go the opposite way. When you're designing an API to accommodate a single consumer, you do things like including data about related resources within the main payload of a different resource because in the frontend, it's easier to just ask for all the data in one giant blob. You end up mashing data together, and it blurs the lines of what any one individual resource is and what it means within the system.

Matte: In my opinion, keeping data concise around individual resources and then expressing the relationships between those individual resources where maybe the representation of any individual resource is small and you need to pull a couple of different resources to get the big picture. I think that's good and that's okay because it's the common view of the primitives of the system where if one consumer needs things in a certain way, they can pull the data they need and manipulate it in one way. A different consumer needs it differently. They can do the same thing, but each of those consumers has a common understanding of the makeup of the system from a primitive point of view.

Idan: In this case, you're talking about an API is going to power an ecosystem. Like you said, it's like a market with two faces. We have the end users which we're all serving together in the end, and then there are the partners, the people that are building the add-ons for this ecosystem or the plugins or whatever the ecosystem is an ecosystem of. The end users are probably not consumers of this API. Who is consuming this API? Is it the partners and they're using this API in order to do the logistical work of provisioning and de-provisioning these things and interacting with billing control surfaces and things like that or are there other audiences for these APIs that you're building?

Matte: Yeah. There are actually. I think when we started going down this path, we thought of this from a perspective of there are companies and people out there who are building developer tools who want to use Sentry data or want to push data into Sentry or use something in an interesting way for end users. In that case, it's not necessarily the end user, but it's this intermediary party that wants to do something to enable those end users. We started with that idea, and a lot of ecosystems and stuff like this is based around that idea. Most marketplaces or add-on systems or whatever are based on that idea that there are three parties. There's you, the company building the platform. There's the partner building the thing to enable people. There's the people being enabled who are end users. That's a big part of it for us.

Matte: In that regard, to what you said about the billing aspect of it, we're very, very focused on the experience side of things. Currently, there's no billing aspect necessarily to our integration platform, which is one difference between I guess Heroku's ecosystem and our current ecosystem, is that there definitely is a billing component obviously through Heroku system. You can purchase the use of other developer tools through Heroku, which means that the provisioning and installation process is slightly different in that case than it is for us because in that case, a lot of the time, it's the database provider or some other tool or whatever actually spinning up infrastructure. In our case, a lot of the time, we assume that you already have an account in the other tool that you want to use. The general idea is we assume you already have a bunch of tools that you're using. We want to connect all those tools for you.

Idan: It's more like sign in with whatever.

Matte: Yeah. It's not quite just sign in with whatever, but it's when you install the installation, implementation wise or whatever, you get redirected to that service. You sign in over there. They do whatever they need to do to take the data that Sentry sends them about this installation to associate it with the person's account on their end. They can send data back to Sentry to do the same thing. It's less about they are spinning up an account or they are spinning up infrastructure or whatever. It's just we have these two accounts and these two things that we want to connect together. We are pretty much solely focused on just the experience of using those tools together. That's the third party aspect of this platform, is what I call it. Third party, meaning there's a developer building something for the end user.

Matte: What we found along the way of doing this when we were talking to people and getting feedback, we learned that there's a lot of teams and organizations that actually want to use a lot of the same functionality but for internal tools, which was an interesting thing to learn. Now we're back to this situation where we have two sets of customers. I guess in reality, it's three though. We have developers wanting to build integrations for our end users. We have the end users who want to consume these things. Now we have this third party that is developers and users of Sentry that also want to use this functionality but for internal tools.

Idan: These users are basically, they're playing both sides of the table in the sense that they are also the provider role in the sense that they're the ones providing these integrations with their own toolings, with their own custom in-house setups. They're coming to you to build these integrations for themselves basically.

Matte: Yeah, exactly. They may have some tool they built in-house that they want to integrate with that's private to them. They want the same functionality to connect two developer tools or two web services together, but they're not necessarily looking to enable other people to use this thing, or they use some tool that's public but maybe just not popular or big enough to warrant somebody building an integration for it, and so they want to just take it on themselves to build it. In that case, they are a hybrid of both of the other parties.

Idan: I'm curious a little bit about what you do to build the right set of interfaces, the right set of hooks across all these different kinds of integrations. I'm sure you have integrations that, okay, a lot of them maybe have to do with workflow, but there's still a lot of different tools out there for a lot of different purposes and a lot of different workflows. How do you go about figuring out what's the common denominator there? How do you build for that diversity of tooling and afford them the buttons and levers that they need in order to make their thing work?

Matte: The approach we take is essentially listening to people. We do a bunch of outreach. We try to talk to customers. We talk to interested developers at other tools that think there's some value in integrating with Sentry. I think so far, we've taken a very use case approach to it where we listen to all these sets of people and what they want and what they need and think they need and try to derive what we enabled through that. Because you're right, there's a million versions of this and a million different types of integrations and things you can build, and choosing which ones to do out of thin air is really hard. I think we're trying as much as possible to take the approach of let's try to find evidence that there is something useful here. Most products or companies or anything, you can do that by listening to the people who are using it and trying to understand what they're really trying to accomplish. That's what we've been doing so far.

Idan: Is that also, like when you talk with users and there's the old saying, the plural of anecdotes is data. When you're talking with customers, it's this not entirely subjective, you are collecting data points, but do you also find yourself relying on more numerical, statistical, whatever, metrics that you're collecting? Which hard numbers do you find yourself relying on, or is it mostly about the points and issues that customers are raising? Maybe dig a little bit deeper into how that process looks like at Sentry.

Matte: Yes. There's definitely some hard number precedents we have. For context, there's been some notion of integrations or we called them plugins previously. That's been around for a while in Sentry. because Sentry is open source, the way that looked previously was a repository with a bunch of code that gets imported into Sentry and then used. It's actually like code that got pulled into the Sentry codebase. Because that's been around for a while, a lot of people use that. We have some hard numbers generally about what kinds of tools and stuff people use. We definitely look to that for some amount of precedents essentially.

Matte: I think our approach is basically use whatever the data we have mixed with what we're hearing from people. The data that we have is what people have used in the past, what tools they've ... What integrations or plugins or whatever that they've used in the past within Sentry, and then mix that with what they're actually asking for.

Idan: What other interesting technical challenges do you see in building these ecosystems or business challenges order of operations challenges? There's a lot that goes into making these successful products. While a lot of it is indeed technical, not any less important is the softer skill of determining what it is that customers actually want from listening to what they are saying because customers aren't always able to articulate those requests in sufficiently higher resolution. When you do have that customer then great. You're set, and you have lots of valuable feedback on a silver platter. When you don't, it's on you to take all these different and sometimes contradictory asks and bake it down into we should do A, B, C, D. Can you maybe share a story or some other interesting technical challenge that you've worked on while building this?

Matte: Sure. Because our integration platform assumes a lot of the time that you have an account and an organization or whatever set up in both developer tools, it introduces this challenge where if you don't or something in that process of connecting those two accounts goes wrong, it's harder for one side to know about that. In the case of ... Back in the day before async provisioning in Heroku, Heroku sent a request to the provider, the provider did a bunch of stuff in line and then responded with a success or a failure. It was pretty obvious when something went wrong on either side because either you got 500 back or some error status or whatever that told you something went wrong or just timed out or whatever. That's pretty obvious. Both sides know something went wrong. Heroku had gotten an error back. They know it didn't work. They can fail accordingly, show the customer whatever they need to.

Matte: When there's this process where you expect something to be set up on both sides and the process involves installing something on Sentry, for example, being redirected to the other service to do configuration or whatever you need to do. If the user gets redirected and then something happens and something goes wrong on the other side and it's out of the scope of the original request, Sentry doesn't know that that happened. I think for us, the eventual consistency thing is a little trickier and more important in this case because we can end up with a bunch of data on the other side that's just out of sync. That's an interesting challenge in contrast to what it was at Heroku a lot of the time.

Matte: I think another one for us specifically is scale, and not necessarily like us scaling. The hard part is Sentry has gotten very, very good at ingesting a lot of data because that's what we do. Exceptions and errors happen all the time, so we've gotten very good at that. If we're building a system that exposes those events and that data in some way, it means that potentially our partners in the integrated services need to also be able to ingest data at the same rate or close to it or whatever as we are. That's not always the case because not every product is an infrastructure scale product or doesn't need to be scaled that way. Part of the challenge is figuring that out and working within the bounds of all these other integrations and services and products and companies to make that work without necessarily taking down other services.

Matte: In some cases, it's great because you work with other developer tools that are also in that same realm, and so they're already set up to ingest that much data. A lot of the time, they're just not. The product just inherently is not about that. One of our challenges is how do we build a system that still gets them all the data they need to function but can do it in a way that is scalable for both parties? It touches on another challenge in all ecosystem work, it's not just you. It's not just your engineering team. It's not just your product team. It's all of that plus entirely separate entity that you don't have any control over, but you need to work just as collaboratively with as you do your internal teams.

Idan: Another question here is if you're working with these teams, some of them might be huge companies. Some of them might be mom and pop like two developers in a garage kind of situations. How do you make sure that they stay successful when you are suddenly pointing a fire hose of traffic and the associated revenue that comes with an exposure that comes from, hey, we featured your add-on or page. Now you need to be able to handle X signups per minute or whatever it is that we're sending your way because we've made your integration the default for this category or whatever. How do you help those engineers and those teams weather that process and come out the other side still looking good to their customers and not impinging upon Sentry's brand?

Matte: I think it's like scaling anything to be totally honest. I think if you think about it, they're another team within your organization. It helps. All the same technical things apply, I think, like gradual roll outs, putting together the operational plan when you're talking about the project before you've ever launched it or deployed it is really important. I think also trying to make sure both parties have a good understanding of at least the hypothetical load based on whatever knowledge you know about your independent systems is really important. Honestly, I think a lot of it is just being prepared for the fact that you might need to figure stuff out as you go. Like anything in software, there's some amount of learning when you're doing the thing that is inevitable.

Matte: I think just trying to be as prepared upfront, communicative with your partners throughout the process. While we were launching a bunch of stuff, for example, we do weekly meetings with our partners to check in and see what challenges they were having, challenges we were having, trying to figure out how to solve those problems, and just generally making sure we were on the same page about everything essentially. I think it's just being in contact with them all the time and mentally thinking about them. At least for the engineering side of things, thinking about them like another engineering team in your organization.

Idan: We're almost out of time. I wanted to ask you one final question, which is, you've really been around the block when it comes to building these these two-party marketplaces or end-party marketplaces for established platforms like Heroku and like Sentry. There are of course a lot of other businesses out there that are doing this thing. If we have some developer or some product person who's listening right now, what is a core bit of wisdom you'd want to impart to people who are working on this class of problems? What should they be thinking about process, tooling, anything? Share some wisdom there.

Matte: I think the point we talked about earlier where you have essentially different customers and they need and want different things. I think that's really, really important to acknowledge and to explicitly think about.

Idan: Does that mean changing in staffing? Does it mean changing go to market strategy? Maybe spell that out a little bit more.

Matte: Sure. I think in my mind right now, I'm thinking from a product perspective. The things you would build for a developer building an integration that they want a million end users to use is probably a different product than what you would build for that end user to consume that integration. A lot of that comes down to what you're offering partners and users. For example, if your ecosystem has this revenue component to it like Heroku does, then I think it's really important that you build tools to allow your partners to understand the revenue they're making through your ecosystem, and even more so, gives them tools to optimize that. In that case, money and revenue is a big motivating factor. For that party, you should be focusing on the things that are going to allow them to understand how to optimize the revenue side of their integration while also optimizing the experience side of it.

Matte: You need to mesh those two things together as much as possible because they go hand in hand. If a user has a good experience, they're going to want to use the tool more. If they use the tool more, the developer is going to make more money.

Matte: Product-wise, I think it's really important to think that way. The things you build ended up being differently. I also think it's just good generally to think about that mentally because how you interpret things that those two different groups say might be different based on that knowledge. That's the one thing I think is super, super important that I don't know if everyone does, but I think it's key to building a successful ecosystem that's equally valuable to end users and to the partners enabling those end users.

Matte: I guess from a technical perspective, I'd say this whole ecosystem idea is trying to bridge different web services or applications or web software. You have a lot of the same distributed systems, problems you have elsewhere where there's two different independent pieces that have their own knowledge that are working together. As far as tooling and what you should build and focus on, it's the space between those two services, which a lot of the time is requests from one to the other and vice versa. I think it's really useful to have a good understanding of what's going on between those two services so that (a) you can understand the health of those integrations, and also you can understand when things are going wrong and you have enough information to figure out what's going wrong and to fix it quickly.

Matte: I think it's really easy to get into a situation where it's a black box and these two services are working together most of the time, but then things go wrong, and you don't have enough information to figure out why, and then it leads to a bad experience for customers and etc. I think it's really important to understand the operational maintenance side of it technically.

Idan: Got you. Well, Matt, thank you so much for joining me today and for talking through all this stuff with me.

Matte: Yeah. Thanks for having me. It was awesome.

Idan: Look forward to having you on the podcast again in the future hopefully. Thank you so much.

Matte: Yeah, thank you very much.