75. gRPC

Doug: Right. With the remote procedure call, you could potentially have issues with the network itself, connecting the machines. You get errors there if the network goes down.

Robert: Whereas, within programming language, procedure call does not get lost on its way from the point where you call it to the code.

Doug: Right. Exactly.

Robert: There have been RPC packages for many years. Could you describe some features common to RPC in general?

Doug: Yeah. RPC, in general, I guess just enables any client and server to exchange messages on a basic level. They might have different features. For instance, with gRPC, it enables you to do streaming call where data is exchanged freeform where the client and server are just continuously streaming messages to each other. There's also a different type of mode where the client would send a request and the server sends a response. It's just a single message in each direction.

Robert: Does the industry agree on canonical architecture or design for RPC implementations or are they all quite different?

Doug: I think they're pretty similar in that regard. There are, obviously, differences between the different types of technologies you can use here. If you talk about things like REST, if you consider that, that's not really RPC. If you look at something like REST, it's similar in some ways to unary RPCs, but it doesn't give you streaming options, for instance.

Robert: What would that canonical architecture based on shared assumptions look like?

Doug: Usually, what you would do is, upfront, you would define a schema for the different types of methods that can be invoked by the client upon the server. Within that schema, you would talk about the different options that you have for best methods themselves. Also, what the data format is for the messages. The types of information that's included by the client and the request of the type of data sent back by the server in the response.

Robert: By schema, is that something like a header file or an interface definition in a conventional programming language?

Doug: Right. Yeah, it's very similar to that or exactly like that. It depends on your RPC framework. In gRPC, typically, you would use an IDL protobuf in order to define your services and your messages.

Robert: IDL stands for?

Doug: Interface Definition Language. It's essentially a separate language that can be used to describe your schema. From there, you typically would generate code or data structures for the language that you're implementing the RPC framework in.

Robert: What are the steps that is generated code does to implement RPC?

Doug: Typically, with protobuf, what would happen is, the IDL file itself, the protobuf definition would be parsed by the protobuf parser. At that point, the intermediate format of that definition would be passed to a language specific library. That would analyze the different services and methods and generate the necessary code for the language from there.

Robert: You mentioned protobuf. Give us a brief overview of what that is.

Doug: Right. Protobuf is an IDL. It was developed by Google. It was used internally for a number of years but has since been open sourced. Essentially, it allows you to define message types and services. It will generate in tens of languages, different language bindings that describe those messages. Also, it enables you to serialize and deserialize messages of those types so that they can be transmitted on the network.

Robert: If I understand where you're going with that, it sounds like you could call an RPC from one language and the recipient side could be in a different language. Is that correct?

Doug: Right. Exactly.

Robert: Now, I've read that Google has had multiple generations of internal RPCs with gRPC being the newest one. How are previous generations on satisfactory and what motivated the development of gRPC?

Doug: Let's see. Actually, the technology that we're using inside of Google is called, Stubby. That's, still, actually widely in use. We have gRPC available internally now, but it's a long process to try to migrate things. In terms of the different features that are provided by gRPC and Stubby, the feature sets are actually very, very similar. gRPC was inspired a lot by Stubby and in fact, was built by a lot of the same people, but the architecture itself is a bit different. I wasn't here when the project started for gRPC. I would imagine that it was considered to open source Stubby itself, but there's a lot of problems with Stubby in terms of the libraries that it requires are all also Google proprietary libraries internal. Those would all need to be open sourced as well in order to open source Stubby. I think that would pose a major problem for trying to open source something like Stubby.

Doug: Also, Stubby is built directly on top of TCP. Meaning, that it needed to implement a lot of the features that gRPC actually gains benefit of more modern transports in the open source community like HTTP/2, which it's built on to provide features like bidirectional streaming and stream multiplexing. There's a lot of good reasons to, essentially, start from scratch on gRPC, which is what we did.

Robert: You mentioned HTTP/2. What does HTTP/2 bring to the RPC world that differentiates it from either RPC built directly on TCP or on http/1 based?

Doug: Okay. Yeah. HTTP/1 is very limited in terms of the types of things that you can do with it. In the context of RPCs HTTP/2, it's a lot more friendly in that regard. The things that HTTP/2 ads that HTTP/1.1 did not have are stream multiplexing. Allowing you to have more than one stream of communication between the two end points and also, bidirectional streaming where the client and server are able to send and receive messages asynchronously, essentially, as the stream life cycle exists. If you were to implement this, obviously, HTTP/2 is built on top of TCP, pretty much, everything is. You could do all these same things over TCP, and that's what Stubby does, but you need to deal with a lot of that complexity yourself and you have to build your own protocol, which is what HTTP/2 defines.

Robert: Would it be fair to say then that gRPC is a much smaller project because it's building on bigger building blocks?

Doug: For sure, versus the internal Stubby tool that we use at Google, it's much simpler in that regard. We're able to leverage the HTTP/2 libraries. In fact, our protocol is a pretty minor extension on top of HTTP/2.

Robert: You mentioned stream multiplexing. Would that work in the manner of Service A is talking to, for instance, Service B that all of the threads in the server running Service A are using the same HTTP/2 connection to service B?

Doug: Right. Yes. That's definitely possible. The gRPC framework allows you to use multiple connections to potentially services running on a number of different machines, but yes, you're able to create multiple streams over a single connection as well.

Robert: When would you use either one or multiple connections based on what considerations?

Doug: An entry level RPC approach would be to have single server for all of your clients, but you'll quickly run into scaling problems that way as your load increases. At that point, it's good to have multiple backends for two purposes. One of which is load balancing to enable you to scale, to support more and more clients. Another one is just for fault tolerance, so that in case a server crashes, then the clients are able to essentially continue uninterrupted by communicating with the services that are still active.

Robert: Just now, you mentioned load balancing. Most networks are set up to load balance HTTP protocols pretty well. Does gRPC get that for free?

Doug: Yeah. You can use existing proxy type load balancer that you have using gRPC because most of these load balancers understand HTTP/2 and gRPC is just a layer on top of HTTP/2. Anything that routes HTTP/2 can also route gRPC.

Robert: You've got all the right ports open and you can get through the network without having to do a lot of work with your network engineering team.

Doug: Right. Right.

Robert: Can you explain the term, head of line blocking and how is it avoided?

Doug: Head of line blocking, in the context of an RPC, would be essentially, you would have multiple RPCs that you're trying to perform simultaneously. If one of them stalls, then you don't necessarily want the other ones to be blocked as well. In HTTP/2, the stream multiplexing allows these streams to have their own flow control, which means that if one of them stalls, the other ones are still able to make forward progress regardless of the progress of that one that's blocked.

Robert: That would be an example of something that gRPC gets from HTTP/2, correct?

Doug: Correct. Although flow control itself, it's a, it's part of the protocol and it's defined by HTTP/2, but it is a bit tricky to manage it correctly as an application, which the HTTP/2 spec even calls out directly. That does take a bit of effort to get that working right.

Robert: Could you give an example of how you could get yourself in trouble by not doing it quite right?

Doug: Sure. We've had bugs in the past related to this. Essentially, the flow control is a way of a receiver pushing back on the sender to let it know that it's not ready to receive data. We have had in the gRPC Go implementation. We have had issues in the past that were caused by head of line blocking type situations where one stream not reading would block all of the streams from the sender from being sent, but we have, since, essentially taken a new approach to the way that we do flow control since then. It's a bit challenging. Essentially, what was happening at that layer is, flow control applies both to a stream on a stream by stream basis, but because you're allowed to have so many streams on a connection, if you only applied it at that level, you would end up potentially with a lot of data sitting on the server needing to be processed, but nothing reading it if you allow all of the streams to send their maximum amount of data.

Doug: You also have connection level flow control, which prevents that from building up as well. We now use that transport level, connection level flow control a bit more intelligently in order to avoid situations like that.

Robert: Is this something that the developer who's using gRPC needs to worry about or is this something that you have to get right in the libraries?

Doug: Yeah, that's mostly a library concern. It potentially impacts your application in that when the flow control limits are reached, your application will receive pushback. In blocking APIs, it's fairly trivial to deal with. Essentially, when you go to send a message, that call will stall until you have sufficient flow control available to send that message. It can be a little trickier for asynchronous APIs where you have to, ahead of time, confirm that you have enough flow control to send a message before you do perform that call. Aside from that, most of the complexities and trickiness of dealing with flow control are handled by the library itself.

Robert: You just now mentioned blocking asynchronous APIs. Do the library support both models?

Doug: In C++ and Java, which are the two of the three main languages that we support, they have both asynchronous and synchronous models. In the Go implementation, which I'm working on, the Go style and community doesn't really use asynchronous in general. Blocking calls are preferred for most operations. For Go, we only have blocking APIs.

Robert: With HTTP, the protocol itself returns a protocol status result. You have the gRPC abstraction where you have a method which might return method result if the status result is some kind of error code or maybe error metadata. How does the client or the developer programming separate the HTTP status issues from the method data.

Doug: Okay. The gRPC library attempts to abstract all of that away from the user. Our API APIs deal with gRPC statuses exclusively and not HTTP status codes and things like that. That said, HTTP status codes do happen because we are built on top of HTTP. If there is a proxy in the middle, then that proxy might send back unusual status codes. Whatever you're communicating with a gRPC backend, that backend actually will always only return status okay and then it would return a gRPC status error, which is what we would convey to the user. We wouldn't normally have to worry about that, but with a proxy in the mix, you potentially get back HTTP status codes that are not okay. We need a way to translate those into something that the user's application is able to understand. Typically, it turns into an unknown status in the gRPC world and then the information about the status, the text data, would include the HTTP status code and any information that came out through that path.

Robert: It's fairly common in microservices. You're chaining RPCs from service A to B to C and I suppose, somewhere further down the chain you hit a bad status. Does each layer wrap that and you get back something failed or do you get full stack trace of where the original failure happened?

Doug: Yeah, that would all be up to the service implementers at that point. When a service, somewhere in the middle of the chain, sees that HTTP error, it would need to handle that in some way, right? It could choose to retry the operation and then no error would be seen by the client calling that service or it could choose to propagate that error directly or indirectly. At that point, really, it comes down to how you've programmed your application and all of your applications together in terms of like, do you get a full on stack trace or does each layer hide the details of what happened over there.

Robert: We've talked a little bit about protobufs. How does protobuf make the cross language RPCs possible?

Doug: Protobuf is really great and that it's supported by so many different languages. When you define your IDL once, then you end up with something that works in all of these different languages. When gRPC comes into the mix, then you end up able to transmit that data in a serialized form over the network and then it doesn't matter whether the remote side is in the same language or a different language. It's able to receive the data off the wire. Deserialize it using the protobuf library that it has and then get it into a format that it's able to work with in a language native way.

Robert: What are some of the languages that protobuf exists in?

Doug: Let's see. We have C++, Java and Go, which are the three main gRPC languages. Also, the C implementation of gRPC is wrapped in a number of different languages like C# and Node and Ruby. All of these languages also have protobuf support as well. There's many more. I can't remember them all.

Robert: Is it possible to call gRPC from JavaScript code that runs inside a browser?

Doug: Yeah. There's a couple different ways to do that. One is gRPC web, which is a project that is more end by the core gRPC team. That enables you to use JavaScript to perform RPCs in a fairly similar way that you would if it was a desktop application or a server application. The other style is, there's a program called gRPC gateway. That's actually an open source contribution, but it enables your servers to essentially provide a REST endpoint that converts a gRPC service into a REST looking endpoint and then browsers are able to access your gRPC service without using any special JavaScript code or anything just normal REST dial calls.

Robert: As a developer working with HTTP/1 based services, it's very convenient to use curl, especially if the data is in JSON format. Is there anything equally as convenient for developers in the gRPC world?

Doug: Yes, for sure. As we mentioned at the beginning here, Google has been using Stubby for a number of years. gRPC is very similar to Stubby. Obviously, in order to develop Stubby applications in a whole suite of microservices, all running that type of framework, then you need an easy way to be able to debug that. Typically, with REST, you would be using JSON and you would be using curl. With gRPC, there is a tool that we built called gRPC CLI. There's also an open source tool that we're moving towards more called, GRPCurl. That, basically, brings curl-like functionality to your gRPC microservices. It's, in fact, better than curl in some ways, because the way that it works is, the service exposes the schema that it provides through a reflection service.

Doug: Now, this GRPCurl or gRPC CLI application is able to discover the actual protocol through that reflection service and learn about exactly what methods are available and see the format of the data for the request and see the data for the response. You can actually do even better debugging through that than you could through curl where you have to know ahead of time what methods you can call and what the schema is.

Robert: We've talked about features provided by HTTP, one of those being TLS. Does gRPC piggyback on the TLS implementation of the HTTP?

Doug: Yeah. TLS is fully supported. The way that works is, when gRPC creates a connection to a remote service, they would negotiate TLS at that time. Actually, TLS then would secure the communication server of that transport. TLS is optional. Also, there are other alternatives that you can use instead of TLS, if that's what you need for your system.

Robert: Let's talk about now the developer workflow. I understand it started with an IDL and specify the methods, parameters, names, and types. What are the steps that the developer now goes through to get that RPC client server up and running?

Doug: The next thing that you would do after defining your IDL is, you would run the protobuf tool itself in order to generate language bindings for those messages and services. The gRPC version of the proto plugin will generate the services in a way that allows the calls to go through the gRPC library. Once that happens, client applications would essentially use this generated code directly to create wrappers around a gRPC connection that then allow you to call into it using language native bindings in order to perform those calls in a very natural way. On the service side, you would implement interfaces that are defined by that step and implementing those interfaces will implement the method calls that you'll be handling on the server side.

Robert: This might be a language specific question. Do I, on the server side, modify generated code by adding implementation to the same files or is the code that the developer writes kept separate from the generated code?

Doug: Right. For gRPC and protobuf, in general, the generated code ends up in files that are not to be modified by the user. If they were done the other way where you added your implementation code into the generated code, then potentially, when you want to change your services, like if you want to add fields to a message or if you want to add methods to a service, then now, you would potentially run into problems when you went to regenerate those language bindings. The code always ends up in a separate file that you then import or include by your application and you go from there.

Robert: If I, for example, added a method parameter to a method, but I forgot to update the code that implements a method, would I get either a compile failure or some kind of runtime error?

Doug: If you add a new method, then at that time, if your service did not implement the method, then when the client made the call to the service for that method, the library would return back an error back to the client saying that this method was unimplemented. Your application will still continue to compile. It's just that the methods that you don't implement return not implemented.

Robert: I want to switch gears again. You mentioned streaming. What is a use case for streaming?

Doug: Right. I guess, when we talk about streaming, there's actually three different kinds of streaming. The base RPC is unary where the client sends one message and server sends one response. There are three types of streaming, which are server streaming, client streaming and bidirectional streaming. Depending on your application, you might choose a different mode for your method. Where you might use server streaming is, if you have, for instance, a database request where the client sends a simple message to indicate it's a request for data from a database and then the server potentially needs to return a lot of information that matches that query. You would often use a streaming RPC for that where every message that you stream would be a database entry, for instance.

Doug: If you move on to client side streaming, this could be something like uploading a file because the client would chunk the data and send it over the course of many, many messages but the server really just needs to listen, receive all of those message. Once the client indicates that it's done, the server would just say, "Okay, yes, I got it." Bidirectional streaming, an example here would be Pub/Sub, for instance, where a server might stream messages to a client for it to process and the client to send back acts indicating that it received the message and processed it.

Robert: Would use cases like a map with cars driving around that I want to be updated in real-time or price charts showing real-time stock quotes? Are those examples of use cases where you'd use the bidirectional?

Doug: Yeah. I guess it would depend on how you wanted to implement it, but either bidirectional or server streaming there, would be appropriate. Because I think in your examples, it could be the case, but the client really just sends an intro message to say, subscribe and then the server, now, will be streaming updates as things change.

Robert: Is this a better solution than what the web has struggled with for more or less forever to the issue of, let's say, events that originate on the server that need to be pushed out to the client? You have things like long polling and comet and other ways of trying to hack a request response protocol into an asynchronous event delivery system?

Doug: Yeah. I think the challenge for the web, when you talk about the web, is that it's HTTP based and really HTTP/2 support isn't that great in a web browser. This type of approach would be great, but I think unfortunately, a lot of the time when you're dealing with the web, you're dealing with HTTP/1. You're stuck with some of those long polls like you're talking about, but if you have a client server application for your enterprise, this is definitely the way to go.

Robert: One of the main competitors is going to be REST. In REST, you have just the four methods but you can take the thing that doesn't fit into those four methods and call it a resource, so kind of ends up being almost the same. Is there any real difference between doing a PUT to account/transfer or having a transfer method?

Doug: I think they're pretty interchangeable in that regard. I think OUT from REST would map pretty similarly to a unary RPC in gRPC.

Robert: When starting up a project ask you for advice, what are some pros and cons of gRPC versus REST?

Doug: For gRPC pros versus REST would be the streaming that we talked about. That thing is not supported in REST. Another one is, if you're using REST, then a type of operation where the client wants to read a value, modify the value and then write the value back, but in a way that no other modifications to that can happen in the meantime. That type of thing is not really possible on REST. You can implement it by workarounds in REST, but otherwise, it's actually pretty tricky. Whereas in gRPC, you could do this with a streaming RPC where the client would indicate to the server, please give me this information and that it would make the local modifications and then it would push back to the server like use this as the new data for that value. I guess some advantages of REST would be that it is an industry standard. A lot of existing end points already know how to deal with REST. There's a lot of community and ecosystem built up around REST.

Robert: In a distributed system, there are a couple of solutions out there for distributed tracing. Does that play well with the gRPC?

Doug: gRPC is essentially built based on all of our learnings at Google from Stubby. We certainly have this type of capability within Google. It was extremely important for us to get that functionality into gRPC as well. The way that that works is through a plugin interface using interceptors or in Go, we call them stats handlers where the transport itself at the low level is able to call out into a third-party application, which would be the distributed tracing library in this case and inform all of the operations going on and allow it to actually interact with the RPC a bit in terms of, essentially, injecting headers onto the RPC to indicate the tracing information that it needs.

Robert: Building distributed system with a microservice architecture, is gRPC a good model for the internal service to service communication?

Doug: For sure. I think, definitely, that's the way I would recommend doing it. It allows you to build your services in a way that creates a lot of small applications owned either by different teams or even if it's the same team owning the code. You can deploy them independently. You can upgrade them independently. gRPC enables you to have this type of architecture, which is a lot more flexible. You can scale different parts of it. If one part of your service needs to be used by a different application for whatever reason, you can start scaling up that part of your service without needing to do anything to the other services. Really, it just opens the door to a lot of possibility. I think it's definitely a great way to develop an application for any enterprise.

Robert: Doug, we've covered a pretty good overview of gRPC here. Was there anything you wanted to talk about that we haven't discussed?

Doug: The gRPC community is all open source. We are part of CNCF. We're always looking for more contributors, more users, our GitHub repos. We're always happy to hear feature requests and especially happy to help you with any bugs you think you might've encountered in our library. We also go out and proactively look on Stack Overflow. We have a Gitter and Slack, I believe, as well. We monitor these and try to help our community as much as we can. Please feel free to reach out to us. Let us know any thoughts you have. Yeah, we're happy to hear from everybody.

Robert: Where's the point people should start on the web to learn more?

Doug: The main website would be grpc.io. That has all of the starter information. It has getting started guides. It has more in-depth API documentation there. The code itself is hosted on GitHub. You can find that at github.com/grpc. That'll take you to the org that would then link to all the different language implementations and the other side projects that we have going on.

Robert: If listeners would like to reach out to you, where can they find you on the internet?

Doug: The best way would be through GitHub. I am dfawley on GitHub. I assume you'll put that in the show notes.

Robert: Yes.

Doug: Yeah. That's probably the easiest way to get ahold of me or if it's gRPC related, feel free to open an issue in our repo. Under the issues, new issue, there's the ask a question button. That's great if you have a question, even if it's not a feature or a bug.

Robert: Great. Doug, thank you so much for speaking to Code[ish].

Doug: Great. Thank you for having me.