gRPC in practice

The four call types, and what half-close means

A gRPC method is one of four shapes, set by where you put stream in the proto:

service Chat {
  rpc Send(Msg) returns (Ack);                       // unary
  rpc Subscribe(Topic) returns (stream Msg);         // server streaming
  rpc Upload(stream Chunk) returns (Summary);        // client streaming
  rpc Talk(stream Msg) returns (stream Msg);         // bidirectional
}

Unary is a normal request/response. In server streaming the client sends one message and reads a sequence back; the server sends all messages, then a final status. In client streaming the client writes a sequence, then half-closes (signals "no more messages"), and waits for one response. Bidirectional gives both sides an independent read-write stream: the two directions are not lock-stepped, so each side reads and writes in whatever order it likes.

The word to know is half-close: the client can finish sending while still reading. It is not a full close. The server keeps its side of the stream open until it sends its status. A common confusion is expecting bidi streaming to be a strict ping-pong; the streams are independent, and a server can send ten messages before the client sends its second.

Always set a deadline

An RPC with no deadline can run until the server's maximum timeout, and while it waits it holds resources: a thread, a connection, memory. Chain a few of those under load and you exhaust the pool. gRPC lets the client say how long it will wait; when that passes the call fails with DEADLINE_EXCEEDED.

Deadline and timeout are two spellings of the same thing. A timeout is a duration ("500ms from now"); a deadline is a fixed point in time. Some language APIs take one, some the other. Pick a real number per call based on what the caller can tolerate, not a default of "forever."

Wrong "gRPC sets a sensible default deadline, so I only override it when I need to." There is no default deadline. If you never set one, the RPC waits indefinitely (up to any server-side max), which is how one slow dependency hangs a whole request path.

gRPC status codes are not HTTP status codes

A finished call carries a status code. OK means success. Errors use gRPC's own set, not HTTP's: INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, RESOURCE_EXHAUSTED, FAILED_PRECONDITION, UNAVAILABLE, DEADLINE_EXCEEDED, UNIMPLEMENTED, INTERNAL, CANCELLED. A few map cleanly to intent: UNIMPLEMENTED means the method does not exist on the server (you called an old server, or a method it does not have), UNAVAILABLE means the server is down or shutting down and is the one you generally treat as retryable, INVALID_ARGUMENT is a bad request the client should fix and never retry blindly.

The basic model is a code plus an optional message string. When you need structured detail (which field was invalid, how long to back off), the richer model carries protobuf messages as trailing metadata, covered in the senior notes.

Evolving a proto without breaking clients

The rule that trips people: the field NUMBER is what travels on the wire, never the field name. string email = 3 is stored on the wire as "field 3, this value." Rename email to email_address and keep the number, and nothing breaks on the wire; it is a source-level change. Change the number, and every already-deployed client disagrees with you about what field 3 means.

message User {
  string id = 1;
  reserved 3;            // was `email`, deleted; lock the number
  reserved "email";      // and the name, so nobody re-adds it
  string phone = 4;      // new fields ALWAYS take a fresh number
}

Wrong "The field was deleted and nobody uses it, so I can reuse number 3 for the new field." An old client still sends its value under tag 3, and the new server parses those bytes as the new field: silent type confusion or data corruption, no error thrown. proto3 keeps fields it does not recognize (unknown fields are preserved, not rejected), so there is no loud failure to catch. Right add-only. New field, new number; reserved the numbers and names you drop.

Safe additive changes: adding a field, removing a field (as long as you reserve its number), adding a value to an enum. Field numbers 1 through 15 encode in one byte, 16 and up take two, so give your hot fields the low numbers.

Reuse the channel

Creating a gRPC channel is not free: it does name resolution and opens an HTTP/2 connection. Build one channel per target and share it across calls for the life of the process. A channel-per-call pattern pays connection setup on every RPC and defeats the connection reuse gRPC is built around. Keep the stub and channel long-lived; use keepalive pings if the connection sits idle between bursts.

Why L4 load balancing starves gRPC backends

gRPC opens one long-lived HTTP/2 connection per target and multiplexes many concurrent RPCs as streams over it. That is great for latency and terrible for a naive load balancer. An L3/L4 balancer works at the connection level: it terminates a TCP connection and opens one to a chosen backend, copying HTTP/2 and gRPC frames across without parsing them. So it places whole connections, not requests. Once a client's single connection lands on a backend, every RPC on it stays there, and scaling the backend pool out changes nothing until connections churn.

Three ways to balance at the RPC level:

• L7 proxy (Envoy, Linkerd). The proxy terminates and parses HTTP/2, sees each stream, and spreads streams from one client across many backends. Cost: it sits in the data path and adds a hop.
• Client-side load balancing. The client resolves all backend addresses (a headless Service in Kubernetes returns every pod IP) and applies a policy such as round_robin itself, opening a subchannel per backend. No extra hop; the client must be trusted and carry the balancing logic.
• Lookaside (external) load balancing. The client asks a dedicated balancer which backends to use, then talks to them directly. Balancing logic lives in one place; the data path stays direct.

Wrong "Put the gRPC service behind a normal L4 load balancer or a Kubernetes ClusterIP and it will spread load like any HTTP service." It spreads connections, and gRPC gives it almost none to spread. You get one hot backend. One more constraint to state in an interview: a stream, once started, cannot be moved to another backend, so all of these balance at RPC start, and a long-lived streaming RPC pins its whole lifetime to the backend it opened on.

Deadline and cancellation propagation down a chain

A deadline is only useful if it flows. When a server handling an RPC makes its own downstream calls, those inner calls should inherit the caller's deadline, so no one downstream works longer than the original caller will wait. gRPC does not ship the absolute time; it converts the deadline to a timeout with the already-elapsed time deducted, which sidesteps clock skew between hosts. Propagation is on by default in Java and Go and must be enabled explicitly in C++.

When the deadline passes, gRPC cancels the server call and reports CANCELLED, but here is the part people miss: the library cannot interrupt your handler. It has no mechanism to reach into application code mid-execution. A long-running handler must poll its context for cancellation and, when it sees it, stop and abandon the downstream calls and goroutines/threads it spawned. In Java, Go, and C++ an outgoing RPC is cancelled automatically when its parent context is cancelled; elsewhere the handler author does it by hand.

Wrong "Setting a deadline on the outer call automatically kills all the work happening downstream." It cancels the calls, and cooperative handlers observe that and unwind, but a handler that never checks for cancellation keeps running to completion, holding resources the caller already gave up on. Cancellation is a signal, not a kill.

Retries, hedging, and not building a storm cannon

Retry policy is set per method in the gRPC service config: maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, and retryableStatusCodes. gRPC applies jitter of plus or minus 20% to the backoff so a fleet of clients does not retry in lockstep. Two things separate a safe retry setup from an outage amplifier:

1. Retry only what is safe to replay. gRPC retries a failed call by replaying it as a new call, and it is only certain to be safe when the server has not processed the RPC. A non-idempotent write (charge a card, decrement stock) can be applied twice on retry, so gate those with an idempotency key or leave them off the retryable list.
2. Throttle. gRPC keeps a token budget per channel: failed RPCs spend tokens, successes add a fraction back, and when the budget drops below half, retries pause. Without throttling, a transient UNAVAILABLE on a struggling backend turns every client into a retry generator and finishes the backend off.

Hedging is the other tool, for tail latency rather than failure. With a hedgingPolicy the client sends the same request to several backends, staggered by hedgingDelay, takes the first response, and cancels the rest. Set hedgingDelay to zero and all attempts fire at once. Because hedging issues parallel copies of the same call, it belongs only on idempotent methods, and nonFatalStatusCodes controls which failures keep the other attempts alive versus cancel them all.

The wire-compatibility catalog

Beyond "never reuse a number," seniors are expected to know which type changes survive the wire and which corrupt it, because the compiler will not stop you:

• int32, uint32, int64, uint64, and bool are wire-compatible with each other. Widening int32 to int64 is safe; values that were negative under a signed type still decode, though int32 truncation rules apply if a value overflows.
• sint32 and sint64 are compatible with each other but not with the other integer types, because ZigZag encoding is different. Swapping int32 to sint32 silently garbles values.
• string and bytes are compatible as long as the bytes are valid UTF-8.
• For string and bytes, singular is compatible with repeated: a reader expecting singular takes the last value from repeated wire data. Message fields are also singular-repeated compatible, but a singular reader MERGES all the repeated elements into one instead of taking the last. (Numeric repeated fields are packed by default and do not survive this change.)
• Moving a field into an existing oneof is not wire-safe, and neither is changing a field's number.

The reason none of this throws is the one to say out loud: unknown and mistyped fields do not fail the parse, so a bad schema change ships clean and shows up later as wrong data. Reserve dropped numbers AND names, keep changes additive, and reserve a package bump (payment.v1 to payment.v2) for a breaking service-shape change, not a routine field add.

When gRPC is the wrong tool

gRPC's strengths (strong contracts, code generation, streaming, HTTP/2 multiplexing) are aimed at your own backend fleet, not the open web. Protocol choice for public and browser-facing surfaces, including the gRPC-Web proxy setup and its streaming limits, lives in the api-protocols module.