gRPC in Production: The Sharp Edges Nobody Warns You About

# gRPC in Production: The Sharp Edges Nobody Warns You About

Six months ago, my team migrated our internal service mesh from REST over JSON to gRPC. The migration took three weeks. The debugging, the tooling gaps, and the mid-production "why is this stream randomly dying" moments? Those are still ongoing.

I still reach for gRPC for new services. But I no longer think of it as "REST but faster." It's a fundamentally different protocol with its own failure modes, tooling requirements, and operational patterns. The tutorials skip all of this. Here's what you actually need to know.

The Good: Why You'd Switch

Let me be clear upfront — gRPC solves real problems:

• Strong contracts. Protobuf schemas are machine-readable, language-agnostic, and support breaking-change detection. No more "is this field a string or an optional string?" guesswork at 2 AM.
• Streaming. Server-side, client-side, and bidirectional streams are first-class citizens. You cannot do WebSocket-level streaming with REST without building your own framing.
• Performance. HTTP/2 multiplexing means one TCP connection handles concurrent RPCs. No connection-per-request overhead. Protobuf binary encoding is ~4x faster to serialize than JSON and produces payloads 3-10x smaller, depending on the data shape.
• Client generation. A protoc invocation gives you typed clients in 10+ languages. No more hand-rolling fetch wrappers.

These numbers are compelling. But they come with strings attached.

The Bad: Things That Will Break

1. Load Balancers Hate Long-Lived Streams

Here's the first thing that bit us: we deployed behind an AWS NLB and suddenly bidirectional streams started dropping after 60 seconds.

The culprit? NLB idle timeout.

Most HTTP load balancers are designed for request-response. A gRPC streaming RPC holds the connection open indefinitely. If your load balancer terminates idle connections — and most do — your streams die silently. Clients reconnect, sure, but you lose whatever state was mid-stream.

The fix:

# AWS NLB — increase idle timeout to maximum (3500 seconds)
# Or use an ALB with HTTP/2 support (but ALB has its own limits)
# Better yet: use a Layer 4 load balancer that doesn't inspect HTTP at all

# For Kubernetes nginx-ingress, you need these annotations:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/grpc-read-timeout: "86400s"
    nginx.ingress.kubernetes.io/grpc-send-timeout: "86400s"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "86400s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "86400s"
    # ☝️ Yes, all of these. Missing one silently drops streams.
spec:
  rules:
    - host: api.rrezvin.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grpc-service
                port:
                  number: 50051

Lesson: Test with bidirectional streaming on Day One, not Day 90. Every hop in your infrastructure — load balancer, ingress controller, service mesh sidecar — has its own timeout defaults. Map them all.

2. The Error Story Is a Mess

A JSON REST API returns this:

{
  "error": {
    "code": "INVALID_ARGUMENT",
    "message": "email field must be a valid email address",
    "details": {
      "field": "email",
      "expected": "email format",
      "received": "not-an-email"
    }
  }
}

A gRPC call returns this (pseudocode of what actually hits the wire):

status = Status{ code: 3, message: "email field must be a valid email address" }

That's it. gRPC has a fixed set of 17 status codes. HTTP has ~70. Your rich error details? There's no standard place for them.

The gRPC docs suggest using the google.rpc.Status message type with google.rpc.ErrorInfo. Does your client library populate it? Does the client know to unpack it? In our experience, nobody does this correctly.

What we do instead — a structured error envelope:

syntax = "proto3";
package errors;

message Error {
  string correlation_id = 1;
  string message = 2;
  string user_message = 3;
  google.rpc.Code code = 4;
  repeated ErrorDetail details = 5;
  // Human-readable stack trace for debugging (never show to end users)
  string debug_info = 6;
}

message ErrorDetail {
  string field = 1;
  string description = 2;
  string actual_value = 3;
}

Now the server extracts this from the context and the client always unpacks the same envelope. No guessing.

3. Debugging Is Painful

curl doesn't speak gRPC. Neither does Postman. Not really.

You can use grpcurl, grpcui, or BloomRPC. They all work — until they don't. Reflecting on a server that doesn't expose the reflection service? You need the .proto files locally. In a monorepo of 200+ protos, good luck.

Here's our go-to debug setup:

# Install grpcurl
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest

# List services (requires reflection or --proto flag)
grpcurl -plaintext localhost:50051 list

# Invoke an RPC with JSON (not protobuf — useful for ad-hoc debugging)
grpcurl -plaintext -d '{"user_id": "abc-123"}'   localhost:50051 users.UserService/GetUser

# For production: server reflection MUST be disabled
# Use a local proto path instead
grpcurl -proto ./api/proto/v1/users.proto   -import-path ./api/proto   -H "Authorization: Bearer $(gcloud auth print-access-token)"   api.rrezvin.com:443   users.UserService/GetUser

The real pro tip: Write a small CLI proxy that converts gRPC to JSON-over-HTTP for debugging. It's 50 lines of Go and saves you hours.

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"strings"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/proto"
	"google.golang.org/protobuf/reflect/protoreflect"
)

// This is a sketch — adapt to your service definitions.
// The key insight: use protojson to bridge the JSON↔Protobuf gap.
func main() {
	cc, err := grpc.Dial("localhost:50051",
		grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	defer cc.Close()

	// Register routes manually (or use reflection for a generic proxy)
	http.HandleFunc("/api/v1/users/", func(w http.ResponseWriter, r *http.Request) {
		if r.Method != http.MethodGet {
			http.Error(w, "only GET supported", http.StatusMethodNotAllowed)
			return
		}

		userID := strings.TrimPrefix(r.URL.Path, "/api/v1/users/")
		// Call the gRPC method
		// This is pseudo — you'd call your generated client
		resp := callUserService(cc, userID)

		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(resp)
	})

	fmt.Println("gRPC debug proxy on :8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

If your team has more than five engineers using gRPC, invest in the developer experience upfront. It's the #1 complaint we hear from teams adopting the protocol: "I can't debug it."

The Ugly: Architecture Decisions That Haunt You

4. Versioning Is Harder Than You Think

REST APIs handle versioning through URL paths (/v1/, /v2/) or headers. It's not elegant, but it's straightforward.

gRPC has package declarations. Your common options:

• package users.v1; vs package users.v2; — Two completely separate packages. No code sharing. Cross-version client code is duplicated or requires adapter layers.
• Compatibility via field semantics. Add new fields, never remove old ones. Clients ignore unknown fields (protobuf wire format preserves them). This works well for additive changes but requires discipline around field numbering and semantic meaning.

The approach that actually works in practice:

package users.v1;

message User {
  // Fields 1-15 use 1 byte varint encoding — reserve for most frequent/stable fields
  string id = 1;
  string email = 2;
  string display_name = 3;

  // Fields 16-2047 use 2 byte encoding — use for new, optional fields
  reserved 4; // was 'phone_number' — never reuse old field numbers
  string profile_picture_url = 5;
  bool email_verified = 16;
  optional string phone_number_v2 = 17;
}

Key rules:

• Never delete fields. Use reserved.
• Never change the type or semantic meaning of an existing field number.
• Always set optional for new fields so old clients distinguish "not set" from "default value."
• Run buf breaking in CI. Always.

5. gRPC-Web Is Not gRPC

If you think you'll use gRPC from the browser: think again. The gRPC-Web protocol is a different beast:

• No bidirectional streaming (client-side streaming is supported; server-side requires workarounds).
• Different content-type headers (application/grpc-web vs application/grpc).
• Requires a proxy (Envoy, grpc-web proxy, or a service mesh sidecar) to translate between gRPC-Web and standard gRPC.

Performance on mobile is also deceptive. Protobuf decoding in JavaScript is slower than JSON parsing in v8. On a low-end Android device, decoding a 50KB protobuf response can take 8-15ms, while parsing the same data as JSON takes 2-4ms. You're adding latency, not removing it.

Our recommendation: Keep REST/JSON as your public API. Use gRPC for internal service-to-service communication. Don't force your mobile and web clients through a gRPC-Web proxy unless you need server-side streaming on the frontend — and you almost certainly don't.

The Advice I'd Give My Past Self

If you're considering gRPC today, here's my honest playbook:

1. Use gRPC for internal services only. Your public API stays REST/JSON with an OpenAPI spec. Keep the boundary clean.

1. Invest in tooling before you need it. Set up server reflection in dev, grpcurl commands in your runbooks, and a debug proxy that bridges gRPC and JSON. Do this before the first on-call rotation.

1. Document your dead connection policy. Every streaming endpoint needs a documented "what happens when the connection drops" section. Because it will drop.

1. Use buf for proto management, not raw protoc. The dependency resolution, linting, and breaking change detection are worth the complexity tradeoff.

1. Keep field numbers under 16 for your most frequent fields. 1-byte vs 2-byte varint encoding adds up when you're serializing millions of messages.

1. Test bidirectional streaming from Day One. Do not wait until production. Every infrastructure layer will have a default timeout that breaks your streams.

Conclusion

gRPC is not "REST but faster." It's a different architecture with different tradeoffs. The performance and type safety are real, but so are the debugging pain, the load balancer configuration nightmare, and the versioning tax.

For internal service meshes? Use gRPC. It's the right tool. For browser-to-server communication? Keep REST. For mobile? Evaluate carefully — the protobuf decode cost on slow devices is non-trivial.

The mistake isn't choosing gRPC. The mistake is choosing it without understanding what you're signing up for.

— Technical content team at Rrezvin