Rate Limiting, Throttling, and Backpressure: The Patterns That Keep Systems Alive

// Redis sorted-set-based sliding window
func (r *RedisSlidingWindow) Allow(ctx context.Context, userId string) (bool, time.Duration, error) {
    now := time.Now().UnixMilli()
    windowStart := now - r.windowMs
    
    key := fmt.Sprintf("ratelimit:%s", userId)
    
    script := redis.NewScript(`
        local key = KEYS[1]
        local now = tonumber(ARGV[1])
        local windowStart = tonumber(ARGV[2])
        local max = tonumber(ARGV[3])
        
        -- Remove old entries
        redis.call("ZREMRANGEBYSCORE", key, 0, windowStart)
        
        -- Count current entries
        local count = redis.call("ZCARD", key)
        
        if count >= max then
            -- Get the oldest entry to calculate retry-after
            local oldest = redis.call("ZRANGE", key, 0, 0, "WITHSCORES")
            local retryAfter = math.ceil((max - count + 1) * (now - tonumber(oldest[2])) / count)
            return {0, retryAfter}
        end
        
        -- Add current request
        redis.call("ZADD", key, now, now)
        redis.call("EXPIRE", key, math.ceil(r.windowMs / 1000) + 1)
        
        return {1, 0}
    `)
    
    result, err := script.Run(ctx, r.client, []string{key},
        now, windowStart, r.maxRequests).Slice()
    if err != nil {
        return false, 0, err
    }
    
    allowed := result[0].(int64) == 1
    retryAfterMs := result[1].(int64)
    return allowed, time.Duration(retryAfterMs) * time.Millisecond, nil
}
``$

### Fixed Window: Don't Use It

Fixed window (count requests in 1-second buckets) has a well-known edge case: if you have a limit of 100 req/s and a client sends 100 requests at 999ms and another 100 at 1001ms, you let through 200 requests in 2ms — twice your intended capacity. This burst amplification makes fixed window unsuitable for anything where accuracy matters.

### Choosing Your Algorithm

| Algorithm | Burst handling | Memory | Distributed | Global accuracy |
|---|---|---|---|---|
| Token bucket | Excellent (configurable) | O(1) | Redis Lua script | Good |
| Sliding window log | Perfect | O(N) | Redis sorted set | Best |
| Sliding window counter | Good (approximate) | O(1) | Redis counter | Very good |
| Fixed window | Poor (burst amplification) | O(1) | Easy | Poor |
| Leaky bucket | None (smooths only) | O(1) | Redis counter | Good (no bursts) |

My rule of thumb: **use token bucket for most APIs** (configurable burst handling, low memory), and **sliding window log for billing-critical limits** where you need perfect counting.

---

## Pattern 2: Throttling — Protecting Your Downstreams

Throttling is what you do when you're the client, not the server. Your service needs to call a database, an external API, or a queue, and that dependency has its own capacity constraints. If you hit it harder than it can handle, you cause failures that cascade back to your users.

### The Concurrency Limiter

The simplest form of throttling is limiting concurrent requests. This prevents a sudden flood of work from overwhelming a downstream service.

// Concurrency-limited HTTP client in Go

type ThrottledHTTPClient struct {

client *http.Client

sem chan struct{}

timeout time.Duration

}

func NewThrottledHTTPClient(maxConcurrent int, timeout time.Duration) *ThrottledHTTPClient {

return &ThrottledHTTPClient{

client: &http.Client{Timeout: timeout},

sem: make(chan struct{}, maxConcurrent),

timeout: timeout,

}

}

func (c ThrottledHTTPClient) Do(req http.Request) (*http.Response, error) {

// Acquire a semaphore slot — blocks if at capacity

c.sem <- struct{}{}

defer func() { <-c.sem }()

// Add a context timeout that respects throttling wait time

ctx, cancel := context.WithTimeout(req.Context(), c.timeout)

defer cancel()

return c.client.Do(req.WithContext(ctx))

}

Adaptive Throttling

Static throttling limits are brittle. Your limit of 50 concurrent database calls might work at 3PM but be too aggressive at 3AM when the database is being backed up. Adaptive throttling adjusts the concurrency limit based on real-time feedback from the downstream.

Google's SRE book describes client-side throttling with request cost accounting. The idea: track the ratio of successful requests to total requests. If success rate drops below a threshold, reduce the concurrency limit. If it recovers, increase it.

// Adaptive throttling — inspired by Google SRE techniques
type AdaptiveThrottle struct {
    mu             sync.RWMutex
    maxConcurrency int64
    minConcurrency int64
    successCount   int64
    totalCount     int64
    lastAdjustment time.Time
}

func (at *AdaptiveThrottle) ShouldAccept() bool {
    at.mu.RLock()
    max := at.maxConcurrency
    total := at.totalCount
    success := at.successCount
    at.mu.RUnlock()

    // If we're below max concurrency, always accept
    // (concurrency tracking happens elsewhere)
    if max <= 0 {
        return true
    }

    // Google SRE formula: accept if requests * (success/total) < max
    // Simplified: reject when success rate drops below 50%
    if total > 100 { // warm-up period
        successRate := float64(success) / float64(total)
        if successRate < 0.5 {
            at.adjust(-1)
            return false
        }
        if successRate > 0.9 {
            at.adjust(1)
        }
    }

    return true
}

func (at *AdaptiveThrottle) Record(succeeded bool) {
    at.mu.Lock()
    defer at.mu.Unlock()
    
    at.totalCount++
    if succeeded {
        at.successCount++
    }
    
    // Decay old counts to avoid stale data
    if at.totalCount > 10000 {
        at.totalCount /= 2
        at.successCount /= 2
    }
}

func (at *AdaptiveThrottle) adjust(delta int64) {
    if time.Since(at.lastAdjustment) < 5*time.Second {
        return // don't adjust more than once per 5 seconds
    }
    at.lastAdjustment = time.Now()
    
    newMax := at.maxConcurrency + delta
    if newMax >= at.minConcurrency && newMax <= 1000 {
        at.maxConcurrency = newMax
    }
}
``$

**Why adaptive matters:** In the incident I described at the start, a static rate limit of 500 req/s on the API gateway was useless because the gateway itself was saturated at 80,000 connections. An adaptive throttle on *every outgoing call* to the database would have started rejecting requests as soon as database latency spiked, before the gateway-level rate limiter ever saw a problem.

### The Queue: When You Can't Drop Requests

Sometimes you can't return 429. Payment processing, order fulfillment, audit logging — these need to go through eventually, even if slowly. That's where queues and throttled producers come in.

// Token-bucket-based throttler for queue producers in TypeScript

class ThrottledQueueProducer {

private bucket: TokenBucket;

private queue: Array<{ item: unknown; retries: number }> = [];

private processing = false;

private maxRetries = 3;

constructor(

private producer: (item: unknown) => Promise<void>,

ratePerSecond: number,

burstSize: number

) {

this.bucket = new TokenBucket(burstSize, ratePerSecond);

}

async enqueue(item: unknown): Promise<void> {

this.queue.push({ item, retries: 0 });

if (!this.processing) {

this.process();

}

}

private async process(): Promise<void> {

this.processing = true;

while (this.queue.length > 0) {

if (!this.bucket.allow()) {

// No tokens available — wait for refill

await this.sleep(1000 / this.bucket.refillRate);

continue;

}

const job = this.queue.shift()!;

try {

await this.producer(job.item);

} catch (err) {

if (job.retries < this.maxRetries) {

this.queue.unshift({ ...job, retries: job.retries + 1 });

} else {

}

}

}

this.processing = false;

}

private sleep(ms: number): Promise<void> {

return new Promise(resolve => setTimeout(resolve, ms));

}

}

Pattern 3: Backpressure — The System-Level Feedback Loop

Backpressure is the most sophisticated of the three patterns and the most misunderstood. It's not rate limiting applied to yourself. It's a protocol between components where the downstream tells the upstream "I'm at capacity" and the upstream either slows down or stops sending.

TCP Backpressure: The Original

TCP congestion control is the canonical example of backpressure. When a receiver's buffer is full, it stops ACKing packets. The sender sees missing ACKs, assumes congestion, and reduces its send window. The receiver never explicitly says "slow down" — the feedback is implicit in the missing ACKs.

This is the model for application-level backpressure.

Reactive Streams / RSocket

The most formalised application-level backpressure model comes from Reactive Streams (adopted by Project Reactor, RxJava, Akka Streams). The core concept: the consumer tells the producer how many items it can handle, and the producer never exceeds that amount.

// Minimal backpressure protocol in TypeScript
interface Subscriber<T> {
  onNext(item: T): void;
  onError(err: Error): void;
  onComplete(): void;
}

interface Subscription {
  request(n: number): void;  // THE KEY METHOD
  cancel(): void;
}

interface Publisher<T> {
  subscribe(subscriber: Subscriber<T>): Subscription;
}

// Example: a database query publisher with backpressure
class DatabaseQueryPublisher implements Publisher<Row> {
  constructor(
    private db: Database,
    private query: string,
    private pageSize: number = 100
  ) {}

  subscribe(subscriber: Subscriber<Row>): Subscription {
    let cancelled = false;
    let demand = 0;
    let offset = 0;
    let inFlight = false;

    const subscription: Subscription = {
      request: async (n: number) => {
        demand += n;
        if (inFlight || cancelled) return;
        inFlight = true;

        while (demand > 0 && !cancelled) {
          const batch = await this.db.query(
            `${this.query} LIMIT ${this.pageSize} OFFSET ${offset}`
          );

          for (const row of batch) {
            if (demand <= 0 || cancelled) break;
            subscriber.onNext(row);
            demand--;
            offset++;
          }

          // If the batch was smaller than pageSize, we're done
          if (batch.length < this.pageSize) {
            subscriber.onComplete();
            return;
          }
        }

        inFlight = false;
      },
      cancel: () => { cancelled = true; }
    };

    return subscription;
  }
}

// Usage: the subscriber controls the pace
const publisher = new DatabaseQueryPublisher(db, "SELECT * FROM orders");
const subscription = publisher.subscribe({
  onNext: (row) => {
    process(row);
    // Request the next item only after processing the current one
    subscription.request(1);
  },
  onError: (err) => console.error(err),
  onComplete: () => console.log("Done")
});

// Start with the first item
subscription.request(1);
``$

This is fundamentally different from a simple "fetch next page" loop because the *producer* respects the *consumer's* capacity. If the consumer processes slowly, the producer pauses. If the consumer processes fast, the producer sends more. The consumer drives the rate.

### gRPC Backpressure via Flow Control

gRPC has built-in HTTP/2 flow control, but it works at the stream level, not the application level. If your server is processing requests slowly, the gRPC library will buffer them until the stream's window is exhausted. At that point, the server stops reading from the socket, and the TCP backpressure propagates to the client.

But this implicit backpressure is coarse-grained. For fine-grained control, use gRPC's **server-side streaming** with a custom backpressure signal:

// gRPC service with explicit backpressure

service DataProcessor {

// Client sends items, server streams back pressure signals

rpc ProcessData(stream DataItem) returns (stream BackpressureSignal);

}

message DataItem {

bytes payload = 1;

}

message BackpressureSignal {

int32 max_inflight = 1; // How many items the server can handle now

int32 retry_after_ms = 2; // Suggest wait time

ProcessStatus status = 3; // READY, BUSY, CRITICAL

}

enum ProcessStatus {

READY = 0;

BUSY = 1;

CRITICAL = 2;

}

// Server-side: send backpressure signals based on internal queue depth
func (s *DataProcessorServer) ProcessData(stream pb.DataProcessor_ProcessDataServer) error {
    const maxQueue = 100
    queue := make(chan *pb.DataItem, maxQueue)
    
    // Worker goroutine — processes items from the queue
    go func() {
        for item := range queue {
            process(item) // This may be slow
        }
    }()
    
    for {
        // Send backpressure signal before receiving the next item
        status := pb.ProcessStatus_READY
        maxInflight := int32(maxQueue - len(queue))
        
        switch {
        case len(queue) > int(float64(maxQueue)*0.8):
            status = pb.ProcessStatus_CRITICAL
            maxInflight = 0
        case len(queue) > int(float64(maxQueue)*0.5):
            status = pb.ProcessStatus_BUSY
        }
        
        if err := stream.Send(&pb.BackpressureSignal{
            MaxInflight:  maxInflight,
            RetryAfterMs: int32(len(queue) * 10), // estimate
            Status:       status,
        }); err != nil {
            return err
        }
        
        // Now receive the next item
        item, err := stream.Recv()
        if err == io.EOF {
            close(queue)
            return nil
        }
        if err != nil {
            close(queue)
            return err
        }
        
        queue <- item
    }
}
``$

The client reads the backpressure signal before sending each batch. If `maxInflight` drops to zero, the client pauses. When it recovers, the client resumes.

### Channel-Based Backpressure in Go

Go channels are a natural fit for backpressure. A buffered channel is a bounded queue; when it's full, the sender blocks. That blocking *is* backpressure.

// Channel-based backpressure pipeline

func DataPipeline(ctx context.Context, input <-chan RawData) <-chan ProcessedData {

const workers = 5

const buffer = 100

type work struct {

data RawData

ack chan struct{} // backpressure ack channel

}

workChan := make(chan work, buffer)

output := make(chan ProcessedData, buffer)

// Dispatcher: wraps each data item with an ack channel

go func() {

defer close(workChan)

for data := range input {

w := work{data: data, ack: make(chan struct{}, 1)}

select {

case workChan <- w:

// Block here until worker ACKs

// This IS the backpressure signal

select {

case <-w.ack:

// Worker confirmed receipt

case <-ctx.Done():

return

}

case <-ctx.Done():

return

}

}

}()

// Workers: process items and ACK

var wg sync.WaitGroup

for i := 0; i < workers; i++ {

wg.Add(1)

go func() {

defer wg.Done()

for w := range workChan {

result := expensiveTransform(w.data)

select {

case output <- result:

case <-ctx.Done():

return

}

close(w.ack) // Signal dispatcher we've processed this

}

}()

}

go func() {

wg.Wait()

close(output)

}()

return output

}

Putting It All Together: A Multi-Layer Defence

Here's what the incident from the intro should have looked like with all three patterns in place:

Client ──► API Gateway ──► Service A ──► Database
              │                │
          Rate limiter      Throttler
          (token bucket)   (adaptive, 50 concurrent max)
              │                │
          Backpressure ◄── Backpressure ◄── Backpressure
          (429 response)   (delayed acks)   (connection pool wait)

The failure I described originally would have played out differently: the gateway's token bucket would have rejected most of the 80,000 req/s (rate limiting). The 500 req/s that got through would have hit Service A's throttler, which would have queued DB calls at 50 qps (throttling). The database would have responded normally, Service A would have responded normally, and the rare 429s from the throttled queue being full would have triggered the client's exponential backoff (backpressure).

Production Gotchas (From Painful Experience)

Gotcha 1: Rate Limiting by IP in a NAT World

If you rate limit by IP behind a corporate NAT, you're throttling an entire office. More importantly, Kubernetes pods share node IPs for outgoing traffic via SNAT. Rate limiting a single caller by source IP in a Kubernetes cluster will throttle all pods on that node.

Gotcha 2: The Circuit Breaker Is Not Rate Limiting

A circuit breaker (Hystrix, Resilience4j, Polly) prevents calls after a failure threshold is reached. Rate limiting prevents calls before capacity is exceeded. They serve different purposes:

• Circuit breaker: "This downstream is broken, stop calling it entirely for a while."
• Rate limiter: "This client is sending too much, slow down."

If your database is down, the circuit breaker saves you from hammering it. If your database is fine but a client is overloaded, the rate limiter saves it.

Gotcha 3: Concurrency Limits vs Rate Limits

These are different:

• Concurrency limit: "No more than 50 requests in flight at once."
• Rate limit: "No more than 100 requests per second."

A burst of 200 requests in one second has a rate of 200 req/s but a concurrency of maybe 5 (each request completes in 25ms). A rate limit would catch this; a concurrency limit would not.

You need both. Use concurrency limits to protect downstream capacity (connections, threads, file handles). Use rate limits to protect upstream throughput (API billing, database write capacity).

Gotcha 4: Retry Storms

// Jittered exponential backoff — always add randomness

func backoff(retries int) time.Duration {

base := time.Duration(math.Pow(2, float64(retries))) * time.Second

jitter := time.Duration(rand.Int63n(int64(base))) // up to 100% jitter

return base + jitter

}