Yuxu Ge
Article · 2025-02-25T10:00:00+00:00

The Latency vs. Consistency Trade-off in Rate Limiting: Why We Need Register-Only Mode

Rate limiting is often visualized as a bouncer at a club: check ID, check capacity, then let them in. In distributed systems, this translates to a synchronous check before processing a request. The catch: when you ask a remote quota service on every request, you add its round-trip time to your critical path.

This article explores a counter-intuitive alternative: Async Register-Only Mode. Instead of blocking requests for quota checks, allow them through and account for usage asynchronously. The result is a deliberate trade-off—sacrificing strict consistency for latency and availability.

The Hidden Cost of Synchronous Checks

In a standard synchronous rate limiter, every request must wait: Request -> Gateway -> RPC to Quota Service -> Gateway -> Backend

The Gateway cannot proceed until the Quota Service responds. If that service lives in another zone or region, you are adding meaningful network round-trip time (RTT) to every single request.

  1. Latency Penalty: A 10ms RTT to the quota service means your API's baseline latency is now 10ms + processing time. For a service targeting single-digit millisecond responses, this is prohibitive.
  2. Availability Coupling: If the Quota Service degrades, your Gateway degrades. You've made a control-plane component a hard dependency for data-plane availability.

Register-Only Mode: Act First, Account Later

The pattern inverts the flow. Instead of asking permission, the Gateway acts and reports:

  1. Process Immediately: The Gateway allows the request to proceed to the backend without waiting for quota approval.
  2. Async Registration: In parallel (via a Goroutine or buffered channel), the Gateway sends a "usage event" to the Quota Service.

The Quota Service aggregates these events and monitors consumption. Once a threshold is breached, it sends a "throttle" signal back to the Gateway. The Gateway then switches to local rejection mode for a short window.

The Trade-off Analysis

This architecture explicitly trades consistency for latency and availability.

Go Implementation: Sync vs. Async

Let's compare the two modes. Strict Mode enforces quota before allowing the request; Register-Only Mode allows the request and reports usage asynchronously.

package main

import (
	"context"
	"fmt"
	"sync"
	"time"
)

// QuotaClient simulates a remote Quota Service with network latency
type QuotaClient struct {
	mu sync.Mutex
}

// QuotaResult represents the response from the service
type QuotaResult struct {
	Allowed bool
	Latency time.Duration
}

func (c *QuotaClient) Acquire(ctx context.Context, id string) QuotaResult {
	start := time.Now()
	// Simulate Network IO: 100ms latency
	select {
	case <-time.After(100 * time.Millisecond):
		return QuotaResult{Allowed: true, Latency: time.Since(start)}
	case <-ctx.Done():
		return QuotaResult{Allowed: false, Latency: time.Since(start)}
	}
}

// Balancer simulates our Load Balancer / Gateway
type Balancer struct {
	quotaClient *QuotaClient
	// Mode toggle: true = Async Register-Only, false = Sync Strict
	registerOnly bool
}

func (b *Balancer) HandleRequest(id string) {
	if b.registerOnly {
		b.handleAsync(id)
	} else {
		b.handleSync(id)
	}
}

// Sync Mode: Block until quota is confirmed
// Downside: Latency spike, hard dependency
func (b *Balancer) handleSync(id string) {
	start := time.Now()
	// Timeout to prevent hanging indefinitely
	ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
	defer cancel()

	result := b.quotaClient.Acquire(ctx, id)
	if result.Allowed {
		fmt.Printf("[Sync]  Req %s: Quota OK (took %v), Processing...\n", id, time.Since(start))
	} else {
		fmt.Printf("[Sync]  Req %s: Rate Limited or Timeout.\n", id)
	}
}

// Async Mode: Fire-and-forget
// Upside: Zero added latency
func (b *Balancer) handleAsync(id string) {
	start := time.Now()
	
	// 1. Optimistic Execution: Process immediately
	fmt.Printf("[Async] Req %s: Allowed (Non-blocking), Processing...\n", id)

	// 2. Async Reporting: Report usage in background
	// Note: In prod, use a worker pool/buffer to avoid unbounded goroutines
	go func() {
		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
		defer cancel()
		
		_ = b.quotaClient.Acquire(ctx, id)
		// This report updates the global counter eventually
	}()

	// The user request completes in microseconds, ignoring the 100ms quota latency
	fmt.Printf("[Async] Req %s: Done (Main path latency: %v)\n", id, time.Since(start))
}

func main() {
	client := &QuotaClient{}

	fmt.Println("--- Scenario A: Sync Strict Mode (Safety First) ---")
	balancerSync := &Balancer{quotaClient: client, registerOnly: false}
	balancerSync.HandleRequest("REQ-001")

	fmt.Println("\n--- Scenario B: Async Register-Only Mode (Performance First) ---")
	balancerAsync := &Balancer{quotaClient: client, registerOnly: true}
	balancerAsync.HandleRequest("REQ-002")

	// Wait for async goroutine to finish for demo purposes
	time.Sleep(200 * time.Millisecond)
	fmt.Println("\nEnd of Demo.")
}

Results

--- Scenario A: Sync Strict Mode (Safety First) ---
[Sync]  Req REQ-001: Quota OK (took 100.12ms), Processing...

--- Scenario B: Async Register-Only Mode (Performance First) ---
[Async] Req REQ-002: Allowed (Non-blocking), Processing...
[Async] Req REQ-002: Done (Main path latency: 45µs)

The benchmark reveals the cost clearly. Sync mode incurs a 100ms penalty on every request. Async mode completes in 45µs—roughly 2000 times faster.

Practical Implications

Different systems have different tolerance for consistency loss.

The choice depends on what your backend can tolerate and what your users expect.

© 2026 Yuxu Ge ·