← Back to Blog
EN中文

The Heartbeat of Load Balancers: Pinger Strategy and Thundering Herd

In distributed systems, a load balancer is not just a traffic distributor; it's the "gatekeeper" of backend services. Through real-time Health Checks, it must swiftly detect and isolate faulty nodes to ensure high availability.

Scenario: Who Guards the Backends?

Imagine a cluster with hundreds or thousands of backend nodes. If a few go down, the load balancer must react within milliseconds to divert traffic away from them. This responsibility falls on the Pinger.

But designing a Pinger is not as simple as while(true) { ping(); sleep(); }.

Strategy Trade-offs: Steady vs. On-Demand

In the design of kernel/pinger/pinger.cpp, we see two distinct strategies:

  1. Steady Mode (Constant Probing):

    • Mechanism: Regardless of traffic, the Pinger sends probes to backends at a fixed frequency (e.g., once per second).
    • Pros: Most timely state awareness. When traffic arrives, the Pinger already knows if the backend is healthy without waiting.
    • Cons: With a massive number of backends, this generates significant overhead traffic, consuming network bandwidth and backend resources.

  2. On-Demand Mode (Probing on Need):

    • Mechanism: Normally idle; probes are triggered only when real traffic fails or after a long period of inactivity.
    • Pros: Greatly saves resources, especially for "cold" backends.
    • Cons: Failure discovery has latency. Often the first unlucky request must fail (Fail Fast) to trigger subsequent isolation.

Thundering Herd and Randomized Jitter

When thousands of Pinger instances start simultaneously (e.g., after a load balancer cluster restart), if they all send probes strictly at Interval=1s, what happens?

0th Second: All Pingers fire requests simultaneously -> Backends are instantly overwhelmed (looks like a DDoS attack). 1st Second: All Pingers fire again...

This is the so-called Thundering Herd.

The solution lies in Randomized Jitter. In our Go language simulation, we demonstrate this elegant defense mechanism:

func (p *Pinger) Start(ctx context.Context) {
    go func() {
        // Key Point: Initial Randomized Delay
        // Prevents thousands of Pingers from firing requests at the exact same moment
        jitter := time.Duration(rand.Float64() * float64(p.delay))
        time.Sleep(jitter)

        ticker := time.NewTicker(p.delay)
        // ... Enter loop
    }()
}

By introducing a random wait time between 0 ~ Interval at startup, the originally synchronous probe spikes are smoothed out across the entire time window, turning the backend load curve from sharp peaks into gentle waves.

Conclusion

The Pinger is one of the most inconspicuous yet critical components in a load balancer. Its design reflects core wisdom in distributed systems:

  • Trade-off: Choosing between timeliness (Steady) and resource consumption (On-Demand).
  • Defense: Foreseeing resonance risks as the system scales and using randomization (Jitter) to eliminate synchronization.

This is not just the art of coding, but the philosophy of system architecture.