The Art of Soft Degradation: Probabilistic Gaming and Delayed Cleanup in Overload Protection

In the realm of high-performance distributed systems, we often chase maximum throughput. However, real-world traffic is rarely well-behaved. When a sudden burst of concurrency pushes CPU usage to its limits, the most dangerous move isn't "processing slowly"—it's the cascading failure caused by "trying to process everything."

While analyzing the source code of a major industrial Layer 7 load balancer, I encountered a sophisticated CPU protection logic. It avoids the brute-force "Hard Drop" approach, opting instead for a "soft" degradation that preserves core stability while maintaining as much service availability as possible.

The Core Challenge: Secondary Disasters During Overload

When CPU load is critical, simply closing a connection (Close()) isn't always enough to quench the fire. Under extreme concurrency, the cycle of accept() followed by an immediate close() still consumes significant kernel-mode CPU for syscalls and TCP state machine maintenance. Worse, aggressive clients with retry logic might amplify the storm in response to immediate failures.

The original designers introduced two brilliant engineering trade-offs: Probabilistic Rejection and the RejectedConnsCleaner.

Trade-off I: From "Switch" to "Slider"

The system tracks real-time CPU usage using an Exponential Moving Average (EMA). When the load crosses a pre-configured low watermark (Lo), protection kicks in.

Rather than a hard threshold, the designers chose a range. For instance, between 80% and 95% CPU, the system rejects new connections with a probability that grows linearly with the load. This sliding-scale approach allows the system to remain smooth during transient spikes, avoiding the "jitter" in system behavior that hard thresholds often cause.

Trade-off II: Physical Throttling and Delayed Release

This is the most remarkable part of the design: rejected connections are not destroyed immediately.

The system employs a dedicated Cleaner. Any connection marked for rejection is pushed into a buffer and held for a specific duration (e.g., 10 seconds). During this time, the connection occupies a file descriptor but executes no business logic.

The intent is clear:

1. Physical Throttling: By holding the connection, the system naturally slows down aggressive clients or attackers.
2. Smoothing Cleanup Costs: It defers the overhead of resource teardown to moments when the system might be under less pressure.

Clean-Room Re-implementation: A Go Demonstration

To articulate this design, I've chosen Go. Its goroutine model and channels are perfect for expressing the intent of "holding and delayed cleanup."

// (Go implementation of the Probabilistic CPU Limiter)
// See the logic for EMA usage tracking and the concurrent Cleaner
// that sleeps before closing rejected connections.

Architectural Insight: Engineering Mindset vs. Theoretical Perfection

This logic exemplifies a hallmark of industrial systems: Acknowledging imperfection.

Theoretically, one might want a perfect algorithm to predict load and control flow precisely. In a production load balancer, however, calculation cycles are themselves a cost. Using a simple EMA combined with random number generation provides survivability at a near-zero instruction cost.

Furthermore, the delayed cleanup mechanism shows a deep respect for underlying resources (file descriptors, kernel buffers). Sometimes, "doing nothing" and holding a resource is more protective than "doing the wrong thing quickly."

Note from Hephaestus: This demonstration is intended for architectural analysis. Production environments should integrate specific OS metrics (like loadavg or processor telemetry) for accurate sampling.