HPA in practice: why CPU-based autoscaling kept scaling at the wrong times

Three months ago, our crypto market data ingestion pipeline hit a wall. During high-volatility events, execution latency spiked from 15ms to over 800ms. Orders failed, state drifted, and our downstream execution service dropped connections.

The standard solution in our Kubernetes cluster was simple: attach a Horizontal Pod Autoscaler (HPA) targeting 60% average CPU utilization to the ingestion deployment. But instead of fixing the issue, CPU-based autoscaling made it worse. The cluster scaled out after latency had already destroyed our execution pipeline, and then it scaled in right as a second wave of market volatility hit, triggering a cascade of OOMKills during pod initialization.

Here is why standard CPU-based metrics fail for event-driven, I/O-heavy workloads, and how we solved it by migrating to custom metrics driven by Prometheus and KEDA.

The problem I hit: CPU is a lagging, misleading metric

Our ingestion engine is written in Go. It maintains persistent WebSockets to eight exchanges, normalizes L2 order book deltas, and flushes stateful diffs to a Redis cluster and a Kafka topic.

When a market shock happens, market data volume jumps 10x in milliseconds. However, Go’s runtime and asynchronous network I/O model mean that CPU utilization does not spike immediately.

The standard Horizontal Pod Autoscaler configuration we relied on looked like this:

This configuration failed in three distinct ways:

1. Lagging Indicator: Go channel buffers filled up long before CPU usage crossed 60%. By the time CPU utilization registered at 65%, the Go runtime was spending most of its cycles on Garbage Collection (GC) pressure from backing up unprocessed frames. Latency was already ruined.
2. The Cold-Start Trap: When the HPA finally spun up new pods during a traffic burst, those new pods had to establish 40+ TLS sessions and pull initial order book snapshots. This snapshot restoration consumed massive CPU and memory instantly, causing fresh pods to crash with OOMKilled errors while existing pods were scaled down during minor CPU dips.
3. Flapping on I/O Wait: When downstream Redis writes blocked due to network saturation, CPU usage dropped to near 15% because threads were sitting in I/O wait state. The HPA saw low CPU and scaled the deployment down from 16 pods to 4 pods right in the middle of peak load.

We observed this failure pattern directly in kubectl describe hpa:

The deployment scaled down to 4 pods because CPU dropped to 18%, while our processing queue length exceeded 45,000 unhandled messages per pod.

My approach: Scale on lag and queue depth, not compute

We threw out native CPU/Memory target scaling and rebuilt our autoscaling architecture around two primary custom metrics:

1. Consumer Group Lag (Kafka): Unprocessed messages waiting in the topic.
2. Channel Buffer Saturation (Go Runtime): Internal queue depth exported via Prometheus /metrics.

To make the HPA act on these custom metrics without custom API aggregation glue, we deployed KEDA (Kubernetes Event-driven Autoscaling) paired with Prometheus.

Target Metric Selection

We defined a composite operational threshold. A pod should scale out if:
– The internal Go channel capacity reaches > 30% utilization for more than 5 seconds, OR
– The total Kafka topic lag for consumer group ingestor-v2 exceeds 5,000 messages.

Autoscaling Flow Architecture

flowchart LR
 A["Prometheus"] -->|"Query Metrics"| B["KEDA Operator"]
 B -->|"Evaluate Metric"| C["Custom HPA"]
 C -->|"Scale Deployment"| D["Ingestor Pods"]
 D -->|"Emit Metrics"| A

The Implementation: Go Instrumenter, Prometheus, and KEDA

1. Exposing Custom Operational Metrics in Go

First, I updated our Go application to expose real-time internal channel saturation and processing latency instead of relying on container CPU metrics.

2. Prometheus Rules for Metric Aggregation

Raw metric spikes can trigger flapping if not properly windowed. I configured a PrometheusRule to compute a rate-adjusted 1-minute metric that represents channel saturation and process latency spikes across the entire deployment.

3. KEDA ScaledObject Configuration

With KEDA installed, we bypass the basic Kubernetes HPA resource definition and create a ScaledObject. This directly maps Prometheus PromQL queries to target thresholds and includes explicit scale-down cooldown stabilization to prevent flapping during volatility cycles.

Results: End-to-End Latency Under Load

We tested the new setup during an scheduled load test designed to simulate high exchange volatility (bursting from 12,000 msgs/sec to 140,000 msgs/sec in a 3-second window).

Here is the comparison between the old CPU-driven HPA and the new Custom Metrics KEDA strategy:

When traffic surged at 14:00:00, channel saturation hit 0.34 within 2 seconds. KEDA evaluated the metric within 5 seconds, instantly doubling deployment replicas from 6 to 12 without waiting for CPU to register the workload.

Lessons Learned

1. CPU metric is a proxy, not the truth: If your application uses non-blocking I/O, async runtime loops, or internal queues, raw CPU is detached from operational reality. Scale on queue depths, channel capacity, or latency histograms instead.
2. Eliminate Scale-Up Stabilization for Fast Spikes: Set stabilizationWindowSeconds: 0 for scale-up events on volatile paths. If queue saturation breaches thresholds, you want capacity now, not in 3 minutes.
3. Control Scale-Down Aggressively: The primary cause of cascading OOMKills during market events was hasty scale-down routines. By enforcing a 300-second stabilization window and limiting scale-down steps to 10% per minute (Percent: 10), pods stay online long enough to absorb multi-wave market spikes without thrashing the cluster.
4. Instrument metric health alongside pod health: Custom metric scaling is dependent on Prometheus readiness. If Prometheus scraping lags, autoscaling stops. Always pair KEDA custom metric scaling with fallback HPA rules or Prometheus alert triggers on scrape failure.