Resource requests and limits: the tuning pass that cut our node count 30%

Our algorithmic execution engine was costing us $14,800 a month in raw EC2 compute across three Kubernetes clusters. The irony was painful: our average CPU utilization across the fleet hovered at a miserable 12%, yet we were constantly battle-fighting alerts. During periods of high market volatility, pods would randomly drop dead with OOMKilled statuses, or our P99 order-routing latency would spike from a clean 4ms to a catastrophic 450ms.

We were trapped in a common Kubernetes anti-pattern. Out of fear of production instability, our engineers had set massive, arbitrary resource requests and limits on our deployments. The Kubernetes scheduler looked at those inflated requests, assumed the nodes were full, and spun up more EC2 instances. We had massive quantities of “slack”—allocated but entirely idle capacity—while still suffering from performance degradation under load.

To fix this, we ran a systematic rightsizing pass. By writing our own metric-extraction pipelines and refactoring how we configure container resources, we cut our node count by over 30%, stabilized our P99 latencies, and took a massive bite out of our infrastructure bill.

Here is exactly how we analyzed the cluster, the failure modes we hit, and the automation we built to keep our resource footprint lean.

flowchart TD
 feed["Market Data Feed"] --> ingester["Ingestion Pods"]
 ingester --> queue["Kafka Message Bus"]
 queue --> execution["Execution Engines"]
 execution --> prometheus["Prometheus Metrics"]
 prometheus --> analyzer["Rightsizing Script"]

The Core Problem: Request vs. Limit Misconceptions

To understand how we ended up over-provisioned, you have to look at how the Kubernetes scheduler handles resources.

• requests are what the scheduler uses to place pods. If a pod requests 2 CPUs, the scheduler will only place it on a node with at least 2 unallocated CPUs. This is a contractual guarantee.
• limits are the hard ceiling. If a pod attempts to use more memory than its limit, the kernel out-of-memory (OOM) killer terminates the process. If it tries to use more CPU than its limit, the CFS (Completely Fair Scheduler) bandwidth control throttles its CPU cycles.

Our team had fallen into the trap of setting requests equal to limits at incredibly high levels (e.g., requesting 4 CPUs and 8GiB of RAM for an ingestion worker that averaged 150m CPU and 800MiB RAM). We did this “just in case” the worker needed to catch up on a backlog. The result? Nodes were fully allocated according to the scheduler, forcing cluster autoscaler to spin up new nodes, while the actual physical hardware sat idle.

The Analyzer: Writing the Script to Find the Waste

Rather than relying on third-party SaaS tools that charge a premium to tell us we are wasting money, we wrote a Python script to query our Prometheus metrics endpoint directly. We wanted to analyze the delta between the 95th percentile of actual resource usage and the configured requests/limits over a 14-day trailing window.

Here is the tool we built to extract this data. It uses the Prometheus query API to calculate the exact headroom of every container in our namespaces.

When we ran this, the output was glaring. Here is a sample from our live cluster terminal:

We were provisioning 4 full CPU cores for the order router, yet at the 95th percentile, it was only using 0.45 of a core. Multiply this across dozens of replicas, and we were burning hundreds of dollars daily on idle virtual machines.

Real-Time Ad-Hoc Audits via CLI

For quick validation during deployments, we wrote a fast shell pipeline using kubectl and jq to instantly surface pods that have a massive discrepancy between their declared request and actual live usage.

The Rightsizing Strategy: Breaking the Limits Safely

When we set out to apply kubernetes cost optimization, we couldn’t just slash every configuration. We learned this the hard way on day two of our refactoring pass.

Failure 1: The Go Runtime GOMAXPROCS Trap

We scaled down our Go-based order-routing engine requests and limits from 4.0 CPUs down to 0.5 CPUs. Within minutes, the P99 execution latency of the order router skyrocketed.

Why? Go’s runtime uses the environment variable GOMAXPROCS to determine how many OS threads can execute user-level Go code simultaneously. By default, Go queries the container’s visible CPUs. In Kubernetes, unless specified, Go sees all the CPUs on the host node, not the container’s CPU limit. It spun up dozens of threads.

Because we restricted the container’s CPU limits to 0.5, the kernel’s CFS scheduler relentlessly throttled these threads within its 100ms quota period. The threads were constantly waiting for CPU cycles, causing latency spikes.

The Fix: We integrated Uber’s automaxprocs library into our main entry points, which reads the container’s actual CPU limits from cgroups and configures GOMAXPROCS dynamically:

Failure 2: The Python Memory Fragmentation / Malloc Trap

Our Python market-data parsers were eating memory. We tried to set a lean memory request of 512MiB and a limit of 1GiB. Every few hours, during bursty market activity, the pods would get hit with an OOMKilled event.

When we inspected the profiling data, we realized that memory usage wasn’t actually leaking; it was fragmenting. CPython’s allocator doesn’t release memory back to the OS immediately. When processing large JSON payloads, it allocated thousands of temporary objects. The OS virtual memory manager kept allocating pages, hitting our limit before Python’s garbage collector had freed up virtual memory fragments back to the kernel.

The Fix: We changed our memory allocation library from the standard glibc malloc to Google’s jemalloc, which handles heap fragmentation far more aggressively, and configured MALLOC_ARENA_MAX:

This single environment change decreased our peak memory footprints by up to 40% without changing a single line of python code, preventing OOMKills even with restricted memory limits.

The Before and After Configurations

After gathering actual profile data and mitigating application-level quirks, we rewrote our Helm values. We decoupled requests and limits completely:

• CPU: We set requests based on the average typical workload, and did not set any limits. This prevents CFS throttling and allows containers to burst up to the node capacity when necessary, provided other workloads aren’t starved.
• Memory: We set requests to match the actual historical 90th percentile usage, and limits to 1.3x that value to allow breathing room for temporary memory spikes without immediately triggering the OOM killer.

Before: Over-provisioned Helm values

After: Optimized and safe Helm values

Results: The Real Impact on the Fleet

After rolling out these changes incrementally across our development, staging, and production environments, we monitored our metrics closely over a two-week period.

By shedding the artificial CPU requests, the Kubernetes scheduler was able to pack our pods much more efficiently onto the existing fleet. The Cluster Autoscaler responded by safely terminating 14 redundant EC2 instances.

Crucially, removing CPU limits on our performance-critical components completely resolved our latency spikes. When trade volumes surge, our pods can now burst into the unreserved CPU capacity of the host nodes without the CFS scheduler stepping in to artificially throttle their cycles.

Key Lessons for Scaling Rightsizing

1. Never set CPU limits on latency-critical applications. Unless you have a multi-tenant cluster where noisy neighbors are a existential threat, setting CPU limits is a recipe for CFS scheduling latency. Set realistic CPU requests to guarantee your performance base-line, and let containers burst freely.
2. Set memory limits with a buffer. Memory is a non-compressible resource. Unlike CPU, which degrades gracefully when starved, running out of memory means instantaneous death. Keep a 30-50% buffer between your memory request and your limit.
3. Bind runtime settings to cgroups limits. If you scale down CPU resources, ensure your runtime engines (Go, Java, .NET, Node.js) are explicitly aware of the limits using tools like automaxprocs or custom memory flags (-XX:ActiveProcessorCount for JVM). Failing to do so leads to thread pools choking on starved virtualized hardware.