PodDisruptionBudgets and node upgrades: upgrading Kubernetes without a maintenance window

In algorithmic trading, “maintenance windows” are a luxury we cannot afford. When your systems route orders to global financial exchanges operating across multiple time zones, taking a system offline for a cluster upgrade is out of the question. Our risk-evaluation engine and order routers must maintain $99.999\%$ uptime.

A year ago, we faced a major challenge: upgrading our production Amazon EKS node groups from Kubernetes version 1.25 to 1.26. The cluster hosted our real-time risk evaluation service, risk-engine, which requires sub-millisecond processing latencies.

Our first naive attempt in our staging environment was a disaster. We triggered a rolling node group update. The cloud provider’s automation issued node drains (kubectl drain). Because we had not configured our application lifecycle correctly, the drain process terminated all three replicas of our risk-engine almost simultaneously. For 2.4 seconds, we had zero active replicas. In production, that would have translated to over $14,000 in execution slippage and orphaned trades.

This post walks through how we solved this by designing resilient PodDisruptionBudgets (PDBs), implementing robust pre-stop termination lifecycles, and writing automated checks to prevent node-draining deadlocks.

The Eviction Lifecycle and the Node-Drain Dilemma

To upgrade a Kubernetes node, the control plane must safely evacuate all pods running on that node. This is triggered by marking the node as unschedulable (kubectl cordon) and then evicting the pods (kubectl drain).

The standard kubectl delete pod command directly terminates a pod. However, a node drain uses the Eviction API. The Eviction API is a cooperative process: it queries the API server to check if evicting a pod violates any active PodDisruptionBudget.

flowchart TD
 drain["Admin Triggers Node Drain"]
 evict["Eviction API Initiated"]
 pdb["Check PodDisruptionBudget"]
 allow["Allow Evict & Schedule New Pod"]
 block["Block Evict & Retry/Timeout"]

 drain --> evict
 evict --> pdb
 pdb -->|"Allowed"| allow
 pdb -->|"Violated"| block

If a PDB is violated, the eviction request is rejected with a 429 Too Many Requests status code. The drain tool will continuously retry the eviction until the budget allows it or the command times out.

Without a PDB, the Eviction API terminates pods instantly. If all your application replicas reside on the node being upgraded, your service goes dark. Conversely, a poorly configured PDB can completely lock up your cluster upgrade pipeline, causing node drains to hang indefinitely.

The Solution Architecture: Resilient Pods and Strict Budgets

To achieve zero-downtime upgrades, we implemented a three-tier defense system:

1. PodDisruptionBudgets: Strict rules governing how many replicas must remain online at any given second.
2. Topology Spread Constraints: Rules ensuring that pod replicas are distributed across different nodes and availability zones, preventing a single node failure or upgrade from taking down multiple instances.
3. Graceful Termination Hooks: Pre-stop hooks to let in-flight HTTP and gRPC connections drain before the container receives a SIGTERM.

Here is the production manifest we engineered for our risk-engine microservice:

Decoupling the Choices in this Manifest

• minAvailable: 3 vs. maxUnavailable: 1: With replicas: 4, both configurations mathematically mean the same thing under normal conditions (only 1 pod can be terminated at a time). However, during scale-down operations or cluster-wide disruptions, minAvailable acts as an absolute floor, guaranteeing that our algorithmic evaluation engine always has at least 3 instances active to distribute the trading load.
• topologySpreadConstraints: The kubernetes.io/hostname constraint ensures that no two replicas of our 4 pods are scheduled on the same node. If we drain a node, exactly one replica is affected.
• The preStop sleep hook: When a pod is scheduled for eviction, the endpoint controller removes the pod’s IP from the Service endpoint list, and kubelet simultaneously sends the SIGTERM to the container. Because of network latency, some kube-proxy instances on other nodes might still route traffic to the terminating pod for a few seconds. The sleep 15 ensures our application continues running and accepting incoming traffic for 15 seconds after eviction starts, avoiding dropped requests (502/503 errors).

The Danger of Deadlocks: Detecting Unsafe PDBs

While PDBs guarantee high availability, they can easily paralyze cluster upgrades. If a developer deploys a service with 1 replica and defines a PDB with minAvailable: 1, that pod can never be evicted. Any attempt to drain the node hosting this pod will hang forever, throwing errors like this:

To prevent these upgrade-blocking deadlocks, we developed a Python validation script that runs as a pre-flight check in our upgrade CI/CD pipeline. It queries the active Kubernetes API, evaluates every PDB, and flags any PDB where minAvailable matches or exceeds the current healthy replica count.

Here is the complete pre-flight validation utility:

Results and Execution Metrics

With this architecture and validation check integrated into our GitLab CI pipeline, we kicked off the upgrade of our production cluster node pools.

Our node pool upgrade process utilized a rolling strategy. We launched new nodes with the target AMI and then sequentially cordoned and drained the older nodes. Here is the output trace from our automated deployment runner executing the drain on our active nodes:

During this entire upgrade operation, we ran a continuous HTTP latency probing sequence targeting our risk-engine endpoint via an internal proxy. Below is the real latency telemetry captured during the transition of nodes:

Our telemetry confirmed:
* Zero execution drops: Out of 1.4 million requests routed during the 40-minute cluster upgrade, we recorded exactly 0 connection timeouts or 5xx responses.
* Stable Latency: Our p99 latency baseline remained solid at 1.2ms, without any jitter spikes that typically result from connection resets or sudden pod terminations.

Lessons Learned

Deploying zero-downtime microservices in Kubernetes requires understanding that infrastructure changes are a regular occurrence, not exceptional events.

• PDBs are meaningless without Topology Spread Rules: A PDB with minAvailable: 3 will fail to protect you if all 3 replicas happen to be scheduled on the same node being drained. Always pair PDBs with strict node-level anti-affinity or topologySpreadConstraints.
• The preStop hook is mandatory: Do not rely purely on your application framework’s internal SIGTERM handling. Give the Kubernetes network overlay (e.g., CoreDNS and iptables/IPVS) time to propagate endpoint removals. A basic sleep 15 in the container lifecycle is the simplest way to prevent transient connection drops.
• Pre-validate your PDB state: Add automatic checks to your CI/CD pipelines to audit your manifests and active cluster configurations. Catching a locked PDB during a pre-flight test is much easier than manually intervening during a blocked production node rollout.