Graceful shutdown on Kubernetes: SIGTERM, preStop hooks, and dropped requests

During a high-volatility trading session last quarter, our execution system experienced a silent failure. We were running a rolling deployment of our core order-routing gateway on Kubernetes. On paper, our setup was redundant, stateless, and fronted by a highly available ingress. Yet, during the five-minute rolling update window, our API gateway logged a surge of HTTP 502 Bad Gateway and 504 Gateway Timeout errors.

We dropped 1.4% of all active client connections. For an algorithmic trading system, dropping 1.4% of order execution requests under load is not just an operational hiccup; it results in unhedged positions, broken state machines, and severe financial slippage.

The post-mortem revealed that we were mismanaging the lifecycle of our Kubernetes Pods. We had assumed that Kubernetes would naturally route traffic away from terminating containers before killing them. We were wrong.

Achieving true zero downtime deploys requires a deep understanding of the asynchronous nature of the Kubernetes control plane, the mechanics of SIGTERM, and how to orchestrate graceful shutdown sequences using preStop lifecycle hooks.

The Root Cause: Why Pods Drop Connections

When you trigger a rolling update (for example, via kubectl rollout restart), the Kubernetes control plane initiates two completely independent, concurrent workflows for the terminating Pod:

1. The Endpoint/Routing Pipeline: The control plane removes the terminating Pod’s IP address from the Endpoint (or EndpointSlice) object for the corresponding Service. This change propagates asynchronously to CoreDNS, Ingress Controllers, and the kube-proxy daemon on every node to update iptables or IPVS routing tables.
2. The Kubelet Lifecycle Pipeline: The kubelet on the node hosting the Pod changes the Pod status to Terminating and immediately sends a SIGTERM signal to the container’s PID 1 process. If the container does not exit within the default terminationGracePeriodSeconds (usually 30 seconds), the kubelet sends a SIGKILL to forcefully terminate the process.

Because these two workflows run in parallel, a race condition occurs.

flowchart TD
 API["API Gateway"] -->|"HTTP Traffic"| Pod["Target Pod"]
 K8s["K8s Control Plane"] -->|"1. Mark Terminating"| Pod
 K8s -->|"2. Update Endpoints"| KubeProxy["Kube-Proxy Nodes"]
 Pod -->|"3. Exec preStop Hook"| Sleep["Sleep 15s"]
 KubeProxy -->|"4. Remove IPVS Rules"| API
 Sleep -->|"5. Send SIGTERM"| App["App Shutdown"]

Updating iptables or IPVS rules across a large cluster can take anywhere from a few hundred milliseconds to several seconds. If your application handles the SIGTERM signal by immediately refusing new connections and shutting down its HTTP listener, it will reject requests routed to it by network components that have not yet received the updated routing table.

To prevent dropped requests, we must force the kubelet to wait until the network routing changes have propagated cluster-wide before we send the SIGTERM signal to our application.

The Architecture: Orhcestrating the Shutdown

To resolve this race condition, we implemented a multi-layered graceful shutdown strategy:

1. The preStop Delay: We use a Kubernetes container lifecycle preStop hook to block the SIGTERM signal. The hook executes a simple shell sleep (e.g., sleep 15). This forces the kubelet to wait 15 seconds before sending the SIGTERM, giving the network routing tables ample time to drop the Pod’s IP from all active endpoints.
2. The Signal Handler: Once the preStop hook finishes, the kubelet sends SIGTERM. Our application intercepts SIGTERM, stops accepting new connections, but keeps the server active to drain existing, inflight HTTP requests and WebSocket connections.
3. The Docker PID 1 Pitfall: We ensure our container is launched using the JSON/exec form in the Dockerfile (CMD ["./server"]) rather than the shell form (CMD ./server). The shell form wraps the execution in /bin/sh -c, which does not forward POSIX signals like SIGTERM to the child process.

The Code: Implementation

Here is the concrete implementation of our graceful shutdown architecture, divided into the Kubernetes manifest, the application containerization, and the application code.

1. The Kubernetes Manifest

This manifest configures a preStop lifecycle hook and extends the terminationGracePeriodSeconds to ensure the application has enough time to finish both the network propagation delay and the application-level connection draining.

2. The Application Code (Go)

Our low-latency order routing gateway is written in Go. Go’s standard library http.Server provides an out-of-the-box Shutdown(ctx) method that cleanly handles connection draining.

3. The Dockerfile

If you package your container using a shell script runner or write your ENTRYPOINT/CMD incorrect, signals will not reach your application. Here is our optimized production Dockerfile:

Verifying the Architecture Under Load

To validate that our preStop sleep hook and application signal handling successfully eliminated dropped connections, we ran a load test using k6.

We simulated a sustained traffic pattern of 5,000 requests per second (RPS) with a concurrent rollout. The rollout replaced all 5 Pods of our order-gateway application under active load.

Pre-Fix Deployment Behavior (No preStop hook, direct SIGTERM exit)

The application dropped several thousand requests because Pods terminated while upstream routers (Ingress controllers and kube-proxy instances) still directed live TCP handshakes toward their IPs.

Post-Fix Deployment Behavior (With 15s preStop hook and Go Shutdown() logic)

During the entire rolling update window, not a single HTTP connection was dropped, and there were no failed requests logged by our Ingress controller.

Key Lessons

• The Shell wrapper kills signals: Never use CMD python main.py or ENTRYPOINT ./start.sh. Always use the exec JSON list syntax: ENTRYPOINT ["/app/binary"]. If you must use a startup shell script, ensure you use exec inside the script (e.g., exec ./gateway-api) to replace the shell process PID 1 with the application process.
• Synchronize your timeouts: Keep your timeout values structured logically:
  $$\text{preStop Delay} + \text{Application Draining Timeout} < \text{terminationGracePeriodSeconds}$$
  If your preStop hook sleeps for 15 seconds, and your Go server takes up to 30 seconds to drain active connections, your terminationGracePeriodSeconds must be at least 45 seconds (we configure 60 seconds to be safe).
• Load balance correctly: If you use cloud provider network load balancers targeting Pod IPs directly via container-native load balancing, your propagation times can be significantly longer than standard iptables rules. Ensure your preStop sleep length matches your target platform’s endpoint synchronization latency.