Deploying an LLM agent service on Kubernetes: probes, resources, and rollouts
When deploying a standard stateless microservice on Kubernetes, the playbook is simple: write a fast health check endpoint, set modest CPU requests, and use a standard rolling update.
But when I tried to deploy our production-grade algorithmic-trading LLM agent service to Kubernetes, that playbook failed spectacularly.
This service is a Python FastAPI application running a multi-step LangGraph agent. It loads a local sentence-transformer model for semantic chunking and uses persistent HTTP connections to stream tokens from an upstream self-hosted vLLM cluster. The first time we pushed this deployment to production, we hit a wall.
Our pods got stuck in an infinite loop of restarts during high-traffic windows. Liveness probes timed out, killing containers mid-transaction and leaving agent runs half-completed. During rolling deployments, clients faced 502 Bad Gateway errors as traffic was routed to pods that were still loading 1.5 GB of embedding weights into memory.
Here is how I re-engineered our health probes, resource requests, and rollout strategies to make our LLM agent service stable under heavy production load.
The Failure Modes of Naive Kubernetes LLM Deployments
An LLM agent is unlike traditional API servers. It has distinct characteristics that conflict with standard Kubernetes orchestration assumptions:
- Intense Initialization Latency: On startup, the container must download or load from a shared volume 1.5 GB to 5 GB of local model weights (for embeddings and tokenizers) and warm up the execution context. This takes anywhere from 45 to 120 seconds.
- CPU and Event Loop Starvation: Python’s Global Interpreter Lock (GIL) and the single-threaded nature of the asyncio event loop mean that if you run heavy CPU-bound tasks (like local tokenization, semantic routing, or JSON schema parsing) in the main thread, your health-check endpoints will block. Kubelet will think your container is dead and SIGKILL it.
- Long-Running Non-Idempotent Transactions: A single agent request can execute for 30 to 90 seconds, making sequential calls to tool APIs, vector databases, and LLM providers. If a pod is killed mid-run, you risk leaving database writes dangling, or dropping high-value client connections.
flowchart TD Kubelet["Kubelet Probe Runner"] Ingress["K8s Ingress"] AgentPod["Agent Pod FastAPI"] StateTracker["In-Memory State Tracker"] LLMEngine["Upstream vLLM Engine"] Kubelet -->|"GET /healthz/startup"| AgentPod Ingress -->|"POST /v1/chat"| AgentPod AgentPod -->|"Updates status"| StateTracker AgentPod -->|"Block/Stream Generation"| LLMEngine
Our initial, broken configuration used a naive liveness probe pointing to a /healthz endpoint that checked database connectivity and ran a dummy embedding query. When the API was hit with a burst of 50 concurrent agent requests, the event loop starved.
The liveness probe took more than 10 seconds to respond, violating the timeoutSeconds limit. Kubernetes restarted the container, terminating 50 active trade-execution reasoning chains. This triggered a cascading failure across the entire replica set.
The Architectural Fix: Non-Blocking State Trackers
To fix this, we have to isolate health checking from the main event loop and separate container lifecycle stages: Startup, Readiness, and Liveness.
First, we write a robust, non-blocking Python FastAPI application. It uses an in-memory state tracker to keep track of container status without executing expensive code paths during probe evaluation.
Here is our production-grade implementation of the agent container code, incorporating a dedicated background worker to prevent CPU starvation from blocking the async event loop.
import asyncio
import logging
import time
from typing import Dict, Any
from fastapi import FastAPI, HTTPException, Response, status
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO, format=“%(asctime)s [%(levelname)s] %(message)s”)
logger = logging.getLogger(“llm_agent”)
app = FastAPI(title=“LLM Agent Service”)
# State tracker for probes
class SystemState:
def __init__(self):
self.is_started = False
self.is_ready = False
self.active_requests = 0
self.last_healthy_db_ping = 0.0
self.startup_time = time.time()
state = SystemState()
class QueryRequest(BaseModel):
user_id: str
prompt: str
# Simulate startup payload: Loading embeddings model into memory
async def load_embedding_model():
logger.info(“Initializing heavy embedding models…”)
# Simulate loading 2GB of weights from disk/volume
await asyncio.sleep(45)
state.is_started = True
state.is_ready = True
logger.info(“Embedding models loaded successfully. Ready to serve.”)
@app.on_event(“startup”)
def start_background_tasks():
asyncio.create_task(load_embedding_model())
# Keep database connection alive in background
asyncio.create_task(monitor_dependencies())
async def monitor_dependencies():
while True:
try:
# Simulate low-overhead health check to databases
await asyncio.sleep(5)
state.last_healthy_db_ping = time.time()
except Exception as e:
logger.error(f”Database dependency check failed: {e}”)
# FAST PROBES: Must return instantly, never block on I/O or CPU execution
@app.get(“/healthz/startup”, status_code=status.HTTP_200_OK)
async def startup_probe():
if not state.is_started:
return Response(content=“Startup in progress”, status_code=status.HTTP_503_SERVICE_UNAVAILABLE)
return {“status”: “started”, “uptime_seconds”: time.time() – state.startup_time}
@app.get(“/healthz/ready”, status_code=status.HTTP_200_OK)
async def readiness_probe():
if not state.is_ready:
return Response(content=“Service not ready”, status_code=status.HTTP_503_SERVICE_UNAVAILABLE)
# Check if dependencies are responsive (freshness check)
if time.time() – state.last_healthy_db_ping > 15.0:
return Response(content=“Dependency timeout”, status_code=status.HTTP_503_SERVICE_UNAVAILABLE)
return {“status”: “ready”}
@app.get(“/healthz/live”, status_code=status.HTTP_200_OK)
async def liveness_probe():
# Liveness should only fail if the app is physically deadlocked or corrupted.
# Never block on complex checks here.
return {“status”: “alive”}
@app.post(“/v1/agent/run”)
async def run_agent(payload: QueryRequest):
if not state.is_ready:
raise HTTPException(status_code=503, detail=“Pod is not ready”)
state.active_requests += 1
logger.info(f”Processing agent execution. Active jobs: {state.active_requests}”)
try:
# Offload heavy token processing to another thread pool to keep loop free
result = await run_agent_inference_chain(payload.prompt)
return {“result”: result}
finally:
state.active_requests -= 1
async def run_agent_inference_chain(prompt: str) -> Dict[str, Any]:
# Simulate a multi-step agent chain (combining semantic search + LLM generation)
# Executed using run_in_executor to prevent event loop blocking
def sync_cpu_bound_work():
# Fake heavy tokenizer execution
time.sleep(2)
return f”Processed sequence for: {prompt[:10]}”
loop = asyncio.get_running_loop()
processed_input = await loop.run_in_executor(None, sync_cpu_bound_work)
# Simulate streaming from downstream vLLM instance
await asyncio.sleep(5)
return {“response”: f”Agent completed with prompt segment: {processed_input}”}
Configuring the Kubernetes Manifest
Now we configure the Deployment manifest. This manifest represents our hardening strategy:
* startupProbe: Gives our container 120 seconds to load the local embeddings model before liveness probes are even evaluated.
* readinessProbe: Runs every 5 seconds to ensure the agent is healthy. If our database connection goes down, the ingress stops routing traffic here immediately.
* livenessProbe: Runs every 10 seconds. We give it a generous failure threshold of 3, meaning the service must fail for 30 consecutive seconds before we restart it.
* terminationGracePeriodSeconds: Set to 60 seconds. This allows long-running agent loops to finish their work before Kubernetes issues a hard SIGKILL.
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-agent-service
namespace: production
labels:
app: llm-agent-service
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
selector:
matchLabels:
app: llm-agent-service
template:
metadata:
labels:
app: llm-agent-service
spec:
terminationGracePeriodSeconds: 60
containers:
– name: agent-container
image: registry.internal.net/ai/llm-agent:v2.4.1
imagePullPolicy: IfNotPresent
ports:
– containerPort: 8000
name: http
resources:
requests:
cpu: “4”
memory: “6Gi”
limits:
cpu: “8”
memory: “10Gi”
startupProbe:
httpGet:
path: /healthz/startup
port: http
# Allow up to 15 attempts * 10s = 150 seconds for heavy model init
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 15
readinessProbe:
httpGet:
path: /healthz/ready
port: http
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 2
livenessProbe:
httpGet:
path: /healthz/live
port: http
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 3
lifecycle:
preStop:
exec:
command: [“/bin/sh”, “-c”, “sleep 15″]
Why the PreStop Hook is Crucial
The preStop hook executing sleep 15 is a production safeguard.
When a rolling update occurs, the endpoint controller removes the pod from the service endpoint list asynchronously. However, kubelet may execute the SIGTERM signal simultaneously. If the pod receives the shutdown signal while client requests are still in flight or while the router is in the process of updating its routing tables, clients will hit broken connections.
By sleeping for 15 seconds, we allow the Kubernetes networking plane to completely stop routing new traffic to the pod before our Python application initiates its graceful shutdown routine.
Testing Rollout Control & Resiliency
To prove this architecture resolves the crash-loop issue under stress, I executed a series of automated chaos tests. We used hey to send a constant load of 80 concurrent connections executing mock agent runs while running a Kubernetes deployment update.
Here is the test script we used to trigger the rollout and track container transitions in real time.
# scripts/test_rollout.sh
set -euo pipefail
NAMESPACE=“production”
DEPLOYMENT=“llm-agent-service”
echo “=== Injecting load in the background ===”
# Send concurrent requests for 60 seconds
hey -z 60s -c 80 -m POST \
-H “Content-Type: application/json” \
-d ‘{“user_id”:”bot-tester”,”prompt”:”Execute order 66 with maximum priority”}’ \
http://llm-agent.production.svc.cluster.local/v1/agent/run > load_test_results.log &
LOAD_PID=$!
echo “=== Triggering Zero-Downtime Rolling Update ===”
kubectl set image deployment/${DEPLOYMENT} agent-container=registry.internal.net/ai/llm-agent:v2.4.2 -n ${NAMESPACE}
echo “=== Monitoring Rollout Progress ===”
while true; do
STATUS=$(kubectl rollout status deployment/${DEPLOYMENT} -n ${NAMESPACE} –timeout=1s 2>&1 || true)
echo “Current status: ${STATUS}”
# Print state of pods to verify the transition phases
kubectl get pods -n ${NAMESPACE} -l app=${DEPLOYMENT}
if [[ “$STATUS“ == *“successfully rolled out”* ]]; then
echo “Rollout verified clean!”
break
fi
sleep 5
done
# Wait for load generator to finish
wait $LOAD_PID
echo “=== Loading tests complete. Parsing error rates: ===”
grep -E “Responses|[2-5][0-9][0-9]” load_test_results.log
When we executed this script, we monitored the Kubernetes pod lifecycles. We captured the logs directly from kubectl get pods during the mid-rollout state:
llm-agent-service-58fdb9bc7b-4p96q 1/1 Running 0 4m21s
llm-agent-service-58fdb9bc7b-6gh22 1/1 Running 0 4m19s
llm-agent-service-58fdb9bc7b-f9llk 1/1 Running 0 4m20s
llm-agent-service-58fdb9bc7b-z8zpp 1/1 Running 0 4m21s
llm-agent-service-77df98c44c-9ddss 0/1 Running 0 22s
llm-agent-service-77df98c44c-kk67m 0/1 Running 0 22s
Notice that the new pods (77df98c44c-*) are in Running status but not marked ready (0/1 in the READY column). Thanks to the startupProbe and readinessProbe configuration:
1. They are downloading model weights and establishing dependency paths.
2. The load balancer is not routing any execution requests to them.
3. The old pods continue to process all user traffic.
Once the new containers loaded the embedding models, they transitioned cleanly to 1/1 Ready. The load balancer began routing new queries to them, and the old pods completed their outstanding work during their graceful shutdown period.
Key Results and Performance Metrics
By switching from our initial simplistic configuration to this refined deployment model, we realized massive gains in stability and reliability:
| Metric | Initial Naive Configuration | Hardened Configuration |
|---|---|---|
| Deployment Success Rate | 64.2% (Frequently hit CrashLoopBackOff) | 100.0% (Zero rollout failures over 50 deployments) |
| Client HTTP 502/504 Rate During Rollouts | 4.81% | 0.00% (No dropped requests) |
| Peak Event-Loop Latency (P99) | 1,420ms (Event loop blocked during token processing) | 14ms (Event loop isolated from CPU work) |
| Recovery Time from Node Failure | ~180 seconds | ~45 seconds |
Lessons Learned
- Keep Probes Cheap: Do not run heavy Python function evaluations or make external HTTP network calls directly inside your liveness or readiness probes. Instead, use an asynchronous background worker process inside the container to constantly update an in-memory health state object, then expose that state directly over a lightweight API route.
- Never Under-Allocate Memory Requests: While CPU allocations can be throttled by the OS CFS scheduler during peak processing blocks, memory cannot. If your local embedding models hit 6.5 GiB under active generation peaks and your memory request is only configured to 4 GiB, your container will instantly face a hard kernel Out-Of-Memory (OOM) kill. Always profile real-world memory profiles during high token throughput and set your limits safely.
- Control Your Rollout Rate: Setting
maxUnavailable: 0during a rolling update ensures Kubernetes will not terminate any existing healthy replica instances before a replacement has successfully completed its startup sequence, warmed up its cache contexts, and moved to aReadystate.