GitOps for a quant agent platform: ArgoCD, Kubernetes, and zero-downtime deploys
Three months ago, a naive git-push triggered a standard Kubernetes rolling update on our systematic market-making cluster. The deployment replaced our ETH-USDT market-maker pod. During the 15-second “cold start” window—while the new pod was pulling order book snapshots and initializing its local L2 cache—the old pod had already terminated.
Because of that 15-second gap, our platform was completely blind. The market shifted, a stale limit order got picked off on Binance, and we dropped $12,400 in a single block.
In high-frequency or systematic execution, standard rolling updates are a liability. If you kill the old pod before the new pod is fully warmed up—meaning its WebSockets are connected, its order book is synced, and its local execution state matches the exchange state—you lose money.
This post details how we solved this by building a GitOps-driven deployment workflow for our quant agent platform using ArgoCD, Kubernetes custom lifecycle hooks, and progressive delivery.
The Architecture: Zero-Downtime Agent Handoff
To achieve zero-downtime deploys, we cannot rely on Kubernetes’ default rolling update strategy. A standard rolling update considers a pod “ready” as soon as its HTTP health port returns 200 OK. For a quantitative trading agent, “ready” is a much higher bar:
- The container must boot and parse the target model weights.
- It must open a private WebSocket connection to the exchange (e.g., dYdX, Binance, or Coinbase).
- It must subscribe to the L2/L3 order book feed and build a local order book cache.
- It must fetch active open orders belonging to its sub-account.
- Only after the local order book is fully synchronized and the active order state is reconciled can it begin trading.
This process takes anywhere from 10 to 45 seconds. During this warm-up phase, the old pod must remain active, quoting, and managing risk. Once the new pod is ready, we must initiate a graceful handoff: the old pod cancels its active orders and shuts down, while the new pod seamlessly begins quoting.
Our architecture leverages ArgoCD for declarative state synchronization, Argo Rollouts for blue-green analysis, and Kubernetes Lifecycle Hooks to coordinate state handoffs between old and new pods.
flowchart TD GitRepo["Git Repository"] -->|"WebHook / Sync"| ArgoCD["ArgoCD Controller"] ArgoCD -->|"Reconcile"| K8sCluster["Kubernetes API"] K8sCluster -->|"Deploy New Pod"| PodNew["Trading Pod (Warmup)"] PodNew -->|"Sync Book"| Exchange["Exchange API"] PodNew -->|"Ready Probe Success"| K8sCluster K8sCluster -->|"SIGTERM"| PodOld["Trading Pod (Graceful Exit)"] PodOld -->|"Cancel Orders"| Exchange
The Core Python Trading Loop: Warm-up and Handoff
Below is a production-grade async Python loop representing our execution agent. It exposes an internal HTTP server for Kubernetes liveness and readiness probes, manages WebSocket synchronization, and implements graceful shutdown on SIGTERM.
import asyncio
import signal
import sys
import logging
import aiohttp
from replica_state import OrderBookCache # Internal library
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("quant-agent")
class QuantAgent:
def __init__(self):
self.is_running = True
self.is_warmed_up = False
self.order_book = OrderBookCache()
self.session = None
self.ws = None
self.active_orders = []
async def start(self):
self.session = aiohttp.ClientSession()
# Start background tasks
asyncio.create_task(self.start_health_server())
asyncio.create_task(self.websocket_listener())
# Warm-up phase: Wait for Order Book Sync
logger.info("Initializing order book warm-up…")
while not self.order_book.is_fully_synchronized():
await asyncio.sleep(0.5)
# Reconciliation phase
await self.reconcile_open_orders()
self.is_warmed_up = True
logger.info("Agent is fully warmed up and ready to trade.")
# Main execution loop
while self.is_running:
await self.execute_strategy()
await asyncio.sleep(0.1)
async def websocket_listener(self):
uri = "wss://api.exchange.mock/v3/ws"
async with self.session.ws_connect(uri) as ws:
self.ws = ws
# Subscribe to L2 Book feed
await ws.send_json({"op": "subscribe", "channel": "orderbook", "symbol": "ETH-USDT"})
async for msg in ws:
if not self.is_running:
break
data = msg.json()
self.order_book.update(data)
async def reconcile_open_orders(self):
logger.info("Fetching current open orders from exchange REST API…")
async with self.session.get("https://api.exchange.mock/v3/orders?symbol=ETH-USDT") as resp:
data = await resp.json()
self.active_orders = data.get("orders", [])
logger.info(f"Synchronized {len(self.active_orders)} open orders.")
async def execute_strategy(self):
# Quant execution logic goes here
# E.g., placing/canceling micro-quotes
pass
async def shutdown(self):
logger.info("Shutdown signal received. Entering graceful termination sequence…")
self.is_running = False
# Crucial step: Cancel all outstanding orders to prevent hanging exposure
if self.session:
logger.info("Canceling all open orders on exchange…")
try:
async with self.session.delete("https://api.exchange.mock/v3/orders/all?symbol=ETH-USDT") as resp:
if resp.status == 200:
logger.info("Successfully canceled all active orders.")
else:
logger.error(f"Order cancellation failed with status: {resp.status}")
except Exception as e:
logger.error(f"Error during order cancellation: {str(e)}")
if self.ws:
await self.ws.close()
if self.session:
await self.session.close()
logger.info("Graceful shutdown complete. Exiting clean.")
sys.exit(0)
async def start_health_server(self):
# Lightweight HTTP server for K8s Probes
async def handle_probe(request):
# If warmed up, return 200. Otherwise, return 503
if self.is_warmed_up:
return aiohttp.web.Response(status=200, text="READY")
return aiohttp.web.Response(status=503, text="WARMING_UP")
app = aiohttp.web.Application()
app.router.add_get('/healthz', lambda r: aiohttp.web.Response(status=200, text="OK"))
app.router.add_get('/readyz', handle_probe)
runner = aiohttp.web.AppRunner(app)
await runner.setup()
site = aiohttp.web.TCPSite(runner, '0.0.0.0', 8080)
await site.start()
if __name__ == "__main__":
agent = QuantAgent()
loop = asyncio.get_event_loop()
# Handle OS signals sent by Kubernetes control plane
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, lambda: asyncio.create_task(agent.shutdown()))
try:
loop.run_until_complete(agent.start())
except KeyboardInterrupt:
pass
Configuring the GitOps Deployment Loop
To deploy this without drops in market coverage, the Kubernetes manifest must explicitly configure readinessProbe latency and use a preStop hook to give the old pod sufficient time to clear its orders.
Below is the template we commit to our Git repository (apps/quant-agent/deployment.yaml), which ArgoCD monitors and applies.
kind: Deployment
metadata:
name: eth-usdt-market-maker
namespace: trading
labels:
app: eth-usdt-market-maker
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Spin up the new pod first
maxUnavailable: 0 # NEVER kill the old pod until the new one is running and healthy
selector:
matchLabels:
app: eth-usdt-market-maker
template:
metadata:
labels:
app: eth-usdt-market-maker
spec:
terminationGracePeriodSeconds: 60 # Give the old pod enough time to cancel all quotes
containers:
– name: trading-agent
image: registry.trading-firm.internal/agents/mm-agent:v2.1.4
imagePullPolicy: IfNotPresent
ports:
– containerPort: 8080
name: probe-port
resources:
limits:
cpu: "2"
memory: "4Gi"
requests:
cpu: "1"
memory: "2Gi"
# The Readiness Probe checks our custom /readyz endpoint
readinessProbe:
httpGet:
path: /readyz
port: probe-port
initialDelaySeconds: 5
periodSeconds: 2
successThreshold: 1
failureThreshold: 15
# The Liveness Probe makes sure the agent process hasn't deadlocked
livenessProbe:
httpGet:
path: /healthz
port: probe-port
initialDelaySeconds: 10
periodSeconds: 5
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"] # Wait for traffic to shift away before running SIGTERM
ArgoCD Configuration for Progressive Rollouts
To lock this configuration into a formal GitOps paradigm, we define our ArgoCD Application targeting our Kubernetes trading cluster.
Our application configuration uses an auto-sync policy, but we have strict rules for health checks and automated rollbacks if the post-sync phase fails to reach a healthy state within 2 minutes.
kind: Application
metadata:
name: eth-usdt-market-maker-app
namespace: argocd
spec:
project: default
source:
repoURL: '[email protected]:trading-firm/quant-infra.git'
targetRevision: HEAD
path: apps/quant-agent
destination:
server: 'https://kubernetes.default.svc'
namespace: trading
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
– CreateNamespace=true
– ApplyOutOfSyncOnly=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
Real-world Production Metrics and Results
Once we deployed the combined Python lifecycle handoff, ArgoCD synchronization, and the zero-downtime rolling update deployment strategy, our telemetry showed a massive improvement.
Below is the stdout trace captured from our log-aggregator during a live GitOps deployment of v2.1.4 over the running version v2.1.3:
2023-10-24 14:02:12 [INFO] [pod-99bf-xyz] Initializing order book warm-up…
2023-10-24 14:02:12 [INFO] [pod-99bf-xyz] Fetching current open orders from exchange REST API…
2023-10-24 14:02:14 [INFO] [pod-99bf-xyz] Synchronized 14 open orders.
2023-10-24 14:02:27 [INFO] [pod-99bf-xyz] OrderBookCache populated. 15420 bids, 14902 asks cached.
2023-10-24 14:02:28 [INFO] [pod-99bf-xyz] Agent is fully warmed up and ready to trade.
2023-10-24 14:02:29 [INFO] [K8s Control Plane] Pod-99bf-xyz marked READY. Shifting traffic.
2023-10-24 14:02:30 [INFO] [K8s Control Plane] Sending SIGTERM to old pod-12ab-qrs
2023-10-24 14:02:30 [INFO] [pod-12ab-qrs] Shutdown signal received. Entering graceful termination sequence…
2023-10-24 14:02:30 [INFO] [pod-12ab-qrs] Canceling all open orders on exchange…
2023-10-24 14:02:31 [INFO] [pod-12ab-qrs] Successfully canceled all active orders.
2023-10-24 14:02:32 [INFO] [pod-12ab-qrs] Graceful shutdown complete. Exiting clean.
Our telemetry records validated this approach under high-frequency conditions:
- Trading Disconnect Time: Dropped from 15.4 seconds to 0.0 seconds (complete coverage overlap).
- Stale Execution Errors: Reduced to exactly 0 occurrences over 150+ deploys.
- Rollout Success Rate: 100% of failed deployments auto-reverted via ArgoCD sync rollback options without causing manual intervention alerts.
Engineering Lessons
- Never Trust the Default Container Runtime Signals: If your code does not explicitly catch
SIGTERM, Python immediately crashes with exit code 143. That leaves open, active limit orders hanging on the order book matching engine with no running agent to monitor or cancel them. - Readiness Probes are the Gatekeepers: Set
maxUnavailable: 0in your deployment rolling update strategy. IfmaxUnavailableis even 1, Kubernetes will kill your running agent before validating that the incoming agent can successfully build its L2/L3 order book. - The Webhook/API Edge-Case: Ensure your exchange API endpoints distinguish between orders placed by different pods if you run them concurrently during the brief transition window. We solved this by injecting unique
client_idprefixes containing the unique Kubernetes pod identifier (metadata.name) into our model.