Argo Rollouts canary deploys: catching a bad release before users did

It was 10:14 PM on a Tuesday when the pager went off. Our primary execution router, which processes high-throughput routing signals for our algorithmic trading desk, had just completed a rolling update to version v2.4.12.

To our Kubernetes deployment manifest, everything looked clean. The readiness probes returned HTTP 200, the pods were marked as Healthy, and the old replicaset scaled down to zero. But in our execution logs, we saw a different story.

Within four minutes of the deployment completing, our p99 latency spiked from its baseline of 12 milliseconds to 320 milliseconds. Downstream execution algorithms, expecting sub-20ms responses, began hitting their circuit breakers. Before we could manually trigger a rollback, we had suffered $14,200 in arbitrage slippage due to stale quotes.

The culprit was a memory leak in a newly introduced pricing calibration array that only manifested under production-grade concurrent request volumes—the exact kind of scale that our staging environment, despite our best efforts, could not realistically replicate without massive, cost-prohibitive load testing on every commit.

This failure exposed a glaring weakness in our CI/CD pipeline: standard Kubernetes RollingUpdate strategies are blind to application-level telemetry. If a pod compiles, starts, and passes a basic HTTP ping, Kubernetes assumes it is healthy and routes 100% of production traffic to it.

To fix this, we migrated our critical deployment paths to Argo Rollouts for canary deployment control. By leveraging progressive delivery, we now run automated analysis during our release cycles, routing a tiny slice of production traffic to new code and letting real-time Prometheus telemetry decide whether to proceed or automatically roll back.

Why Standard Kubernetes Rolling Updates Failed Us

In a native Kubernetes Deployment, you specify a update strategy like this:

When you trigger an update, Kubernetes spins up new pods, waits for their readinessProbe to pass, and kills the old ones. This model has three major architectural limitations:

1. All-or-Nothing Traffic Routing: You cannot easily route a controlled 2% or 5% of your live user traffic to the new version to observe its real-world behavior. It is a rapid transition from 0% to 100%.
2. Telemetry Blindness: The deployment engine does not consult your monitoring system (Prometheus, Datadog, etc.) during the rollout. It only checks local container runtime states.
3. No Automated Rollback on Regression: If the application crashes or degrades after the deployment completes, you must manually execute a rollback command or rely on an external operator script to intervene.

The Progressive Delivery Architecture

To implement a safer deployment model, we integrated Argo Rollouts. It replaces the standard Kubernetes Deployment object with a custom Rollout resource. The Rollout controller orchestrates the creation of stable and canary ReplicaSets, coordinates traffic splitting at the ingress layer, and executes queries against our Prometheus servers to validate system health.

Here is the data and traffic flow during an active canary deployment:

flowchart TD
 Ingress["ALB Ingress Controller"]
 ArgoRollouts["Argo Rollouts Controller"]
 StableSvc["Stable Service (90% Traffic)"]
 CanarySvc["Canary Service (10% Traffic)"]
 Prometheus["Prometheus Server"]
 Analysis["AnalysisTemplate Engine"]
 Ingress --> StableSvc
 Ingress --> CanarySvc
 ArgoRollouts --> Ingress
 Analysis -->|"Queries"| Prometheus
 Prometheus -->|"Scrapes"| CanarySvc
 Analysis -->|"Triggers Rollback"| ArgoRollouts

When we push a new image tag:
1. Argo Rollouts spins up the new version (the canary).
2. It reconfigures the ingress controller (or service mesh) to route a small, configurable percentage of traffic (e.g., 10%) to the canary service.
3. An AnalysisRun is instantiated. It polls Prometheus every minute, checking metrics like error rates and p99 latency on the canary pods.
4. If the metrics stay within safe thresholds, the canary steps up (e.g., to 50%, then 100%).
5. If any metric breaches our defined thresholds, the traffic split immediately drops back to 0% canary, routing all users to the stable ReplicaSet without human intervention.

Implementing the Canary Infrastructure

Let’s look at the concrete configuration we designed to safeguard our execution router. This setup uses Prometheus-based metrics to evaluate canary health.

1. The Application: A Simulated Latency Leak

To illustrate how we caught this, here is a simplified version of our Go/Python execution router. Under load, if a specific payload format is sent, it appends data to an unbounded global slice, degrading performance over time.

2. The Argo Rollouts Manifest

This manifest replaces our traditional Deployment. It specifies a step-by-step promotion phase: 10% traffic for 5 minutes, running background telemetry checks, before scaling up further.

3. The AnalysisTemplate: Automating Telemetry Decisions

The AnalysisTemplate defines what a healthy release looks like. We track two main SLIs (Service Level Indicators): HTTP 5xx error rates and p99 latency. If either of these metrics fails our assertions, the analysis triggers a rollback.

Catching the Bad Release in Action

To demonstrate how this infrastructure protected our environment, we initiated a deployment of our broken application code containing the memory-accumulation loop.

We applied the manifests to our cluster:

To watch the progression, we utilized the Argo Rollouts kubectl plugin:

As traffic hit our service mesh, the ingress controller split 10% of the active trading execution load to our canary pod instance. Within two minutes, the calibrate endpoint executions triggered the state-accumulation block in our Python application.

Here is the terminal output captured during the analysis phase:

The Telemetry Breakdown

Looking at the Prometheus metrics during this 2-minute window:

Because our AnalysisTemplate was configured with a failureLimit: 1 for latency, the second metric scrape immediately marked the metric as Failed. The Argo Rollouts controller picked up this state change, aborted the rollout step sequence, and instantly set the canary target weight back to 0%.

Instead of routing all 10,000 active trade connections to the degraded service and requiring a manual pager response, only 10% of our test traffic was exposed for exactly 60 seconds. The remaining 90% of trade execution requests experienced normal latency under the stable v2.4.11 version.

We managed to isolate and eliminate a catastrophic production memory leak before our core trading desk suffered any notable execution slippage.

Lessons Learned

1. Synthetic Readiness Probes Are Liars: Do not trust simple /healthz or /ready endpoints to determine code safety. They can report positive health while downstream latency or database connection pools are silently blowing up. Let live traffic and real-time metrics drive your progression decisions.
2. Analysis Windows Must Fit Your Volume: If your transaction volumes are low, 30-second metric windows will produce statistically insignificant metrics, leading to false-positive rollbacks. Scale your interval and range relative to your expected query volumes.
3. Automate Rollback Notifications: An automated rollback is great, but engineers still need to know it happened. We tied our Argo Rollout events to our slack notifier, alerting us immediately to inspect the failed image: