Progressive delivery of a model service with ArgoCD: canary and instant rollback

A silent failure mode in production machine learning is rarely a clean crash. It is usually a subtle drift in the feature pipeline, an unhandled NaN in a newly engineered feature, or a p99 latency spike that cascades through your execution pipeline.

Last quarter, my team deployed a updated XGBoost-based market impact model. Offline validation was pristine: AUC improved by 1.8%, and execution latency in our sandbox was a comfortable 8 milliseconds. But when we pushed it live via a standard rolling update, the real-world feature distribution caused a memory leak in our custom C++ prediction binding. Within ten minutes, the model pods hit their Kubernetes memory limits, triggered an OOMKill loop, and our automated trading engine fell back to a static heuristic model. This cost us roughly $142,000 in inefficient execution fills before we could manually run a kubectl rollout undo.

That incident forced us to redesign our deployment architecture. We realized that for critical ML models, standard Kubernetes rolling updates are too blunt. We needed progressive delivery: canary deployments where a fraction of live traffic is routed to the new model version, evaluated against real-time telemetry, and instantly rolled back without human intervention if metrics degrade.

This is how we built a zero-trust progressive delivery pipeline for our model services using ArgoCD, Argo Rollouts, and Prometheus.

The Architecture: Progressive Traffic Splitting

To run a true canary deployment for a model service, you cannot rely on simple Kubernetes Service round-robin routing. If you have 10 replica pods of your stable model and spin up 1 replica pod of your canary, you get a fixed 10% traffic split. But if you want a precise 1% or 2% canary split, you are forced to scale your stable replicas to 99, which is a massive waste of memory and GPU resources.

We decoupled traffic routing from pod replica counts using Argo Rollouts combined with an Istio Service Mesh (though an NGINX Ingress controller works similarly). This allows us to declare an exact percentage split of live HTTP/gRPC traffic at the network level, regardless of how many pods are running in our cluster.

flowchart TD
 ingress["Ingress Router"]
 activeSvc["Active Service 90%"]
 canarySvc["Canary Service 10%"]
 stablePod["Stable Model Pod v1.1.0"]
 canaryPod["Canary Model Pod v1.2.0"]
 prom["Prometheus Analyzer"]
 argocd["Argo Rollouts Controller"]

 ingress --> activeSvc
 ingress --> canarySvc
 activeSvc --> stablePod
 canarySvc --> canaryPod
 prom -->|"Scrapes Metrics"| canaryPod
 argocd -->|"Queries"| prom
 argocd -->|"Promotes or Rolls Back"| ingress

The system works as an automated feedback loop:
1. ArgoCD detects a Git commit changing the container image tag in our GitOps repository and applies the updated Rollout manifest.
2. The Argo Rollouts Controller provisions the canary pods containing the new model version.
3. The controller instructs our ingress router to route exactly 5% of production traffic to the canary pods, while 95% remains on the active stable pods.
4. An AnalysisRun is launched. This background job queries our Prometheus server every 30 seconds, executing pre-defined statistical validation queries against the canary pods.
5. If the canary metrics satisfy our criteria (e.g., error rate < 0.1%, p99 latency < 15ms) across a 10-minute window, the controller steps up the traffic to 20%, then 50%, and finally fully promotes the new version to stable.
6. If a single metric query fails the threshold, the controller instantly rewrites the routing rules back to 100% active stable traffic and terminates the canary pods. This rollback takes less than 2 seconds.

The Code: Model Service and Declarative Infrastructure

Below is the complete implementation of our resilient deployment. We start with a high-performance Python-based model service using FastAPI and Prometheus integration.

1. The Model Service Code (app.py)

This service exposes a /predict endpoint and generates simulated latency and error spikes to show how our automated rollback handles degraded models.

2. The Rollout Configuration (rollout.yaml)

This manifest replaces our standard Kubernetes Deployment. It uses the Argo Rollouts custom resource definition (argoproj.io/v1alpha1) to define our canary steps and links directly to our metric-based safety net.

3. The Analysis Template (analysis.yaml)

The AnalysisTemplate contains the concrete assertions we make against our system metrics. Here, we query Prometheus every 30 seconds. If our error rates exceed 1% or p99 latency goes over 25ms, the analysis fails, which commands Argo Rollouts to execute an immediate rollback.

4. Kubernetes Service & Ingress Config (services.yaml)

These components handle routing configuration. Our active and canary services point to the same pods under normal circumstances, but Argo Rollouts dynamically mutates our Ingress annotations to implement the exact traffic split.

The Rollout Execution and Instant Rollback

When we commit this deployment configuration to our GitOps repository, ArgoCD detects the out-of-sync status and applies our changes. Let’s look at what actually happens when we deploy version 1.2.0 with the TRIGGER_BUG=true variable active.

Live CLI Output During Deployment Failure

Applying the rollout changes immediately deploys a canary replica pod. Using the kubectl argo rollouts plugin, we can monitor the state transition live:

Because our latency hit 91 milliseconds (violating our limit of 25ms twice), the system aborted the promotion.

The broken pod was immediately stripped of its incoming traffic split and set to Terminating. Production operations suffered zero downtime because 90% of our user traffic was completely isolated from the faulty release, and the remaining 10% was instantly re-routed back to the stable pods.

Lessons Learned: Fine-tuning Canary Architectures

Implementing progressive delivery for low-latency models exposed three specific engineering problems we had to solve:

1. Cold Start Latency vs. Metric Analysis: Model pods loading weights or initializing TensorRT engines can experience extremely high latency during their first 10-20 inferences. If your AnalysisTemplate starts querying immediately, it will trigger false-positive rollbacks. To fix this, always include a healthy readinessProbe with a warm-up script that hits the /predict endpoint with dummy inputs before declaring the pod ready.
2. Prometheus Scraping Intervals: If your Prometheus server scrapes your pods every 15 seconds, and your AnalysisTemplate runs an evaluation every 10 seconds, you will query identical metric data twice. This leads to duplicate evaluation failures on the exact same error window. Ensure your AnalysisTemplate interval is at least $2\times$ your Prometheus scraping interval.
3. Database and Schema Migrations: Progressive delivery is excellent for stateless services and model parameters. If your model release depends on modifications to your feature store schema or database write structures, a rolling back can break backward compatibility. We address this by keeping our schemas strictly backward-compatible across at least two versions.