Running PostgreSQL on Kubernetes with the CloudNativePG operator

I run an algorithmic trading system that processes high-frequency tick-level data and serves real-time feature vectors to ML models. Historically, our database tier lived on AWS Aurora PostgreSQL. It was expensive, costing us over $1,400 a month for a relatively modest setup, but more importantly, the network latency was killing us. Our Kubernetes-based trading workers in EKS were experiencing network round-trips of 1.8ms to 3.5ms to the RDS endpoint. When loading a batch of 50,000 feature records for a backtesting run, those milliseconds snowballed into minutes.

We made the decision to migrate the database inside our Kubernetes cluster to achieve sub-millisecond, intra-cluster network latency.

Our first attempt was a naive migration using a standard StatefulSet and the Bitnami PostgreSQL Helm chart. It was a disaster. During a routine EKS node upgrade, Kubernetes drained the node running the primary database pod. The replica failed to promote cleanly because of a race condition in the recovery configuration, resulting in a split-brain scenario. Both pods thought they were the primary node, writing conflicting transactions. It took four hours of manual database surgery, WAL stitching, and database restoration from an S3 cold backup to recover.

That failure led me to re-evaluate how we manage state in Kubernetes. We needed an operator that treats PostgreSQL not as an arbitrary containerized application, but as a complex stateful system with its own clustering logic. That is when we migrated to the CloudNativePG (CNPG) operator.

The Approach: Why CloudNativePG?

Most traditional PostgreSQL operators (such as those based on Patroni) rely on an external Distributed Consensus Store (DCS) like Consul, ZooKeeper, or etcd inside the pod to manage HA (High Availability) state. This adds a massive layer of operational complexity. If your Consul cluster experiences a network partition, your database failover mechanism can break.

CloudNativePG takes a different approach. It does not use a DCS. Instead, it delegates consensus directly to the Kubernetes API server itself, using native leader election and custom resource state transitions. The operator controls the PostgreSQL instances directly via an agent process that runs inside each pod alongside the PostgreSQL engine.

flowchart TD
 CNPG["CloudNativePG Operator"]
 SVC["Service (Read/Write)"]
 P1["Primary Node (pod-1)"]
 P2["Replica Node (pod-2)"]
 P3["Replica Node (pod-3)"]
 S3["S3 WAL Archive"]
 CNPG -->|"Manages"| P1
 CNPG -->|"Manages"| P2
 CNPG -->|"Manages"| P3
 SVC -->|"Writes"| P1
 P1 -->|"Streaming Replication"| P2
 P1 -->|"Streaming Replication"| P3
 P1 -->|"WAL Archive"| S3

Here is how the architecture handles traffic and replication:
1. The Write Service always routes traffic to the designated primary pod (pod-1).
2. Replication is handled natively via PostgreSQL streaming replication (asynchronous or synchronous depending on requirements) to the replicas (pod-2 and pod-3).
3. WAL archiving runs directly out of the primary pod, pushing segments directly to an external object store (S3 or MinIO).
4. The Operator constantly reconciles the actual state of the pods with the desired declarative state. If the primary goes down, the operator handles the demotion, promotion, and routing updates automatically within seconds.

The Code: Production-Grade Infrastructure

To run this reliably in production, you need more than a default manifest. You must configure storage classes correctly, define explicit resource limits to prevent the Kubernetes Out-Of-Memory (OOM) killer from terminating your primary database, set up WAL archiving to S3, and configure a connection pooler.

1. Installing the CloudNativePG Operator

We install the operator using the official manifest. We lock this to a specific version to prevent unexpected upgrades in our CI/CD pipeline:

Verify that the operator pod is up and healthy:

Output:

2. Creating the S3 Backup Secret

Before creating our PostgreSQL cluster, we need to provision an AWS S3 bucket for WAL archiving and continuous backups. The operator requires AWS credentials to talk to S3. We create a Kubernetes secret with these details:

Apply the secret:

3. The PostgreSQL Cluster Configuration

Below is our complete, production-hardened Cluster definition. We use a 3-node topology with gp3 storage, strict CPU/Memory resource limits, physical WAL archiving to S3, and synchronous replication parameters to guarantee no data loss (RPO = 0) for transactions that complete.

Before applying this, we need to create the database password secret specified in the bootstrap.initdb.secret section:

Apply the files:

Monitor the bootstrapping process:

During initialization, you will see the operator spin up the pods sequentially to ensure primary-replica handshakes occur cleanly:

4. Setting up the Connection Pooler (PgBouncer)

Directly connecting our application pods to PostgreSQL is an anti-pattern. If a failover occurs, the application needs to locate the new primary. CloudNativePG provides a managed PgBouncer pooler that automatically targets the correct active primary node.

Apply the pooler configuration:

This creates a stable service named trading-db-pooler-rw inside the database namespace. Our applications will point to this service instead of the raw database pods.

Testing Resilience and Failover Performance

To prove that this configuration is production-ready, we must simulate a violent node failure and measure the recovery metrics. I wrote a Python script that continuously writes fake trade features to the database while we trigger a hard node failure.

The Failover Test Script

Run this script inside a temporary testing container in the same Kubernetes cluster:

While the writes are flowing, we simulate a primary node crash by deleting the primary pod (trading-db-prod-1) with a --grace-period=0 flag to prevent a clean shutdown:

Checking the Logs and Recovery Output

Here is the actual output from our db-tester pod during the simulated failure:

Let’s check the events from the CNPG operator:

The logs show that the write outage lasted approximately 4 seconds. The operator detected the node loss, evaluated which replica was the most up-to-date (trading-db-prod-2), promoted it to primary, changed the service endpoints, and PgBouncer routed the application client connections to the new primary automatically.

The Results: Real-World Performance and Savings

The migration from AWS Aurora PostgreSQL to self-hosted Kubernetes with CloudNativePG yielded immediate improvements across both our financial metrics and operational execution latencies.

By running our database directly inside our primary compute plane, we bypassed the AWS VPC Peering boundary. Read operations that used to bottleneck our model training loops now execute at the speed of internal pod networking.

Lessons Learned & Gotchas

Despite the success, this migration wasn’t without hurdles. If you deploy CNPG to production, you must avoid several critical gotchas.

1. The CPU Throttling Archiving Trap

If you enforce strict CPU limits on your operator manager sidecar, WAL compression can become bottlenecked. PostgreSQL generates WAL segments quickly under heavy write loads. If the operator’s sidecar is CPU throttled, it will fall behind in pushing WALs to S3. This causes your local PG WAL directory to balloon in size, which can quickly trigger a DiskPressure state on your Kubernetes node, evicting your database pod.
* The Fix: Never set CPU limits too low on the database containers, and assign a dedicated WAL volume (as we did in the walStorage section of our YAML above) so that WAL accumulation never starves the main database partition of space.

2. PgBouncer Transaction Pooling Limitation

PgBouncer in transaction mode is excellent for performance, but it breaks certain PostgreSQL features. Specifically, prepared statements, temporary tables, and LISTEN/NOTIFY patterns will fail or behave unpredictably.
* The Fix: If your application code heavily relies on prepared statements, you must configure PgBouncer to run in session pooling mode, or bypass PgBouncer entirely for those specific microservices by targeting the -ro (read-only) or -rw (read-write) native services exposed by the operator directly.

3. Volume Expansion Capabilities

You must ensure that the StorageClass you use (e.g., AWS EBS gp3) has allowVolumeExpansion: true configured. When you hit 80% disk utilization, CloudNativePG can automatically scale up your persistent volumes dynamically without database downtime—but only if the underlying Kubernetes storage driver supports it. Always verify your storage configuration before loading production datasets.