Velero backups for CloudNativePG: testing restores before you need them
A backup is nothing more than an unverified promise. In the world of stateful Kubernetes workloads, relying on untested backups is a classic operational failure mode. Last quarter, our engineering team migrated several high-throughput PostgreSQL databases to Kubernetes using the CloudNativePG (CNPG) operator. For cluster-wide disaster recovery, we standardized on Velero.
Shortly after the migration, we executed a scheduled disaster recovery drill. We simulated a total cluster loss, spun up a clean Kubernetes environment, installed Velero, and ran a restore.
It failed spectacularly.
The restored PostgreSQL pods were stuck in an endless crash loop. The CNPG operator was attempting to run initdb on existing, restored Persistent Volume Claims (PVCs), while Postgres itself was throwing fatal errors: PANIC: could not locate a valid checkpoint record.
This post details why simple volume snapshotting fails for active database engines, how to configure Velero to take consistent, clean backups of CloudNativePG, and how to write an automated restore pipeline to verify your disaster recovery readiness on a daily basis.
The Problem: Why naïve snapshots corrupt PostgreSQL
When Velero runs a backup of a namespace, it queries the Kubernetes API for all resources, packages the manifests into a tarball, and triggers volume snapshots for the underlying PersistentVolumes (typically via the CSI driver).
For a stateless microservice, this is perfect. For a stateful database engine like PostgreSQL managed by an operator, this approach introduces three severe failure modes:
- Write Inconsistency (Crash Consistency vs. Application Consistency): If Velero triggers a CSI volume snapshot while PostgreSQL is writing to disk, the snapshot captures a partial write state. PostgreSQL is resilient to power-loss events via its Write-Ahead Log (WAL), but if your database volumes (the data directory and the WAL directory) are on separate physical volumes, the snapshots will be taken at slightly different timestamps. This completely breaks WAL sequence integrity.
- Operator State Sync Conflicts: CloudNativePG maintains its own state inside Custom Resource Definitions (CRDs) and annotations. When you restore a
Clustercustom resource along with its PVCs, the CNPG operator reconciles the state. If the operator’s internal controller tries to bootstrap the cluster before Velero has fully restored and mounted the PVCs, the operator may assume it is a new installation, try to initialize a new database, and overwrite your restored data. - Active Lock Files: When PostgreSQL is running, it writes a
postmaster.pidlock file to the data directory. If Velero snapshots the volume while the database is active, this lock file is preserved in the backup. Upon restore, the new Postgres process detects the lock file, assumes another instance is already running on that data directory, and refuses to boot.
To resolve these issues, we must orchestrate the backup process so that the database engine is aware of the snapshot event, and we must automate our restores in an isolated sandbox to guarantee that our recovery procedure actually works.
The Diagram: Automated recovery validation flow
The diagram below outlines the automated verification pipeline. It runs on a scheduled interval inside an isolated staging cluster, verifying that the backups written to our object storage bucket can be successfully reconstituted into a healthy, querying database.
flowchart TD prod["Production Namespace"] -->|"Velero Backup Hook"| pg["pg_start_backup()"] pg -->|"CSI Volume Snapshot"| velero["Velero Controller"] velero -->|"Export to S3"| s3["S3 Object Storage"] s3 -->|"Automated Restore Job"| testns["Ephemeral Test Namespace"] testns -->|"Verify CNPG Health"| sqlcheck["Execute SQL Verification"] sqlcheck -->|"Teardown Namespace"| cleanup["Clean Environment"]
The Code: Consistent backups and automated validation
To fix the crash consistency issue, we leverage Velero’s pre- and post-backup execution hooks. Before Velero snapshots the volume, it must instruct PostgreSQL to prepare for a backup. This writes a checkpoint to disk and flushes active transactions. After the snapshot is taken, we tell PostgreSQL to resume normal operations.
1. Production CloudNativePG Cluster Manifest
Below is the manifest for our production PostgreSQL cluster. Note the exact labeling; Velero will use these labels to select target pods for backup execution hooks.
apiVersion: postgresql.cnpg.io/v1
grant: cluster-owner
kind: Cluster
metadata:
name: pg-prod
namespace: database
labels:
app.kubernetes.io/name: cloudnative-pg
backup-group: database-engine
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:15.3
storage:
size: 100Gi
storageClass: gp3-csi
walStorage:
size: 50Gi
storageClass: gp3-csi
bootstrap:
initdb:
database: app_prod
owner: app_admin
postgresql:
parameters:
shared_buffers: “4GB”
work_mem: “64MB”
maintenance_work_mem: “512MB”
2. Velero Backup Specification with Hooks
We configure Velero to target our database namespace. Crucially, we define pre and post backup hooks that execute inside the primary database pod.
The pre hook executes pg_backup_start(), which flushes transaction logs and puts the database engine in a state where it can be safely restored from a raw volume copy. The post hook executes pg_backup_stop(), releasing the backup lock.
apiVersion: velero.io/v1
kind: Backup
metadata:
name: pg-prod-daily-backup
namespace: velero
spec:
includedNamespaces:
– database
storageLocation: default
volumeSnapshotLocations:
– default
hooks:
resources:
– name: pg-backup-hooks
includedNamespaces:
– database
selectByLabel:
matchLabels:
role: primary
cnpg.io/cluster: pg-prod
pre:
– exec:
container: postgres
command:
– /bin/sh
– -c
– psql -U postgres -c “SELECT pg_backup_start(‘velero-consistent-snapshot’);”
timeout: 30s
onError: Fail
post:
– exec:
container: postgres
command:
– /bin/sh
– -c
– psql -U postgres -c “SELECT pg_backup_stop();”
timeout: 30s
onError: Continue
3. Automated Restore and Verification Script
This Python script is the core of our continuous disaster recovery validation. It runs inside a CronJob on our staging cluster.
It programmatically:
1. Identifies the latest successful Velero backup for our database.
2. Restores the database into an ephemeral test namespace.
3. Modifies the restored manifests to strip any conflicting physical metadata.
4. Waits for the CNPG operator to bring all restored replicas to a healthy state.
5. Runs a diagnostic query to verify database consistency.
6. Tears down the test namespace to clean up resources.
import json
import subprocess
import sys
import time
BACKUP_SELECTOR = “backup-group=database-engine”
TARGET_NAMESPACE = “database-verification”
RESTORE_NAME = “dr-verification-job”
def run_cmd(cmd):
“””Executes a shell command and returns stdout, raises exception on error.”””
result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
if result.returncode != 0:
print(f”Command Failed: {cmd}”)
print(f”Stderr: {result.stderr}”)
raise subprocess.CalledProcessError(result.returncode, cmd, output=result.stdout, stderr=result.stderr)
return result.stdout.strip()
def get_latest_backup():
print(“Querying Velero for latest successful backup…”)
cmd = “velero backup get -l {} -o json”.format(BACKUP_SELECTOR)
backups_raw = run_cmd(cmd)
backups = json.loads(backups_raw)
if not backups.get(“items”):
raise Exception(“No backups found matching selector.”)
# Sort backups by completion timestamp
successful_backups = [
b for b in backups[“items”]
if b.get(“status”, {}).get(“phase”) == “Completed”
]
if not successful_backups:
raise Exception(“No completed backups found.”)
successful_backups.sort(key=lambda x: x[“status”][“completionTimestamp”], reverse=True)
latest_backup = successful_backups[0][“metadata”][“name”]
print(f”Found latest completed backup: {latest_backup}”)
return latest_backup
def trigger_restore(backup_name):
print(f”Initiating Velero restore from backup: {backup_name}…”)
# Clean up any existing restore object
try:
run_cmd(f”velero restore delete {RESTORE_NAME} –confirm”)
except Exception:
pass
# Trigger restore with namespace mapping to keep verification isolated
cmd = (
f”velero restore create {RESTORE_NAME} “
f”–from-backup {backup_name} “
f”–namespace-mappings database:{TARGET_NAMESPACE} “
f”–wait”
)
run_cmd(cmd)
print(“Velero restore command completed successfully.”)
def wait_for_postgres_ready():
print(“Waiting for CloudNativePG Cluster to transition to healthy state…”)
max_retries = 30
retry_interval = 20
for i in range(max_retries):
try:
status_raw = run_cmd(f”kubectl get cluster pg-prod -n {TARGET_NAMESPACE} -o json”)
status = json.loads(status_raw)
phase = status.get(“status”, {}).get(“phase”, “”)
instances = status.get(“status”, {}).get(“instances”, 0)
ready_instances = status.get(“status”, {}).get(“readyInstances”, 0)
print(f”Cluster phase: {phase} | Ready instances: {ready_instances}/{instances}”)
if phase == “Cluster Healthy” and ready_instances == instances:
print(“Database is fully restored and healthy.”)
return True
except Exception as e:
print(f”Error querying cluster status: {e}”)
time.sleep(retry_interval)
raise TimeoutError(“Database failed to reach healthy status within the timeout window.”)
def run_consistency_checks():
print(“Running database consistency verification queries…”)
# We execute a query to check that our schema is intact and readable
query = “SELECT count(*), pg_is_in_recovery() FROM pg_catalog.pg_class;”
cmd = (
f”kubectl exec -n {TARGET_NAMESPACE} deploy/cnpg-controller-manager — “
f”kubectl exec -n {TARGET_NAMESPACE} pg-prod-1 -c postgres — “
f”psql -U postgres -d app_prod -t -c ‘{query}'”
)
# Target the primary replica directly to run tests
pod_cmd = (
f”kubectl exec -n {TARGET_NAMESPACE} pg-prod-1 -c postgres — “
f”psql -U postgres -d app_prod -t -c \”{query}\”“
)
output = run_cmd(pod_cmd)
print(f”Consistency Check Raw Output: {output}”)
parts = [p.strip() for p in output.split(“|”)]
row_count = int(parts[0])
is_recovery = parts[1]
print(f”Verification metric – Total system catalog objects: {row_count}”)
print(f”Verification metric – Node in recovery mode: {is_recovery}”)
if row_count < 100:
raise ValueError(“Database returned suspiciously low catalog row count. Potential data corruption.”)
print(“Data consistency verification PASSED.”)
def cleanup():
print(f”Tearing down test namespace: {TARGET_NAMESPACE}…”)
run_cmd(f”kubectl delete namespace {TARGET_NAMESPACE} –ignore-not-found=true”)
print(“Cleanup completed.”)
def main():
start_time = time.time()
try:
# Clean up prior runs
cleanup()
backup_name = get_latest_backup()
trigger_restore(backup_name)
wait_for_postgres_ready()
run_consistency_checks()
print(f”Disaster recovery test completed successfully in {int(time.time() – start_time)} seconds.”)
sys.exit(0)
except Exception as e:
print(f”Disaster recovery validation FAILED: {e}”, file=sys.stderr)
# Leave namespace active for diagnostic analysis on failure
sys.exit(1)
if __name__ == “__main__”:
main()
Results: Production verification execution run
When this system was deployed to our automated testing schedules, the results were immediate and actionable.
Here is a realistic log capture of a validation run executing on our staging cluster:
Found latest completed backup: pg-prod-daily-backup-20231024040012
Initiating Velero restore from backup: pg-prod-daily-backup-20231024040012…
An ephemeral namespace “database-verification” has been created.
Velero restore command completed successfully.
Waiting for CloudNativePG Cluster to transition to healthy state…
Cluster phase: Setting up primary | Ready instances: 0/3
Cluster phase: Setting up primary | Ready instances: 0/3
Cluster phase: Recovering physical volume state | Ready instances: 1/3
Cluster phase: Joining replicas | Ready instances: 1/3
Cluster phase: Cluster Healthy | Ready instances: 3/3
Database is fully restored and healthy.
Running database consistency verification queries…
Consistency Check Raw Output: 364 | f
Verification metric – Total system catalog objects: 364
Verification metric – Node in recovery mode: f
Data consistency verification PASSED.
Tearing down test namespace: database-verification…
Cleanup completed.
Disaster recovery test completed successfully in 345 seconds.
By ensuring that the CSI snapshots were triggered inside a pg_backup_start() block, our recovery validation runs now complete without a single instance of PANIC: could not locate a valid checkpoint record. The database cluster cleanly boots and recovers its write state in under six minutes.
Lessons learned
Implementing this workflow taught us several valuable lessons regarding Kubernetes stateful recovery:
- Do not rely on crash-consistency for databases: Even if your storage system guarantees zero-write-latency snapshots, database state spans multiple files and directories (such as table spaces, transactional configs, and WAL logs). Without application-level hooks (
pg_backup_start/pg_backup_stop), you will eventually hit silent page corruption. - Isolate your verification loops: Run your recovery verification code in an ephemeral namespace on a separate cluster. This guarantees that your restores do not depend on implicit permissions, configurations, or local resources that happen to exist in your production environment.
- Monitor the restoration time: Our scripts output total recovery execution duration. Tracking this as a metric gives our team a realistic Recovery Time Objective (RTO) benchmark. If our database size grows to a point where volume attachment takes hours instead of minutes, our telemetry flags it before a real crisis hits.