cert-manager and Let’s Encrypt: TLS automation and the renewal that silently failed

At 03:14 UTC, our automated trading backend started dropping connections from three major institutional liquidity providers. In our execution logs, the errors looked like this:

This shouldn’t have happened. We were using cert-manager inside our Kubernetes clusters to automate TLS certificates via Let’s Encrypt. The certificates were supposed to rotate automatically 30 days before expiration. The deployment was configured with a standard DNS-01 challenge solver using Cloudflare.

Yet, a critical production certificate had expired.

This is the post-mortem of how our automated TLS pipeline silently failed, why our alerting missed it, and the precise configuration, Prometheus rules, and validation scripts we implemented to guarantee it never happens again.

The Failure Mode: Behind the Status “True”

We had a Certificate resource configured to secure our ingress gateway. When we ran our first diagnostic command during the outage, the output was deeply misleading:

The status was True. According to Kubernetes, the certificate was “Ready”.

However, running openssl directly against the gateway endpoint painted a completely different picture:

The live certificate had expired exactly 14 minutes prior.

The Diagnostic Trail

We started tracing the certificate generation hierarchy: Certificate -> CertificateRequest -> Order -> Challenge.

Here was the mismatch. The Certificate resource reported Ready: True because it was evaluating the existing, loaded Kubernetes Secret (which was valid when evaluated weeks ago but was now expired). Meanwhile, the pending renewal request (CertificateRequest) had been stuck in a state of failure for 29 days.

We checked the Order resources:

And finally, we inspected the failed Challenge:

Four weeks prior to the expiration, our infrastructure team had tightened RBAC policies in our Cloudflare account. They scoped down the API token used by our staging and production clusters. During this cleanup, they accidentally omitted the zone ID of our primary trading domain from the allowed resources in the production token policy.

Because cert-manager could not present the DNS-01 TXT record to Cloudflare, the Let’s Encrypt ACME challenge failed. But instead of bubbling this failure up to trigger a hard status change on the parent Certificate resource, the system kept serving the old, valid-but-aging certificate inside the target Secret.

The renewal failed silently in the background. The Certificate status remained True until the actual day of expiration, at which point the ingress controller was left serving a dead certificate.

The Corrected Architecture

To prevent this silent failure, we refactored our TLS lifecycle. We transitioned to explicit token management, implemented deep status propagation checks, and introduced an independent external validator.

flowchart TD
 cert["Certificate CRD"] -->|Triggers| certReq["CertificateRequest"]
 certReq -->|Spawns| order["Order CRD"]
 order -->|Executes| challenge["Challenge CRD"]
 challenge -->|Updates TXT| dnsAPI["Cloudflare DNS API"]
 dnsAPI -->|Validates| ca["Let's Encrypt CA"]
 ca -->|Issues Cert| certReq
 certReq -->|Rotates Secret| cert
 prometheus["Prometheus / Alertmanager"] -->|Scrapes Metrics| cert
 extCheck["Python Canary Watchdog"] -->|Direct TLS Handshake| dnsAPI

Code and Configuration

Below are the exact Kubernetes manifests, monitoring configurations, and automation scripts we deployed to resolve the failure and build absolute observability into our TLS layers.

1. The Secure and Scoped ClusterIssuer

We dropped the over-privileged legacy Cloudflare tokens and created a strictly scoped Cloudflare API token with only the following permissions:
– Zone - DNS - Edit
– Zone - Zone - Read

Here is the corrected and hardened production manifest. We explicitly separated staging and production ACME directory URLs to avoid Let’s Encrypt rate limits during testing.

Here is the corresponding Secret containing the highly restricted Cloudflare API token:

And the production Certificate resource, which now explicitly mandates renewal 30 days before expiration (renewBefore: 720h):

2. Monitoring and Alerting: Prometheus Rules

Relying on the Certificate status field was our main mistake. We had to alert directly on metrics derived from the actual certificates stored in Secret keys, and we needed to watch the failing CertificateRequest objects.

We deployed the following PrometheusRule resource to alert our team 15 days before expiration, and to flag any failed certificate request immediately.

3. External Network Canary: Independent Verification Script

To completely bypass internal Kubernetes metric misrepresentations, we implemented an external canary monitor. This Python script runs as an isolated daemon inside a secure management zone. It performs a live TLS handshake against our edge nodes, extracting the certificate dates directly from the wire.

Results and Verification

Once we updated our RBAC permissions, configured the alerts, and deployed the new ClusterIssuer, we ran a verification test.

First, we cleared the old failed resources and forced cert-manager to execute a renewal run:

We observed the controller logs to watch the execution flow:

The live verification confirmed the dynamic secret rotation worked seamlessly. We ran our Python validation script immediately after:

Our Prometheus environment picked up the new metrics. Querying certmanager_certificate_expiration_timestamp_seconds for our certificate returned the expected epoch timestamp, reflecting the full 90-day validity window.

Lessons Learned

Automated systems can create a false sense of security. To ensure zero-downtime TLS operations, we now follow these rules:

1. Never rely on high-level CRD status fields alone: The Certificate CRD reporting Ready: True is a lagging indicator that only tells you whether a Secret exists and was once successfully written. It does not mean the Secret is still valid, nor does it guarantee the background renewal process is succeeding.
2. Alert on the lower-level CRD states: Always monitor CertificateRequest and Order objects. A failing CertificateRequest is the earliest warning indicator of an impending renewal failure, triggering up to 30 days before the active certificate actually expires.
3. Verify from the outside: Keep an independent, external monitoring agent in place that performs actual TLS handshakes against your production ingress nodes. If a certificate is expired, missing, or misconfigured, this external canary will catch it regardless of what Kubernetes API objects report inside your cluster.