TLS in Kubernetes: Why Your Certificates Are About to Expire (and How to Stop Worrying)

# TLS in Kubernetes: Why Your Certificates Are About to Expire (and How to Stop Worrying)

Every Kubernetes cluster shared the same origin story. You deployed your first service, slapped a TLS certificate on the ingress, and felt good about yourself. Six weeks later, your users see "NET::ERR_CERT_DATE_INVALID" and you're scrambling to figure out why kubectl get certificate shows "Ready: False."

I've managed clusters where certificate expiration took down production three times in six months. Not because the team was incompetent—but because nobody thought about certificate management as a first-class operational concern. We do for databases. We do for load balancers. We do for DNS. But TLS certs? "Let cert-manager handle it."

And it does—until it doesn't.

This post covers the four things I wish someone had told me before I ran my first cluster in production: choosing the right certificate strategy, configuring cert-manager without the footguns, debugging certificate failures under pressure, and monitoring certs before they expire.

Strategy One: Pick Your Poison

There are three fundamentally different approaches to TLS in Kubernetes. Your choice depends on whether you're building an internet-facing SaaS, an internal platform, or a hybrid.

I have strong opinions here: Do not use manual certificate secrets in a cluster with more than three services. I don't care if it's "just for staging." Let's Encrypt offers free certificates and cert-manager is a standard helm install away. There is no excuse for manual certificate rotation in 2026.

If you're running internal services that don't face the public internet, set up an internal CA with cert-manager's built-in CA issuer or Vault's PKI engine. They both support automated renewal. The setup cost is one afternoon. The cost of forgetting to rotate a certificate is your entire team's Friday evening.

The cert-manager Setup That Doesn't Bite You

Here's the problem with most cert-manager tutorials: they show you the happy path. They don't show you what happens when the Let's Encrypt rate limit kicks in, or when your DNS provider changes its API, or when the ACME challenge takes 45 seconds instead of 5.

The ClusterIssuer That Works

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # This email is critical—Let's Encrypt sends expiry warnings here
    email: infrastructure@rrezvin.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          # DNS-01 challenge is preferred over HTTP-01 for production
          # because it doesn't require the service to be internet-facing
          # during renewal, and it supports wildcard certificates.
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: token
        selector:
          # Only use DNS-01 for wildcard and internal domains
          dnsZones:
            - "rrezvin.com"
            - "internal.rrezvin.com"
      - http01:
          ingress:
            class: nginx
          # HTTP-01 is simpler for straightforward public endpoints
          # But beware: it requires port 80 to be accessible from the internet
---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  token: "your-api-token-here" # Use external-secrets or SOPS, not this

Critical details that will bite you:

1. Prefer DNS-01 over HTTP-01 for production. HTTP-01 requires your service to be reachable on port 80. If your cluster is behind a firewall or your DNS doesn't point to it yet, HTTP-01 fails. DNS-01 only needs API access to your DNS provider.

1. Set the email on the issuer, not the certificate. Let's Encrypt sends expiry, revocation, and rate-limit notices to this address. Make it a team mailing list, not one person's inbox.

1. Rate limits are real. Let's Encrypt allows 50 certificates per registered domain per week. If you request a cert for every subdomain individually, you'll hit this limit fast. Use wildcard certificates (*.rrezvin.com) to stay under it.

The Certificate Resource That Won't Fail Silently

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-rrezvin-com
  namespace: default
spec:
  secretName: api-rrezvin-com-tls
  # 90-day default; Let's Encrypt doesn't offer longer.
  # cert-manager auto-renews at 30 days before expiry.
  duration: 2160h # 90 days
  renewBefore: 360h # 15 days — give yourself a buffer
  commonName: api.rrezvin.com
  dnsNames:
    - api.rrezvin.com
    - api.staging.rrezvin.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  # Private key options that matter
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  # Don't let cert-manager delete the secret when certificate is deleted
  # This prevents accidental deletion of a cert that's still in use
  secretTemplate:
    labels:
      app.kubernetes.io/managed-by: cert-manager
    annotations:
      sealed: "true"
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    # Force TLS 1.3 only — no reason to support older versions in 2026
    nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.3"
    # Redirect HTTP to HTTPS
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # cert-manager will auto-provision the cert for this ingress
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - api.rrezvin.com
      secretName: api-rrezvin-com-tls
  rules:
    - host: api.rrezvin.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 443

The footgun nobody warns you about: the secretName in the spec.tls block of your Ingress must match the secretName in the Certificate resource. If they don't match, the Ingress controller won't find the cert and you'll get TLS errors. cert-manager.io/cluster-issuer annotations don't automatically create certificates for you—they create Certificate resources that need that Ingress to reference the right secret.

Renewal Gotchas: What Happens When cert-manager Breaks

cert-manager handles renewal automatically. When it works, it's invisible. When it fails, it's invisible too—until the certificate expires.

The Three Most Common Renewal Failures

1. DNS zone changes. Your DNS provider rotates their API keys, your team migrates to a new provider, or someone removes the API token from the Kubernetes secret. cert-manager can't complete DNS-01 challenges. The cert doesn't renew. You find out 30 days later when the ingress controller starts serving a self-signed fallback. How to catch it:

# Check certificate status across all namespaces
kubectl get certificates --all-namespaces

# Look for any that aren't "Ready=True"
kubectl get certificates --all-namespaces   -o jsonpath='{range .items[?(@.status.conditions[0].status!="True")]}{.metadata.namespace}/{.metadata.name}: {.status.conditions[0].message}{"
"}{end}'

# Check certificate renewal events
kubectl describe certificate api-rrezvin-com | grep -A 10 Events

# Check if the secret actually has fresh data
kubectl get secret api-rrezvin-com-tls -o json | jq -r '.data["tls.crt"]' | base64 -d |   openssl x509 -noout -enddate -subject

2. Rate limiting. You request 75 certificates for individual subdomains in one week. Let's Encrypt blocks the 51st request. cert-manager keeps retrying with exponential backoff. If the backoff exceeds the renewal window, the cert expires. Fix: Use wildcard certificates. *.rrezvin.com covers every subdomain with one certificate. 3. ACME challenge timeout. cert-manager's default ACME challenge timeout is 30 seconds. If your DNS provider takes longer to propagate records, the challenge fails. This is especially common with DNS-01 when using less common DNS providers.

# Increase the challenge timeout in the ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # ... other config
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: token
          # Increase timeout from default 30s to 60s
          # This is a per-solver config, not a top-level field
          # Actually, set this as a pod-level env var on the cert-manager controller
    # ... other solvers

Actually, the timeout is set via controller flags. Don't hunt for it in the CRD:

# For the Helm chart, pass these values:
# controller.extraArgs:
#   --acme-http01-solver-nameservers=8.8.8.8:53,1.1.1.1:53
#   --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53
#   --dns01-recursive-nameservers-only
helm upgrade --install cert-manager jetstack/cert-manager   --namespace cert-manager --create-namespace   --set installCRDs=true   --set 'extraArgs[0]=--dns01-recursive-nameservers-only'   --set 'extraArgs[1]=--dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53'

Monitoring Certificates Before They Expire

"cert-manager handles renewal" is a comforting lie. It handles renewal attempts. The attempt can fail. Here's what you need:

Prometheus Rules That Will Save Your Weekend

# prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: certificate-expiry-alerts
  namespace: monitoring
spec:
  groups:
    - name: cert-manager
      rules:
        - alert: CertificateExpiringSoon
          expr: |
            certmanager_certificate_expiration_timestamp_seconds{namespace=~".+"}
            - time() < 604800  # 7 days
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "Certificate {{ $labels.name }} in {{ $labels.namespace }} expires in less than 7 days"
            description: "Certificate {{ $labels.name }} (issuer: {{ $labels.issuer_group }}/{{ $labels.issuer_name }}) expires in {{ $value | humanizeDuration }}. Check renewal status."

        - alert: CertificateExpired
          expr: |
            certmanager_certificate_expiration_timestamp_seconds{namespace=~".+"}
            - time() < 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Certificate {{ $labels.name }} in {{ $labels.namespace }} has expired"
            description: "This will cause TLS handshake failures for any service using this certificate. Investigate immediately."

        - alert: CertManagerNotReady
          expr: |
            certmanager_certificate_ready_status{condition="True"} == 0
          for: 30m
          labels:
            severity: critical
          annotations:
            summary: "Certificate {{ $labels.name }} in {{ $labels.namespace }} is not ready"
            description: "cert-manager reports this certificate as not ready. Describe the certificate for more details."

The metric you need to watch but probably don't have: certmanager_certificate_expiration_timestamp_seconds is exported by cert-manager's controller metrics. If you don't have it in Prometheus, you're flying blind. Enable metrics on your cert-manager Helm install:

helm upgrade --install cert-manager jetstack/cert-manager   --namespace cert-manager   --set 'metrics.enabled=true'   --set 'metrics.serviceMonitor.enabled=true'   --set 'metrics.serviceMonitor.namespace=monitoring'

And here's a bash script that every infrastructure engineer should have in their back pocket—it checks every TLS secret in the cluster:

#!/bin/bash
# check-cert-expiry.sh — run this weekly in CI or cronjob

NAMESPACES=$(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}')

for ns in $NAMESPACES; do
  SECRETS=$(kubectl get secrets -n "$ns" -o jsonpath='{.items[?(@.type=="kubernetes.io/tls")].metadata.name}')
  for secret in $SECRETS; do
    DATA=$(kubectl get secret "$secret" -n "$ns" -o jsonpath='{.data.tls.crt}')
    if [ -n "$DATA" ]; then
      EXPIRY=$(echo "$DATA" | base64 -d | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
      EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null)
      NOW_EPOCH=$(date +%s)
      DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
      if [ "$DAYS_LEFT" -lt 7 ]; then
        echo "⚠️  CRITICAL: $ns/$secret expires in $DAYS_LEFT days ($EXPIRY)"
      elif [ "$DAYS_LEFT" -lt 30 ]; then
        echo "⚠️  WARNING: $ns/$secret expires in $DAYS_LEFT days ($EXPIRY)"
      else
        echo "✅ $ns/$secret: $DAYS_LEFT days remaining"
      fi
    fi
  done
done

mTLS: When One-Way TLS Isn't Enough

Mutual TLS is where most teams get overwhelmed. cert-manager handles server certificates beautifully but has no built-in concept of client identity or certificate revocation.

If you're building an internal service mesh with mTLS, you have three real options:

1. Istio (or Linkerd) with Automatic mTLS

The service mesh approach automates everything: certificate issuance, rotation, and identity binding. Each pod gets a short-lived (24-hour) SPIFFE-compliant certificate. If a pod gets compromised, the cert automatically expires.

# Istio PeerAuthentication — enforce mTLS for internal traffic
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT  # STRICT = reject non-mTLS traffic
---
# AuthorizationPolicy — granular access control
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: api-to-db
  namespace: database
spec:
  selector:
    matchLabels:
      app: postgres
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/api/sa/api-service"]
      to:
        - operation:
            ports: ["5432"]

Tradeoff: You introduce a service mesh, which is its own operational burden. Istio adds latency per hop (2-5ms for mTLS handshake on first connection). Linkerd is lighter but less feature-rich.

2. cert-manager + Custom Issuers for mTLS

If you don't want a full service mesh, cert-manager can issue client certificates. You need a CA that supports client certificates and a way to distribute private keys to pods.

# Generate a CA for internal mTLS
# Store the CA cert as a Kubernetes secret referenced by cert-manager
# Then issue client certificates per service

The problem: Distribution. How does each pod get its unique client cert and key? Sidecar containers work, but you're building infrastructure that Istio already provides.

3. Cloud-Native mTLS (GCP, AWS, Azure)

In 2026, the major cloud providers all support workload identity that includes mTLS at the VPC level. GCP's Traffic Director, AWS's VPC Lattice, and Azure's Service Mesh all offer mTLS between services without managing certificates yourself.

My recommendation: If you're building a greenfield microservices architecture, use a service mesh with automatic mTLS. Istio is the safest choice for multi-cloud. If you're on a single cloud, use the native service mesh offering and skip managing cert-manager for mTLS entirely. Only build custom mTLS distribution if you have a compliance requirement that prevents you from using either.

When Things Go Wrong: A Debugging Flowchart

You're on call. A service can't connect to another service. The error: x509: certificate has expired or is not yet valid. Here's your debug path:

1. Is the error a server cert or client cert?
   ├─ Server cert → check Ingress + Certificate resource
   │  ├─ kubectl get certificate -n <ns> → not Ready?
   │  │  ├─ kubectl describe certificate → check Events, Conditions
   │  │  └─ Fix: check ClusterIssuer, DNS, or secret reference
   │  └─ Ready but still failing?
   │     └─ Check the secret has the right cert data
   │        ├─ kubectl get secret -n <ns> <secret-name> -o yaml
   │        │  └─ Base64 decode tls.crt, check with openssl
   │        └─ Also check: Ingress secretName matches Certificate secretName
   │
   └─ Client cert → mTLS issue
      ├─ Are you using a service mesh? → Check mTLS mode (PERMISSIVE vs STRICT)
      ├─ Custom cert-manager mTLS? → Check if the client cert Secret exists and is valid
      └─ Is the client cert in a different namespace?
         └─ Check RBAC: does the client have access to the secret's namespace?

The most common fix for "but it was working yesterday":

# Restart the cert-manager pod to retry cert issuance
kubectl rollout restart deployment/cert-manager -n cert-manager

# Sometimes the webhook is stale — delete the Certificate to force re-creation
# (This doesn't delete the Secret if you have secretTemplate configured)
kubectl delete certificate api-rrezvin-com

# cert-manager will recreate it from the Ingress annotation
# In just a few seconds (usually)

The Playbook

Here's the tl;dr — the playbook I'd give any team running Kubernetes today:

1. Use cert-manager with DNS-01 challenges and wildcard certificates. It handles 95% of use cases. Set the email on the ClusterIssuer to your team mailing list.

1. Monitor certificate expiry with Prometheus. The certmanager_certificate_expiration_timestamp_seconds metric is non-negotiable. Set alerts at 14 days and 7 days before expiry.

1. Set renewBefore to 15 days, not the default 30 days. You need buffer if the ACME challenge fails. 15 days gives you time to fix DNS issues without emergency paging.

1. Use ECDSA P-256 keys, not RSA 2048. Smaller certificates mean faster TLS handshakes. On Kubernetes, this matters because each pod-to-pod connection involves a new handshake. ECDSA handshake is ~50% faster than RSA at equivalent security.

1. For mTLS, use Istio or Linkerd. Don't build your own. The certificate distribution, rotation, and identity binding are harder than they look.

1. Keep a local copy of your CA cert. When something breaks and you can't reach the cluster API, you need to verify server certs offline. Store the CA cert in your password manager and source control.

1. Run the cert-check script weekly as a CronJob inside the cluster. Not as a CI job. If CI is down, you still want to know about expiring certs.

Conclusion

TLS certificate management isn't hard. It's just fiddly. The defaults will burn you—most default renewal windows, challenge timeouts, and monitoring are designed for demos, not production. But with cert-manager, a few thoughtful configuration choices, and proactive monitoring, certificate expiration becomes a non-event rather than a 3 AM emergency.

The cluster doesn't care about your certificates. It's up to you to care.

— Technical content team at Rrezvin