Real-World Kubernetes Pitfalls and How to Debug Them

This is especially common with Cilium and Calico in large clusters (50+ nodes) where CNI churn is high. The race window is milliseconds wide but it hits constantly at scale.

# Step 1: Confirm it's DNS, not application logic
# Exec into the failing pod and try a lookup
kubectl exec -it <pod> -- nslookup kubernetes.default.svc.cluster.local

# Step 2: Check the actual resolv.conf
kubectl exec <pod> -- cat /etc/resolv.conf
# Expected: nameserver pointing to kube-dns / CoreDNS (usually 10.96.0.10 or 10.43.0.10)
# Broken: empty, wrong IP, or pointing to a dead node's IP

# Step 3: Check CoreDNS itself — is it running?
kubectl get pods -n kube-system -l k8s-app=kube-dns
# If CoreDNS is crashing or has high restart count, that's a separate issue

# Step 4: Check CoreDNS logs for SERVFAIL on internal lookups
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Step 5: Test DNS from a known-good pod (a debug pod with tools)
kubectl run dns-test --image=busybox:1.36 --rm -it -- nslookup kubernetes.default

# On the node itself
cat /etc/resolv.conf
# If it points to cloud provider DNS (e.g., 169.254.169.254 for AWS),
# your pods may inherit this if CoreDNS is misconfigured

The NodeLocal DNSCache DaemonSet is the single best fix. It installs a local DNS cache on each node, eliminating the per-pod race entirely:

# Deploy NodeLocal DNSCache
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/deploy.yaml
# No, that's wrong — use the actual addon:
# https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

Pitfall #2: The OOMKilled Mystery (Pod Restarts But Memory Limits Look Fine)

# Step 1: Confirm OOMKilled
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Should return "OOMKilled"

# Step 2: Check the actual cgroup memory stats at the time of crash
# You need node-level access for this
kubectl node-shell <node>  # requires krew plugin: node-shell

# Inside the node, find the cgroup path for the dead container
# Container IDs are visible via: kubectl describe pod <pod>
cat /sys/fs/cgroup/memory/kubepods/burstable/<pod-id>/<container-id>/memory.usage_in_bytes
cat /sys/fs/cgroup/memory/kubepods/burstable/<pod-id>/<container-id>/memory.limit_in_bytes

# If you're on cgroup v2 (most modern distros):
cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<pod-id>.slice/memory.current
cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<pod-id>.slice/memory.max

# Inside the container's cgroup
grep '^cache ' /sys/fs/cgroup/memory/kubepods/burstable/<pod-id>/<container-id>/memory.stat
# If cache is > 50% of limit, page cache is your OOM
``$

**Step 4: For Java specifically, check off-heap memory:**

# Inside the container, before it crashes

# Native memory tracking

jcmd <pid> VM.native_memory summary

# Or via /proc

cat /proc/<pid>/smaps | grep -i "pss|rss" | awk '{sum+=$2} END {print sum " kB"}'

The critical Kubernetes resource monitoring insight: PromQL container_memory_working_set_bytes is closer to what the kernel counts for OOM than container_memory_rss$. If you're alerting on RSS, you're alerting too late or missing events entirely.

# Step 1: Verify it's a hairpin issue
# Check from the pod — what IPs does it resolve?
kubectl exec <pod> -- nslookup <headless-service-name>
# If the result includes the calling pod's own IP, you have potential hairpin

# Step 2: Check if the service is truly headless
kubectl get svc <service-name> -o jsonpath='{.spec.clusterIP}'
# Should be "None" for headless

# Step 3: Check which pod IP is being used when the connection hangs
# Enable debug logging in the application to log peer IPs
# Or use tcpdump:
kubectl exec <pod> -- tcpdump -ni eth0 port <peer-port> -c 20
``$

**Prevention:**

The most practical fix: **don't use headless services for pod-to-pod communication that includes self-discovery.** Use an API-based discovery mechanism (like a registration endpoint) instead.

If you must use headless:

$

apiVersion: v1
kind: Service
metadata:
  name: myapp-headless  # For peer discovery (DNS gives all IPs)
spec:
  clusterIP: None
  selector:
    app: myapp
---
apiVersion: v1
kind: Service
metadata:
  name: myapp  # For regular pod-to-pod traffic (ClusterIP load balances)
spec:
  clusterIP: 10.96.0.50
  selector:
    app: myapp
  ports:
    - port: 8080

1. StorageClass doesn't exist — The PVC references storageClassName: fast-ssd$ but no StorageClass named fast-ssd$ exists in the cluster.
2. Zone mismatch — The PV is in us-east-1a$, but the pod is scheduled on a node in us-east-1b$. EBS and GCE PDs are zonal — they can't be mounted across zones.
3. AccessModes conflict — PVC requests ReadWriteMany$, but the underlying storage only supports ReadWriteOnce$.
4. Capacity exhaustion — The CSI driver has a limit on the number of volumes per node (AWS EBS: max 40 per node on nitro instances; GCE PD: max 128 per node).

# Step 1: Why is the PVC pending?
kubectl describe pvc <pvc-name>
# Look for Events section — Kubernetes tells you exactly why
# Common messages:
#   "no persistent volumes available" → no PV matches
#   "failed to provision volume with StorageClass" → CSI driver issue
#   "waiting for first consumer to be created" → volume binding mode is WaitForFirstConsumer

# Step 2: Check the StorageClass
kubectl get storageclass
kubectl describe storageclass <name>

# Step 3: If using WaitForFirstConsumer, check pod scheduling
kubectl get pod <pod-name> -o wide
# The pod might be Pending too, waiting for a node

# Step 4: Check CSI driver pods (if using dynamic provisioning)
kubectl get pods -n kube-system | grep csi

# Step 5: Check node volume limits
kubectl describe node <node-name> | grep -A5 "Attached Volumes|Allocatable"
# Look for counts like: ephemeral-storage: 85710412Ki
# CSI-attached volumes: 25 (if this is near 40, you're at the limit)
``$

**Prevention:**

| Root Cause | Fix |
|---|---|
| Missing StorageClass | Verify `kubectl get storageclass$. Default StorageClass is set via annotation `storageclass.kubernetes.io/is-default-class: "true"$ |
| Zone mismatch | Use `WaitForFirstConsumer` binding mode — it delays volume provisioning until the pod is scheduled, ensuring zone alignment |
| AccessMode conflict | Check that your storage provider supports the mode. Most NFS/CephFS solutions support RWX. Most cloud block storage (EBS, PD) only supports RWO. |
| Node volume limit | Use `volumeAttachmentLimit` on CSI driver. Spread workloads across nodes. Use larger EBS volumes shared across multiple pods via NFS or EFS. |
| FS type mismatch | Explicitly set `fsType: ext4` in StorageClass parameters. Some CSI drivers default to `xfs$ which the pod may not handle. |

The most insidious variant: **a PVC that binds but the pod fails to mount because of a security context mismatch.** The PV is owned by root, but the pod runs as non-root:

$

The fix: add an initContainer that chowns the mount path, or configure the CSI driver to set correct permissions at provision time via csi.storage.k8s.io/pod.safe-to-evict: "true"$ and StorageClass parameters like gid: 1000$.

1. Sidecar startup race — The Envoy proxy takes longer to initialize than the application container. The app starts, tries to make an outbound connection, but Envoy isn't ready yet. Connection fails. Pod restart.

1. mTLS certificate rotation — Istio rotates certificates every 24 hours. If the SDS agent fails to refresh the certificate (DNS outage, control plane overload), mutual TLS starts failing silently — the connection is rejected, but the error message is a generic "connection reset by peer."

1. Circuit breaker cascade — A single slow upstream triggers circuit breakers across the entire mesh, causing cascading failures that look like a capacity problem but are actually a misconfigured outlier detection.

1. Protocol detection failure — The sidecar misdetects HTTP/2 as TCP and strips all the headers. Your gRPC calls fail with cryptic UNAVAILABLE errors.

# Step 1: Check if the sidecar is actually running
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}'
# Should include "istio-proxy" or "linkerd-proxy"

# Step 2: Check sidecar logs — usually more useful than app logs for mesh issues
kubectl logs <pod> -c istio-proxy --tail=50

# Step 3: Check sidecar readiness
kubectl exec <pod> -c istio-proxy -- pilot-agent request GET /ready
# Should return 200

# Step 4: Check for mTLS cert issues
kubectl exec <pod> -c istio-proxy -- openssl s_client -connect <peer>:<port> 2>&1 | grep "certificate|error"

# Step 5: Dump Envoy config to see what routes/settings are applied
kubectl exec <pod> -c istio-proxy -- curl -s http://localhost:15000/config_dump | jq '.configs[0].dynamicActiveSecrets' | head -30

# Step 6: Check for circuit breaker triggering
kubectl exec <pod> -c istio-proxy -- curl -s http://localhost:15000/stats | grep -i "circuit|outlier|eject"
``$

**Prevention:**

| Problem | Fix |
|---|---|
| Sidecar startup race | Set `holdApplicationUntilProxyStarts: true$ (Istio 1.16+). This delays the app container start until the sidecar is healthy. |
| mTLS cert rotation flake | Increase certificate TTL to 7 days in IstioConfigMap. Add a liveness probe to the sidecar. |
| Circuit breaker cascade | Start with permissive outlier detection. Never enable circuit breakers with default settings in production. |
| Protocol detection | Explicitly set `appProtocol: grpc` or `appProtocol: http2` in your Service definition. Don't let the mesh guess. |

**My opinion:** Service meshes add value at scale (200+ services, multiple teams, compliance-mandated mTLS). Below that threshold, they add complexity without proportional benefit. If you're running 20 microservices and thinking "let's install Istio," start with mutual TLS via Linkerd — it's dramatically simpler — or skip the mesh entirely and use mTLS at the application layer with a library like `spiffe/spire$.

---

## Pitfall #6: The Node That's "Healthy" But Doesn't Schedule

**Symptoms:** A node shows `Ready` status, but new pods never land on it. Or: pods get stuck in `Pending$. `kubectl describe pod` shows "0/4 nodes are available" but all nodes say Ready.

**Root Cause:** The node is Ready from the kubelet's perspective, but the scheduler's internal cache is stale. This happens when:

1. The kubelet stops heartbeating to the API server
2. The node controller hasn't marked it `NotReady` yet (default timeout is 40 seconds for `node-monitor-grace-period`)
3. The scheduler's node cache (which lists ready nodes independently) hasn't been updated
4. Node conditions are fine, but taints that would normally repel pods are still present

**Debugging workflow:**

# Step 1: Check node status — is it really ready?

kubectl get nodes

# If it shows Ready but you suspect issues, check conditions:

kubectl describe node | grep -A5 "Conditions"

# Look for: MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable

# Any of these set to True will block scheduling

# Step 2: Check the scheduler's view of the node

kubectl get pods -n kube-system -l component=kube-scheduler -o name | head -1 | xargs kubectl logs -n kube-system --tail=20

# Look for messages about "node not found in cache" or "failed to list nodes"

# Step 3: Check taints — are there taints that don't match tolerations?

kubectl describe node | grep -A10 "Taints"

# Common problematic taints:

# node.kubernetes.io/unreachable

# node.kubernetes.io/out-of-disk

# node.kubernetes.io/memory-pressure

# Step 4: Check if the kubelet is actually heartbeating

# This is the most common cause. Check node lease:

kubectl get lease -n kube-node-lease -o wide | grep

# The renewTime should be within the last 40 seconds

# Step 5: Check kubelet logs on the node

kubectl node-shell -- journalctl -u kubelet --no-pager --since "5 minutes ago" | tail -30

# Look for: "Failed to update node status", "out of disk", "eviction manager"

`$

# Remove the stale taint
kubectl taint nodes <node-name> node.kubernetes.io/unreachable:NoSchedule-
``$

---

## Pitfall #7: The ConfigMap Update That Never Arrives

**Symptoms:** You update a ConfigMap, but pods keep using the old values for hours. Even after restarting the pods, the old config persists.

**Root Cause:** Two separate issues, often confused:

**Issue A — SubPath mounts don't update:** If you mount a ConfigMap via `subPath$, the kernel doesn't support atomic symlink swapping on subPath mounts. The mounted file is a regular file, not a symlink to the auto-updating ConfigMap store. The pod will never see ConfigMap updates for subPath mounts without a restart.

**Issue B — The pod's in-memory cache is stale:** The ConfigMap file on disk is updated (the atomic symlink swap happens), but the application reads its configuration once at startup and never re-reads from disk. The file changes but the process doesn't care.

**Debugging workflow:**

$

# In your deployment YAML or Helm template
kubectl annotate deployment <name>   configmap.reloader.stakater.com/reload="<configmap-name>"
``$

Or roll your own: