Kubernetes Network Debugging: What `kubectl exec` Won't Tell You

# Kubernetes Network Debugging: What kubectl exec Won't Tell You

Every Kubernetes user has been here: you deploy your application, everything looks healthy (Running, Ready 1/1), but the frontend can't reach the API. You exec into the frontend pod, run curl http://api-service:8080/healthz, and it hangs. Or returns Connection refused$. Or worse — works on your laptop but not in production.

The default debugging strategy is to flail: exec into pods, install curl and ping$, make educated guesses, escalate to "must be a CNI issue," and eventually give up and restart everything.

I've debugged Kubernetes network issues across bare-metal clusters, managed EKS/GKE/AKS, and edge setups with K3s. I've seen DNS, iptables, network policies, service meshes, and CNI plugins each produce the exact same symptoms — and each requires a different diagnostic approach.

This post is a systematic field guide to those diagnostics. It assumes you know the Kubernetes networking basics (pods get IPs, services abstract pod IPs, kube-proxy programs iptables/IPVS). What it covers is what you do after the obvious checks pass.

Step 0: What You Need Before You Start

When diagnosing Kubernetes networking issues, you need certain tools. If you don't have them, get them:

The Swiss Army knife for Kubernetes networking is nicolaka/netshoot — a Docker image with every network tool you could need. Deploy it as a sidecar or standalone pod:

# Launch a disposable debug pod in your namespace
kubectl run tmp-shell --rm -it --image nicolaka/netshoot -- /bin/bash

From this pod, you have dig$, nslookup$, curl$, tcpdump$, iperf$, mtr$, and a dozen others. No apt-get. No fighting Alpine's package manager.

Every pod's /etc/resolv.conf is injected by the kubelet. It looks like this:

nameserver 10.96.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

That search line and ndots:5 are where most of the pain comes from.

1. api-service..svc.cluster.local (A record)
2. api-service.svc.cluster.local (A record)
3. api-service.cluster.local (A record)
4. api-service..svc.cluster.local (AAAA record)
5. api-service.svc.cluster.local (AAAA record)
6. api-service.cluster.local (AAAA record)

If the third one times out (because api-service.cluster.local exists as a DNS zone but has no matching record), the whole query stalls for the NXDOMAIN timeout — typically 5 seconds. That's a 30-second delay on a simple name resolution.

# In your pod spec or at the cluster level via kubelet config
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"  # Queries with fewer than 2 dots won't be qualified by search domains
  containers:
    - name: app
      image: my-app:latest

With ndots:2$, a query for api-service (no dots) gets search domain expansion. A query for api.svc.local (2 dots) gets tried as an absolute name first. The goal is to reduce the number of unnecessary lookups.

The "DNS Works Everywhere Except My App" Pattern

This is the most infuriating DNS bug: dig$ and nslookup$ resolve fine, but your app times out connecting to services by name.

The cause is almost always DNS caching at the application side. Many language runtimes and HTTP clients cache DNS results with TTLs, but they also stale cache — they'll serve a cached entry past its TTL if the resolver is momentarily unreachable. CoreDNS has a default cache TTL of 30 seconds. Your app's connection pool might hold onto a resolved IP for 5 minutes.

The diagnostic:

# From the app pod, not a debug pod
# Check what the app currently has cached
# For Go apps:
curl http://localhost:6060/debug/pprof/goroutine?debug=2 2>/dev/null | grep -i dns

# For Java apps:
jcmd $(pgrep java) VM.native_memory summary | grep -i dns

# For Node.js:
dns.getServers()

The pragmatic fix: ensure your app respects DNS TTLs and doesn't cache stale entries. In Kubernetes, pod IPs change constantly (deployments, rollouts, scaling events). A 5-minute stale DNS cache means the app resolves to a pod that may no longer exist.

// Node.js example — respect DNS TTLs
import { createConnection } from 'net';
import dns from 'dns';

// Force DNS lookups to respect TTL
dns.setServers(['10.96.0.10']); // CoreDNS cluster IP

// In your HTTP client (axios, node-fetch, fetch):
// Set connection pooling to a reasonable value
const http = require('http');
const agent = new http.Agent({
  keepAlive: true,
  keepAliveMsecs: 30000,  // 30 seconds, matching CoreDNS cache TTL
  maxSockets: 25,
});

// This ensures your connections respect the service's changing endpoints

The Domain Search Path Mutation Problem

Some CNI plugins (looking at you, Calico with certain configurations) mutate the pod's /etc/resolv.conf on the fly. The kubelet injects one thing, the CNI plugin overwrites it, and suddenly your pod can't resolve kubernetes.default.svc.

Check it:

# From within the pod
cat /etc/resolv.conf

# Compare with what kubelet thinks it should be
# On the node:
cat /var/lib/kubelet/pods/<pod-uid>/etc-hosts
# Or from kubectl:
kubectl get pod <pod-name> -o jsonpath='{.spec.dnsConfig}'

If they don't match, your CNI plugin is tampering with DNS. This is a known issue with some Calico configurations and older Flannel versions.

Step 2: Service Connectivity — Why "Connection Refused" Doesn't Mean "Down"

When curl http://api-service:8080 returns Connection refused$, most people assume the target pod is down. It's often not — the connection is reaching something that isn't listening on that port.

A Kubernetes Service is backed by an EndpointSlice (or the legacy Endpoints object). When you hit ClusterIP:Port$, kube-proxy (or your CNI's replacement for it) programs NAT rules to forward traffic to a backend pod. The key insight: if there are no ready endpoints, kube-proxy drops the packet at the iptables level.

# Check if the service has endpoints
kubectl get endpoints api-service

# Or with the newer EndpointSlice API (available since 1.21)
kubectl get endpointslice -l kubernetes.io/service-name=api-service

If ENDPOINTS is empty or shows ``<none>`, the problem is not network. It's a selector mismatch. Check your pod labels:

# Show service selector
kubectl get svc api-service -o jsonpath='{.spec.selector}'

# Show pod labels
kubectl get pods -l app=api-service --show-labels

The most common selector bug: the service uses app: api-service but the pod has app.kubernetes.io/name: api-service$. Those are different labels. No match means no endpoints.

iptables Mode vs IPVS Mode

kube-proxy supports two proxy modes. The default in most distributions is iptables$, but many managed Kubernetes clusters use ipvs$. The debugging approach differs:

Diagnose the current mode:

# On any node
kubectl -n kube-system logs daemonset/kube-proxy | grep "Using"
# Or
curl -k https://localhost:10249/proxyMode 2>/dev/null

iptables debugging

If kube-proxy is in iptables mode, the rule chain for a service looks like this:

# On a node, inspect the iptables chain for a specific service
iptables-save -t nat | grep KUBE-SVC- | grep -A 10 "<api-service>"

# The output will show:
# -A KUBE-SVC-XYZ123 -m comment --comment "default/api-service:http" -j KUBE-SEP-ABC789
# -A KUBE-SVC-XYZ123 -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-DEF456
# -A KUBE-SVC-XYZ123 -j KUBE-SEP-GHI012

# Each KUBE-SEP-* chain routes to a specific pod IP:
iptables-save -t nat | grep "KUBE-SEP-ABC789"
# Returns something like:
# -A KUBE-SEP-ABC789 -p tcp -m tcp -j DNAT --to-destination 10.244.1.15:8080
``$

If the DNAT target shows a pod IP that doesn't exist anymore (pod was deleted, replaced, but the conntrack entry is stale), you get a black hole. Restarting kube-proxy flushes conntrack, which often fixes "the application was working but now it's not" problems.

#### IPVS debugging

# List all virtual services and their real servers

ipvsadm -L -n

# Output example:

# IP Virtual Server version 1.2.1 (size=4096)

# Prot LocalAddress:Port Scheduler Flags

# -> RemoteAddress:Port Forward Weight ActiveConn InActConn

# TCP 10.96.0.1:443 rr

# -> 192.168.1.10:6443 Masq 1 0 0

# TCP 10.96.0.10:53 rr

# -> 10.244.1.5:53 Masq 1 2 5

# -> 10.244.1.6:53 Masq 1 3 4

# TCP 10.96.123.45:8080 rr

# -> # <-- NO BACKENDS!

`$

If you see $ on the real server line, kube-proxy hasn't yet populated the IPVS table for that service. Wait a moment (kube-proxy syncs approximately every 30 seconds by default), or trigger a sync:

# Trigger kube-proxy resync (if it supports the reload signal)
kubectl -n kube-system exec daemonset/kube-proxy -- kill -HUP 1
``$

---

## Step 3: Network Policies — The Silent Traffic Blocker

Network Policies are the silent killers of Kubernetes networking. They don't produce error messages. They don't show up in logs. They just drop traffic silently at the CNI level, and the sending application sees a TCP connection timeout.

### Diagnosing Policy Blocking

Most CNI plugins expose network policy state. Here's how to check them for Calico and Cilium:

# Calico — check endpoint-level policy

calicoctl get wep -n default -o wide

# Calico — check if a pod is being denied

calicoctl get policy -o wide | grep deny

calicoctl get networkpolicy -A | grep deny

# Cilium — check endpoint identity and policy verdict

kubectl -n kube-system exec daemonset/cilium -- cilium endpoint list

kubectl -n kube-system exec daemonset/cilium -- cilium policy trace --src-k8s-pod default/frontend-pod --dst-k8s-pod default/api-pod --dport 8080

`$

The cilium policy trace command is gold. It simulates the entire policy engine and tells you exactly which rule allows or denies a connection, including the CiliumIdentity and label-based logic.

# This policy denies all ingress to all pods in the namespace
# If deployed alongside an "allow specific" policy, ordering matters:
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress   # Denies ALL ingress by default
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-ingress
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: 8080

# Launch netshoot on both sides of the connection
kubectl run src --rm -it --image nicolaka/netshoot -- /bin/bash
# From src pod:
nc -zv api-service 8080
# If it hangs, try connecting directly to the pod IP:
nc -zv 10.244.1.15 8080
``$

If the ClusterIP connection fails but the pod IP connection succeeds, it's almost certainly a policy issue (or a kube-proxy issue — see Step 2).

---

## Step 4: CNI Issues — When the Network Itself Is Broken

CNI problems are less common but more catastrophic. They affect *all* pod-to-pod traffic, not just specific services.

### Major Symptoms of CNI Breakage

1. **Pods can't communicate across nodes** but work within the same node.
2. **Pod IPs overlap** because the CNI's IPAM (IP Address Management) has a bug.
3. **DNS works for some pods but not others** because the CNI handles pod DNS differently.

### Per-CNI Debugging

**Cilium:**

$

# Check Calico health
kubectl -n kube-system get pods -l k8s-app=calico-node

# Check BGP peer status
kubectl -n kube-system exec ds/calico-node -- calicoctl node status

# Check IP pool allocations
calicoctl get ippool -o wide

# Check for IP address conflicts (especially after cluster reboot)
calicoctl get ipamblock -o wide | grep -i conflict
``$

**Flannel:** Flannel is simpler, so the debug is simpler:

$

# On the node where the source pod lives
# Check if the pod IP is reachable from the node itself:
ping 10.244.1.15   # Target pod IP

# If ping fails, the issue is at the host network / CNI level
# Check routing table:
ip route show | grep 10.244

# Expected output:
# 10.244.0.0/16 via 192.168.1.10 dev eth0  # Cross-node
# 10.244.1.0/24 dev cbr0 scope link        # Local

# If the cross-node route is missing, the CNI's routing announcement is broken
# For Calico: BGP peering issue
# For Flannel: VXLAN tunnel misconfiguration
# For Cilium: eBPF program issue on the target node
``$

The key insight: **if the node can't reach the pod IP, the CNI is fundamentally broken.** No amount of pod-level debugging will fix it. You need to fix the node-level networking first.

---

## Step 5: Packet-Level Debugging — Going Full Tcpdump

When all higher-level checks pass but the app still can't connect, it's time to look at the actual packets.

This is what you do in production when nothing else works. Yes, it's invasive. Yes, you should have RBAC and audit rules. But sometimes you need to see the actual TCP handshake.

### Sidecar tcpdump with netshoot

$

# Option B: Share network namespace with the target pod
# Find the source pod:
SOURCE_POD=$(kubectl get pod frontend -o jsonpath='{.status.podIP}')

# Launch a debug pod with hostNetwork: true to see ALL traffic
kubectl run netshoot-host --rm -it --image nicolaka/netshoot   --overrides='{"spec":{"hostNetwork":true}}' -- /bin/bash

# Now tcpdump from the host network perspective
tcpdump -i any -nn port 8080 and host $(SOURCE_POD)
``$

### What to Look For on the Wire

When the connection hangs, the packet capture will show one of these patterns:

| TCP State | What It Means | Most Likely Cause |
|---|---|---|
| SYN sent, no SYN-ACK | Target isn't responding | Network policy blocking, CNI routing issue, pod not listening |
| SYN sent, RST received | Target actively rejected | Port not open on target, pod not ready (readiness probe failing) |
| SYN sent, SYN-ACK, then RST | Target accepted then rejected | App connection pool full, health check failures |
| SYN sent multiple times, no response | Packet dropped in transit | Firewall, security group, or network policy |
| SYN sent, SYN-ACK, ACK, then FIN | Connection established but immediately closed | Application protocol mismatch (e.g., HTTP talking to MySQL) |

$

# Istio — check proxy state
istioctl proxy-status
istioctl proxy-config cluster <pod-name>.<namespace>
istioctl proxy-config listener <pod-name>.<namespace>
istioctl proxy-config routes <pod-name>.<namespace>

# Linkerd
linkerd viz tap deployment/api-service -n default
linkerd check --proxy
``$

The most common service mesh network issue is **mutual TLS (mTLS) misconfiguration**. Two services that can't establish mTLS will see connection drops that look identical to network policy blocks:

$

If connections work with PERMISSIVE mode but fail with STRICT$, the issue is certificate provisioning, not networking. Check the cert-manager or Istio CSR flow.

Step 7: The Host Network — Sometimes It's Not Kubernetes

Finally, don't forget that your nodes have firewalls too. A GKE cluster with a firewall rule that blocks inter-node traffic in VXLAN's UDP port (8472 for Flannel, 4789 for VXLAN, 51820 for WireGuard in Calico) will produce exactly the same symptoms as a broken CNI.

``bash

$