DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD
INCIDENT RESOLUTION · 24 MIN READ
Container networking logo

Container Networking Troubleshooting: CNI, Packet Drops & DNS Runbook

A production SRE troubleshooting runbook for container and Kubernetes networking: dissecting packet drops, CNI plugin failures (Calico, Cilium), CoreDNS resolution latency, conntrack table exhaustion, and overlay VXLAN encapsulation.

By DevOps Tribune Engineering Staff · Published November 17, 2026

In containerized production clusters, networking failures are notoriously insidious. When a microservice suddenly encounters intermittent 504 Gateway Timeouts, 5-second database connection delays, or silent TCP resets, the root cause is rarely found in the application source code. Instead, the failure resides deep within the container networking virtualization stack: a saturated Linux kernel connection tracking (conntrack) table silently dropping SYN packets, an MTU mismatch causing VXLAN encapsulation packet fragmentation, CoreDNS ndots:5 search domain storms overwhelming DNS pods, or an errant default-deny NetworkPolicy silently blackholing inter-namespace communications.

Because container networking abstracts physical network interfaces across virtual ethernet (veth) pairs, Linux network namespaces, virtual software bridges, iptables / IPVS packet transformation chains, eBPF programs, and overlay tunneling protocols (such as VXLAN and Geneve), traditional networking diagnostic intuition often fails. A developer pinging an IP address from their local laptop or attempting to traceroute a Kubernetes ClusterIP will receive misleading results, because ClusterIPs are virtual IP constructs that exist only as packet manipulation rules in the kernel memory of worker nodes.

This comprehensive engineering runbook provides a methodical, layer-by-layer troubleshooting framework for diagnosing and resolving container networking failures. We unpack the Linux container networking model, compare Container Network Interface (CNI) architectures, analyze conntrack table saturation, debug CoreDNS latency traps, resolve cross-namespace security policy blocks, and provide an essential CLI toolkit for capturing and dissecting container packet flows under production pressure.


1. The Container Networking Model: From veth Pairs to Overlay Encapsulation

To effectively troubleshoot container communications, engineers must understand how packets traverse the operating system from a process inside a container to the physical network card:

1. The Virtual Ethernet (veth) Pair

Every container exists inside its own isolated network namespace (created via the Linux clone(CLONE_NEWNET) system call). The container namespace has its own loopback interface, IP routing table, and firewall filters. To connect this isolated namespace to the host operating system, the CNI plugin provisions a veth pair—a virtual bidirectional pipe. One end of the pipe resides inside the container namespace (typically named eth0), while the opposite peer end remains in the root host namespace (named something like veth1234abcd or cali1234abcd).

2. Host Bridging vs Point-to-Point Routing

In standard Docker bridge networking (docker0), all host-side veth endpoints are connected to a software bridge operating like a Layer 2 switch. In modern Kubernetes CNIs, bridging is often replaced with direct Layer 3 host routing: each node is allocated a distinct Pod CIDR subnet (e.g., 10.244.1.0/24), and the host Linux kernel acts as a router forwarding packets directly into container veth interfaces.

3. Overlay Tunneling (VXLAN / Geneve) vs Flat Routing (BGP / AWS VPC CNI)

When Pod A on Node 1 (IP: 10.244.1.15) communicates with Pod B on Node 2 (IP: 10.244.2.42), the packet must cross the physical network separating the nodes. CNIs handle cross-node transit using two contrasting paradigms:

  • Overlay Networks (Flannel, Calico VXLAN, Cilium VXLAN): The original Layer 2/Layer 3 packet from the pod is encapsulated inside a standard UDP packet (typically on UDP port 4789 for VXLAN) by the host kernel. The physical network only sees UDP traffic between Node 1 IP and Node 2 IP. While highly portable across any underlying cloud or bare-metal network, encapsulation adds a 50-byte header overhead per packet and consumes host CPU cycles.
  • Flat Underlay Routing (Calico BGP, AWS VPC CNI, Azure CNI): Pods receive native IP addresses directly from the underlying physical or cloud VPC subnet (e.g., an AWS secondary private IPv4 address attached to an ENI). Packets traverse the physical routers without encapsulation overhead. However, this approach consumes cloud VPC IP allocations rapidly and requires strict provider API limits.

2. CNI Architectures Compared: Calico, Cilium, and Cloud-Native CNIs

The choice of Container Network Interface plugin dictates how packets are inspected, filtered, and routed inside the cluster:

CNI Plugin Data Plane Technology Routing Mechanism Troubleshooting Complexity Best Fit
Project Calico Linux iptables / IPVS / eBPF BGP peer mesh or VXLAN overlay Moderate (Inspectable via iptables-save, calicoctl) Enterprise on-prem bare-metal and hybrid multi-cloud clusters.
Cilium eBPF (Extended Berkeley Packet Filter) Native routing or Geneve/VXLAN overlay Advanced (Requires cilium-dbg, bpftool, Hubble UI) High-scale clusters (>10k services), zero-trust mTLS, and L7 security.
AWS VPC CNI AWS Elastic Network Interfaces (ENI) VPC route tables (Direct flat IP) Low (Standard AWS VPC Flow Logs, CloudWatch) Native AWS EKS clusters requiring direct IAM and VPC peering integration.

3. Systematic Triage Phase 1: Conntrack Table Exhaustion & Packet Drops

The Linux kernel uses the Netfilter Connection Tracking (conntrack) subsystem to maintain stateful tracking for all active TCP, UDP, and ICMP sessions. Every time a container establishes an outgoing HTTP request or receives an incoming connection, an entry is added to the host's conntrack table. When Kubernetes services utilize kube-proxy in iptables mode, NAT rules (DNAT/SNAT) rely heavily on conntrack to map virtual ClusterIPs to actual pod IPs.

Diagnosing Conntrack Exhaustion

When the conntrack table fills to maximum capacity (nf_conntrack_max), the Linux kernel immediately begins dropping all new connection attempts without notifying the application. In container logs, this manifests as random connection timeouts (Connection timed out or i/o timeout) while existing long-lived connections continue operating normally.

Execute the following diagnostic commands directly on the affected worker node:

# Check current conntrack usage vs maximum allowed limit
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Calculate table utilization percentage
python3 -c "
count = int(open('/proc/sys/net/netfilter/nf_conntrack_count').read())
max_limit = int(open('/proc/sys/net/netfilter/nf_conntrack_max').read())
print(f'Conntrack Utilization: {count} / {max_limit} ({count/max_limit*100:.2f}%)')
"

# Inspect kernel drop statistics
sudo conntrack -S

If conntrack -S outputs a non-zero counter for drop or early_drop, your node is actively dropping packets due to table exhaustion. Furthermore, check the kernel ring buffer via dmesg -T | grep -i "nf_conntrack: table full".

Production Remediation for Conntrack Saturation

  1. Increase Table Size: Increase the kernel limit dynamically: sudo sysctl -w net.netfilter.nf_conntrack_max=1048576 (requires ensuring host RAM can accommodate the ~320 bytes per tracked connection).
  2. Shorten TCP TIME_WAIT Timeout: High-throughput HTTP/1.1 microservices without connection pooling flood the table with connections in TIME_WAIT. Reduce the timeout from the default 120 seconds: sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30.
  3. Migrate from iptables to IPVS or Cilium eBPF: IPVS uses efficient in-memory hash tables with O(1) lookup complexity, eliminating the massive iptables sequential chain traversal overhead in large clusters.

4. Systematic Triage Phase 2: CoreDNS Resolution Delays & ndots:5

DNS resolution failures represent the single most common cause of intermittent 5-second latency spikes in Kubernetes. In standard Linux environments, when a process issues a DNS query (e.g., calling api.stripe.com), the resolver checks /etc/resolv.conf.

The ndots:5 Search Domain Multiplication Trap

By default, Kubernetes injects an ndots:5 configuration into every container's /etc/resolv.conf:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local us-east-1.compute.internal
options ndots:5

The ndots:5 setting instructs the resolver: "If a hostname contains fewer than 5 dots, append every single search domain sequentially before attempting an absolute DNS lookup." When an application requests api.stripe.com (which has only 2 dots), the container issues four consecutive failing DNS queries:

  1. api.stripe.com.production.svc.cluster.local. → NXDOMAIN
  2. api.stripe.com.svc.cluster.local. → NXDOMAIN
  3. api.stripe.com.cluster.local. → NXDOMAIN
  4. api.stripe.com.us-east-1.compute.internal. → NXDOMAIN
  5. api.stripe.com. → SUCCESS (Finally resolved!)

Every single external API call multiplies DNS load on CoreDNS by 5x. When coupled with the well-documented Linux glibc 5-second DNS lookup bug (a race condition where concurrent A and AAAA UDP queries share the same conntrack entry), external queries stall for exactly 5.00 seconds.

Remediation Strategies for CoreDNS

  • Fully Qualified Domain Names (Trailing Dot): In application code and configuration files, append a trailing dot to external URLs: https://api.stripe.com./v1/charges. The trailing dot indicates an absolute domain, completely bypassing the ndots:5 search domain traversal.
  • Customize Pod dnsConfig: For external-heavy microservices, override ndots in the Pod specification:
    dnsConfig:
      options:
        - name: ndots
          value: "2"
        - name: single-request-reopen
  • Deploy NodeLocal DNSCache: Deploy the Kubernetes NodeLocal DNSCache DaemonSet. NodeLocal runs a lightweight DNS caching agent on a local link IP (169.254.20.10) on every worker node. Queries are answered locally via memory cache in < 0.5 ms and forwarded upstream via TCP, eliminating conntrack UDP race conditions.

5. Systematic Triage Phase 3: Cross-Namespace NetworkPolicy Blocks

Kubernetes networking is "open by default": any pod in any namespace can initiate TCP connections to any other pod across the entire cluster. However, when security teams enforce zero-trust isolation by introducing NetworkPolicy resources, misconfigured selectors often block legitimate microservice traffic.

Diagnosing NetworkPolicy Drops with Cilium and Calico

Unlike traditional firewalls that generate explicit ICMP "Port Unreachable" or TCP RST packets, a blocked NetworkPolicy simply discards packets silently. The client experiences connection timeouts. To diagnose policy drops:

Cilium Policy Tracing

# Stream live packet drops directly from kernel eBPF hooks
kubectl exec -n kube-system -ti cilium-xxxxx -- cilium-dbg monitor --type drop

If policy drops are occurring, Cilium outputs: Policy denied by filter (UNKNOWN) → Drop (Policy denied), specifying the exact source pod, destination pod, and port.

Calico Felix Policy Logging

In Calico, enable logging on the target policy by adding action: Log to the rule definition. Then inspect Felix logs via journalctl -u calico-node | grep "POLICY_DROP".


6. Systematic Triage Phase 4: MTU Mismatches & VXLAN Fragmentation

Maximum Transmission Unit (MTU) specifies the largest physical packet size (in bytes) that can traverse a network interface without fragmentation. Standard physical Ethernet interfaces use an MTU of 1500 bytes (or 9000 bytes for jumbo frames).

The 50-Byte Encapsulation Black Hole

When using an overlay CNI (such as VXLAN or Geneve), the host encapsulates the container's packet inside an outer UDP packet. A standard VXLAN header consists of:

  • Outer Ethernet Header: 14 bytes
  • Outer IPv4 Header: 20 bytes
  • Outer UDP Header: 8 bytes
  • VXLAN Header: 8 bytes
  • Total Encapsulation Overhead: 50 bytes

If the physical network interface MTU is 1500 bytes and the container virtual interface (eth0) is also configured for 1500 bytes, a full-sized 1500-byte packet produced by the container becomes 1550 bytes after encapsulation. If intermediate physical routers have the DF (Don't Fragment) bit set, the packet is discarded. Small packets (like TCP SYN handshakes or HTTP GET headers) succeed, but large responses (like database bulk fetches or file downloads) hang indefinitely. This symptom—where connections establish but data transfers freeze—is the classic signature of an MTU black hole.

Verifying MTU Configuration

Ensure the container veth interface MTU is strictly configured to Physical MTU - 50 (e.g., 1450 bytes for a standard 1500-byte host):

# Test Path MTU with ping (using Don't Fragment flag -M do)
ping -M do -s 1422 <TARGET-POD-IP>


7. Ingress Gateway Timeouts & HTTP Keep-Alive Race Conditions

A frequent and frustrating networking failure in cloud-native platforms is the intermittent 502 Bad Gateway or 504 Gateway Timeout observed under steady-state production traffic. When engineers inspect backend pod logs, there are zero recorded 502 errors; the application appears completely healthy. The error is generated at the proxy boundary between the Ingress Gateway (e.g., NGINX Ingress, Envoy, or AWS ALB) and the upstream application pods.

The HTTP Keep-Alive Timeout Race Condition

To reduce the latency overhead of establishing new TCP handshakes and TLS sessions for every HTTP request, reverse proxies maintain a pool of persistent, idle TCP connections to backend pods using HTTP Keep-Alive. Both the proxy and the upstream application container maintain an idle connection timeout timer:

  • Scenario: Upstream application (e.g., Node.js, Spring Boot, Go net/http) has an idle keep-alive timeout of 60 seconds. The Ingress Gateway has an idle keep-alive timeout of 65 seconds.
  • The Race: At second 60.001, the application server decides the socket has been idle long enough and transmits a TCP FIN packet to close the connection. Concurrently, at second 60.002, the Ingress Gateway receives a new incoming user HTTP request and reuses that same connection from its pool, transmitting an HTTP GET.
  • The Crash: The packets cross in transit on the network wire. The application receives the HTTP GET on a socket it has already closed, immediately responding with a TCP RST (Reset) packet. The Ingress Gateway receives the RST and immediately aborts the client request with a 502 Bad Gateway: Connection reset by peer error.

Production Resolution Rule

In all cloud architectures, the upstream application container's keep-alive timeout must always be greater than the upstream reverse proxy's keep-alive timeout. In AWS ALB environments, where the ALB client timeout is 60 seconds, configure application servers (Node.js, Python Gunicorn, Java Netty) to maintain an idle timeout of 65 seconds or higher. This guarantees that the load balancer initiates connection closure, eliminating the transit race condition.


8. Advanced Kernel Packet Drop Tracing: dropwatch and pwru

When packets disappear inside the Linux operating system without matching any user-space firewall rule, SREs must drop down to kernel-level packet inspection tools:

1. Kernel Dropwatch: Pinpointing the Dropping Function

The Linux kernel tracks every discarded packet through the internal kfree_skb kernel tracepoint. By using the dropwatch utility, engineers can pinpoint the exact kernel C function responsible for discarding packets:

# Start dropwatch in interactive mode
sudo dropwatch -l kas

# Dropwatch Output Example:
# 4 drops at nf_hook_slow+0xd6/0x110 [kernel]
# 1 drops at ip_rcv_finish_core.constprop.0+0x184/0x3a0 [kernel]

In this trace, nf_hook_slow reveals that a Netfilter firewall rule (iptables or nftables) dropped the packet, whereas ip_rcv_finish_core indicates an IP checksum corruption or invalid IP destination route.

2. eBPF Packet Tracing with pwru (Packet Where aRe yoU)

pwru is an open-source eBPF-based packet tracking tool that attaches to all Linux network-related kernel functions. It prints the full kernel call stack for specific packets identified by IP or port:

# Trace exactly where packets destined for port 8080 are dropped
sudo pwru --filter-dst-port 8080 --output-tuple

pwru prints each kernel function the packet enters (e.g., ip_local_deliver → tcp_v4_rcv → tcp_v4_do_rcv), highlighting the precise line of kernel code where the packet was discarded, turning hours of blind guesswork into seconds of definitive forensic certainty.


9. Step-by-Step Incident Triage Flowchart & Decision Tree

During an active container networking incident, follow this strict, ordered diagnostic sequence to isolate the failure domain in under 5 minutes:

  1. Step 1: Scope the Outage: Is connectivity failing intra-pod (localhost), intra-node (same worker node), inter-node (different worker nodes), or egress (to the public internet)? If intra-node works but inter-node fails, immediately suspect CNI overlay encapsulation, MTU limits, or node security group firewall rules.
  2. Step 2: Check DNS vs IP Connectivity: Attempt to connect via raw IP: curl -v http://10.244.2.15:8080. If raw IP succeeds but hostname fails, the issue is strictly isolated to CoreDNS or ndots:5 search domains.
  3. Step 3: Inspect Conntrack Capacity: Verify conntrack table count on the source and destination worker nodes: cat /proc/sys/net/netfilter/nf_conntrack_count. If near nf_conntrack_max, immediately raise the table limit.
  4. Step 4: Check NetworkPolicies: Check for active isolation policies in the source and destination namespaces: kubectl get netpol -A. Temporarily label test pods to bypass selectors or use Cilium monitor to verify policy drops.
  5. Step 5: Verify MTU and Jumbo Frames: Execute a ping test with the Don't Fragment flag: ping -M do -s 1422 <Target-IP>. If small pings succeed but full MTU pings fail, adjust CNI MTU allocations.

10. Production SRE Network Diagnostic CLI Toolkit

Modern distroless and scratch production containers do not contain curl, ping, or netstat. SREs must use ephemeral debug containers to attach directly to target pod network namespaces:

# 1. Attach an ephemeral diagnostic container sharing the target pod's network namespace
kubectl debug -it target-pod-name --image=nicolaka/netshoot --target=container-name

# 2. Inside the debug container: trace socket states and listening ports
ss -tulpn

# 3. Test DNS resolution latency against CoreDNS
dig @10.96.0.10 target-service.production.svc.cluster.local +stats

# 4. Capture live packet flow and filter by TCP flags
tcpdump -nn -vv -i any "tcp[tcpflags] & (tcp-syn|tcp-rst) != 0" and port 8080

# 5. Measure network throughput and MTU path
mtr --tcp --port 8080 target-service.production.svc.cluster.local

11. Frequently Asked Questions

Why does pinging a Kubernetes ClusterIP fail even when services are healthy?

ClusterIPs are virtual constructs implemented via iptables or IPVS packet rewrite rules on worker nodes; they do not possess a virtual network interface that responds to ICMP echo requests. To test connectivity, use Layer 4 or Layer 7 tools like curl or nc -zv <ClusterIP> <Port>.

What causes random 5-second DNS timeouts in Kubernetes pods?

This is commonly caused by a Linux kernel conntrack race condition when resolving external domains under ndots:5. Concurrent A and AAAA DNS queries sent over UDP can collide in the conntrack table, causing one query to be dropped and forcing a 5-second retransmit. Deploying NodeLocal DNSCache or appending a trailing dot to external URLs eliminates this issue.

What is an MTU mismatch and how does it manifest in production?

An MTU mismatch occurs when overlay network encapsulation (like VXLAN) adds header overhead that causes packets to exceed physical network limits. This typically manifests as successful TCP handshakes (small packets) followed by stalled data transfers (large packets exceeding MTU being silently dropped).


12. Verifiable Sources & Further Reading