Every cloud-native abstraction—Kubernetes worker nodes, container runtimes, serverless micro-VMs, and cloud storage gateways—executes as processes on a Linux kernel. When latency spikes or nodes enter NotReady states, high-performing platform engineers do not randomly restart servers; they systematically inspect kernel performance metrics to isolate the exact saturated subsystem.

This reference guide organizes essential Linux diagnostic commands into four operational domains: CPU scheduling, memory pressure, storage I/O, and low-level socket networking.


1. CPU & Process Scheduling Saturation

1.1 Load Average vs CPU Utilization: vmstat

Linux load average measures the number of tasks currently in the Runnable state (R) plus tasks in the Uninterruptible Sleep state (D). When load average spikes while CPU utilization remains near zero, tasks are blocked waiting on disk, network storage, or kernel mutexes.

# Run vmstat at 1-second intervals for 5 iterations
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  4      0 184520 214500 4829100    0    0   120  4800 1200 3400 12  4 22 62  0

Key columns to inspect:

  • procs > r: Tasks waiting for CPU execution time. If r consistently exceeds total CPU cores, CPU starvation is occurring.
  • procs > b: Tasks blocked in uninterruptible sleep. A high b value indicates I/O bottlenecks.
  • cpu > wa: Percentage of CPU time spent waiting for outstanding disk I/O requests.
  • cpu > st: CPU “steal” time. Indicates hypervisor over-subscription where the cloud provider is starving your virtual machine of physical CPU cycles.

1.2 Isolating Uninterruptible Sleep Tasks: wchan

To identify the exact kernel function causing processes to stall in uninterruptible sleep, inspect the wchan field:

# Find all processes in uninterruptible sleep and display their kernel wait channel
ps -eo pid,user,stat,wchan:24,comm | grep ' D'

# Inspect the exact kernel stack trace for a specific stalled process
cat /proc/<PID>/stack

2. Memory Pressure & Page Cache Mechanics

2.1 Interpreting Memory: Available vs Free

A common junior mistake is triggering alerts when free memory is low. Linux uses all unallocated memory to cache disk reads (Page Cache) and disk metadata (Buffers):

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           31980        8400        1240         410       22340       23170
Swap:              0           0           0

Always alert on available memory, which estimates how much RAM can be immediately allocated to applications without swapping, by reclaiming clean page cache.

2.2 Container Cgroup v2 Memory Boundaries

In modern container environments (Kubernetes 1.25+ on cgroup v2), check container memory pressure directly from the filesystem:

# Inside a container or pod cgroup directory:
cat /sys/fs/cgroup/memory.current      # Current byte consumption
cat /sys/fs/cgroup/memory.max          # Hard limit (OOM trigger threshold)
cat /sys/fs/cgroup/memory.pressure     # Linux PSI (Pressure Stall Information)

3. Storage Saturation & Inode Exhaustion

3.1 Device Latency Profiling: iostat

To determine if high latency is caused by EBS volume IOPS throttling or saturated local NVMe disks:

# Run extended I/O statistics, displaying only active devices (-z) in megabytes (-m)
$ iostat -xzm 1 5
Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm  r_await  w_await   aqu-sz  %util
nvme0n1         45.00  850.00      2.10     48.50     0.00    12.00   0.00   1.39     0.82    14.20     4.80  96.40
  • r_await / w_await: Average time (in milliseconds) for read/write requests to be served. If await exceeds 10–15ms on flash storage, the device is saturated.
  • %util: Percentage of elapsed time during which the device was handling requests. Values approaching 100% indicate saturation.

3.2 Hidden Disk Full Errors: Inodes & Unlinked Open Files

A disk can throw No space left on device even when df -h shows 40% free space. This is caused by inode exhaustion (millions of tiny files) or deleted files held open by running processes:

# Check inode usage across all mounted filesystems
df -i

# Find deleted files still held open by running processes (holding disk space hostage)
lsof +L1

# Safe fix: Identify the holding process PID and restart/terminate it, or truncate via /proc/<PID>/fd

4. Low-Level Networking & Socket Diagnostics

4.1 Socket Inspection: ss

The modern replacement for deprecated netstat is ss, which queries kernel socket tables directly via Netlink:

# Print overall socket memory and connection summary
$ ss -s
Total: 4120
TCP:   3890 (estab 1420, closed 2140, orphaned 12, timewait 2080)

# Display listening TCP sockets with process names and numerical ports
ss -tulpn

# Find sockets with overflowing receive/send queues (Send-Q > 0 for listening sockets indicates unaccepted connection backlog)
ss -ltn

4.2 Real-Time Packet Sniffing: tcpdump

When microservices return intermittent connection resets or TLS handshake failures:

# Capture TCP SYN, RST, and FIN packets on port 443 without DNS lookup (-n)
sudo tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-rst|tcp-fin) != 0 and port 443' -c 50

5. Related Operational Runbooks