Kubernetes CrashLoopBackOff: The Production Platform Engineering Triage Runbook
A systematic diagnostic framework for decoding container exit codes (137, 127, 1), resolving liveness probe race conditions, aligning memory cgroup limits, and restoring failing pods without guessing.
1. The Kubelet State Machine: What CrashLoopBackOff Actually Means
In Kubernetes operations, CrashLoopBackOff is often misconstrued as an error code. Mechanically, it is an exponential backoff waiting state applied by the local node agent (kubelet) to protect cluster resources.
When a container inside a Pod starts, crashes, and terminates unexpectedly, the kubelet’s restartPolicy (defaulting to Always) initiates a restart. To prevent an unstable process from consuming 100% of the host CPU and flooding node disk space with crash dumps, the kubelet delays subsequent restarts using an exponential backoff algorithm:
Restart 1: Immediate
Restart 2: Delay 10 seconds
Restart 3: Delay 20 seconds
Restart 4: Delay 40 seconds
Restart 5: Delay 80 seconds
...
Restart N: Capped at 300 seconds (5 minutes)
ImagePullBackOff indicates an image repository authorization, network, or registry tag failure before the container can launch. OOMKilled (Out Of Memory) is a specific kernel or cgroup termination trigger that manifests as Exit Code 137 and often causes a container to enter a CrashLoopBackOff cycle.
2. The Exit Code Decision Matrix
Every terminated container leaves behind a numeric exit code recorded by the Linux kernel and captured in the Pod's status.containerStatuses[*].lastState.terminated metadata. Consulting this exit code isolates the failure domain in under 30 seconds:
| Exit Code | Linux Signal / Status | Root Cause Category | Typical Failure Scenario |
|---|---|---|---|
0 |
Success / Clean Exit | Premature Completion | Container entrypoint script completed its execution (e.g. executed a migration script or missed an exec "$@" daemon command) instead of running a foreground loop. |
1 / 255 |
General Error | Application Fatal Exception | Unhandled runtime exception, missing environment variable, database connection refusal, syntax error, or failed configuration schema validation. |
126 |
Command Not Executable | Permission / File Flag | The binary or entrypoint script lacks the executable bit (chmod +x), or a non-root runAsUser cannot read the binary path. |
127 |
Command Not Found | Binary / Linker Missing | Specified command is not in PATH, or a compiled Go/Rust binary in a scratch/distroless image failed to locate dynamic shared libraries (e.g. musl vs glibc mismatch). |
137 |
SIGKILL (128 + 9) | OOMKilled or Hard Termination | Linux kernel cgroup OOM killer terminated the container for exceeding its configured memory limit, or the container was forcibly killed via kill -9. |
139 |
SIGSEGV (128 + 11) | Segmentation Fault | Memory access violation in compiled C/C++ libraries, incompatible native extensions, or hardware memory faults. |
143 |
SIGTERM (128 + 15) | Graceful Termination / Probe Kill | The kubelet sent SIGTERM because a liveness or startup probe failed repeatedly, or the pod was evicted due to node drain. |
3. Six-Step Triage Workflow for Production Incidents
When an alert fires for a production pod cycling in CrashLoopBackOff, follow this structured command workflow rather than restarting pods blindly:
Step 1: Inspect Container Termination Reason and Exit Code
Extract the exact termination metadata without scrolling through massive describe blocks:
# Extract termination exit code, reason, and message across all containers
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{":\n ExitCode: "}{.lastState.terminated.exitCode}{"\n Reason: "}{.lastState.terminated.reason}{"\n Message: "}{.lastState.terminated.message}{"\n"}{end}'
Step 2: Read Previous Container Logs
Running standard kubectl logs <pod> often returns an empty stream because the container has just restarted. Use the --previous (or -p) flag to capture the stderr output of the container instance that actually crashed:
# Stream the crash log from the terminated container instance
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous --tail=100
Step 3: Audit Cluster Events Chronologically
Events provide operational context that logs cannot reveal, such as volume mount timeouts, failed security contexts, or node-level kernel OOM kills:
# Inspect events specifically tied to this pod, sorted by creation timestamp
kubectl get events -n <namespace> \
--field-selector involvedObject.name=<pod-name> \
--sort-by='.metadata.creationTimestamp'
Step 4: Check Node-Level OOM and Memory Pressure
If the termination reason is OOMKilled with exitCode: 137, check whether the container violated its container-level limit or if the entire node suffered memory exhaustion:
# Query node memory pressure conditions
kubectl describe node <node-name> | grep -E "MemoryPressure|Allocatable|Allocated"
# Inspect active container memory utilization
kubectl top pod <pod-name> -n <namespace> --containers
Step 5: Live Container Introspection with Ephemeral Debug Containers
If logs fail to yield actionable root causes, attach an ephemeral debug container to the target pod sharing the process namespace (PID namespace) without restarting the workload:
# Attach a debug container to inspect files, environment variables, and filesystem mounts
kubectl debug -it <pod-name> -n <namespace> \
--image=nicolaka/netshoot \
--target=<crashing-container-name>
4. Health Check Death Spirals: Liveness vs Startup Probes
One of the most common causes of CrashLoopBackOff in large production services (Spring Boot, Django, Rails, Java microservices) is a probe race condition.
When an application requires 45 seconds to initialize database connections, run schema pre-checks, and warm caches, a prematurely configured livenessProbe will fire before the application is ready. If failureThreshold: 3 with a periodSeconds: 10 is defined without a startup probe, the kubelet sends SIGTERM after 30 seconds—killing the application during its own bootstrap sequence and creating an infinite restart loop.
# Correctly decoupling initialization from ongoing liveness
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
# Gives the service up to 120s to complete initialization (30 * 4s)
failureThreshold: 30
periodSeconds: 4
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
# Only begins evaluating AFTER the startup probe succeeds
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 2
5. Memory Cgroup Limits vs Runtime Heap Configuration
When running runtime environments with managed garbage collection (such as JVM, V8/Node.js, or .NET), configuring container memory limits in Kubernetes without informing the runtime leads directly to OOMKilled crashes.
By default, runtimes that are not container-aware query the physical host memory (e.g. 128 GB on a cloud worker node) rather than the container's cgroup boundary (e.g. 2 GB). When the runtime allocates heap based on the node's capacity, the Linux cgroup memory controller terminates the container with SIGKILL.
| Runtime | Root Cause | Required Production Configuration |
|---|---|---|
| Java / JVM (JDK 17+) | Heap sizing based on node RAM rather than container cgroup limit. | Set flags: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 to reserve 25% of memory for off-heap and metaspace. |
| Node.js (V8) | V8 default memory limit (~1.4 GB / ~4 GB) exceeds container memory limit. | Pass NODE_OPTIONS="--max-old-space-size=1536" for a 2 GB container limit. |
| Go (v1.19+) | Go GC does not trigger early enough under rapid allocation spikes. | Set GOMEMLIMIT=1800MiB for a 2048Mi container limit to force GC before kernel OOM. |
6. Production-Hardened Kubernetes Deployment Manifest
Below is a production-grade Deployment manifest incorporating startup probes, explicit resource boundaries, read-only root filesystems, and graceful shutdown lifecycle hooks:
apiVersion: apps/v1
kind: Deployment
metadata:
name: platform-api
namespace: production
labels:
app.kubernetes.io/name: platform-api
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
selector:
matchLabels:
app.kubernetes.io/name: platform-api
template:
metadata:
labels:
app.kubernetes.io/name: platform-api
spec:
terminationGracePeriodSeconds: 60
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: api
image: registry.devopstribune.com/platform/api:v2.4.1
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi
env:
- name: GOMEMLIMIT
value: "1800MiB"
- name: PORT
value: "8080"
lifecycle:
preStop:
exec:
command: ["/bin/sleep", "15"]
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 30
periodSeconds: 4
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 2
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir:
sizeLimit: 64Mi
CrashLoopBackOff is a symptom, not a cause. By following the exit code decision matrix (137 for OOM, 127 for binary linkage, 143 for probe timeouts) and decoupling initialization via startup probes, platform teams can eliminate intermittent restart loops and ensure rock-solid service stability.