Kubernetes Best Practices: Production Reliability, Security & Operations
An enterprise operational guide to architecting resilient, production-grade Kubernetes workloads: CPU and memory resource sizing, probe configuration, PodDisruptionBudgets, multi-AZ topology spread, graceful connection draining, and policy admission controls.
Kubernetes is exceptionally powerful, but out of the box, its default configuration values are designed for rapid local developer experimentation rather than mission-critical production reliability. Deploying a workload without explicit resource requests allows a single container experiencing a memory leak to destabilize an entire worker node. Configuring an aggressive liveness probe can transform a transient downstream database slowdown into a catastrophic, cluster-wide pod restart cascade. Omitting pod disruption budgets turns routine worker node kernel security patching into an unplanned customer-facing outage.
Operating Kubernetes at scale demands rigorous, defense-in-depth operational discipline. Production workloads must be decoupled from node lifecycles, scheduled across independent hardware failure domains, guarded by admission control policies, and tuned for graceful shutdown. When an underlying virtual machine terminates unexpectedly, a well-architected Kubernetes system absorbs the failure seamlessly without dropping a single active TCP connection or violating service level agreements (SLAs).
This comprehensive technical guide details enterprise Kubernetes best practices. We examine the mechanics of Linux cgroups v2 resource quotas, explore the critical distinctions between startup, liveness, and readiness probes, configure multi-zone TopologySpreadConstraints and PodDisruptionBudgets, detail zero-downtime graceful shutdown sequences, and provide complete, copy-paste-safe production manifests.
1. Sizing Resource Requests and Limits: Avoiding the CPU Throttling Trap
Resource management represents the most misunderstood architectural boundary in Kubernetes operations. Kubernetes utilizes two primary parameters for container compute sizing: requests and limits.
Requests vs Limits: The Underlying Linux Mechanics
- Resource Requests (Scheduling Baseline): The Kubernetes scheduler (
kube-scheduler) usesrequestsexclusively to determine which worker node has sufficient capacity to host the pod. It represents the guaranteed minimum compute allocated to that container. - Resource Limits (Kernel Enforcement): Limits dictate the maximum ceiling of compute a container may consume. However, CPU and memory limits are enforced through fundamentally different Linux kernel mechanisms:
- Memory Limits (Hard Ceiling / OOMKill): Enforced via Linux
cgroupsmemory controllers. When a container's resident set size (RSS) memory exceeds its declared limit, the Linux kernel Out-Of-Memory (OOM) killer immediately issues aSIGKILL(exit code 137). Memory cannot be compressed or throttled. - CPU Limits (Completely CFS Throttled): Enforced via the Linux Completely Fair Scheduler (CFS) quota mechanism over a 100ms period. When a multi-threaded application consumes its quota within the first 20ms of a period, the kernel completely freezes process execution for the remaining 80ms, introducing severe artificial p99 latency spikes.
- Memory Limits (Hard Ceiling / OOMKill): Enforced via Linux
Quality of Service (QoS) Classes
Kubernetes assigns every pod an internal Quality of Service class based on its resource definitions, governing node eviction order during node memory starvation:
| QoS Class | Configuration Rule | Node Eviction Priority | Production Suitability |
|---|---|---|---|
| Guaranteed | requests == limits for both CPU and Memory across all containers |
Lowest (Evicted last) | Ideal for stateful databases, latency-critical caches, and financial transaction engines |
| Burstable | requests < limits, or limits defined without requests |
Medium (Evicted after BestEffort) | Ideal for standard web APIs and worker jobs with variable burst requirements |
| BestEffort | No requests or limits specified | Highest (Evicted first immediately under pressure) | Strictly forbidden for production workloads; causes noisy neighbor outages |
500m) based on historical p95 telemetry, set strict memory limits to protect against leaks, and either omit CPU limits or set them 3x-5x higher than requests to allow burst processing.
2. Configuring Health Probes Correctly: Preventing Cascade Death Spirals
Kubernetes provides three distinct health check probe mechanisms. Conflating their purposes is one of the most common causes of cluster-wide cascading failures:
1. Startup Probes: Handling Slow Initialization
Legacy Java (Spring Boot) or large Node.js applications frequently require 30 to 90 seconds to load JVM classes, initialize connection pools, and run migrations. If you only configure a liveness probe, the kubelet will repeatedly kill the container before it finishes booting, trapping the pod in CrashLoopBackOff. A Startup Probe pauses all liveness and readiness checks until the initialization endpoint returns 200 OK.
2. Liveness Probes: Deadlock Recovery
A liveness probe answers one question: "Has this process entered an unrecoverable deadlock requiring a container restart?" Never point a liveness probe at an endpoint that checks external dependencies (such as PostgreSQL or Redis). If the database slows down, all microservice pods will fail their liveness checks simultaneously, causing the entire cluster to restart in unison and generating a devastating thundering herd outage.
3. Readiness Probes: Traffic Routing Gating
A readiness probe answers: "Is this pod currently capable of accepting HTTP traffic?" When a readiness probe fails, Kubernetes does not restart the pod; it simply removes the pod's IP address from the Service endpoints list. If a pod experiences heavy CPU load or temporary queue saturation, failing its readiness probe safely pauses traffic until it recovers.
# Correctly Configured Production Health Probes
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 30 # Allows up to 150s for slow startup
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz/liveness # Checks internal process thread health ONLY
port: 8080
initialDelaySeconds: 0 # Handled by startupProbe!
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz/readiness # Checks local connection pool capacity
port: 8080
periodSeconds: 10
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 2
3. High Availability: Topology Spread Constraints and PodDisruptionBudgets
Deploying multiple replicas does not guarantee high availability if the Kubernetes scheduler places all three replicas onto the same physical worker node or within the same AWS availability zone. If that single hypervisor crashes or AZ loses power, your entire service drops offline.
Enforcing Multi-Zone Distribution with TopologySpreadConstraints
Modern Kubernetes clusters utilize topologySpreadConstraints to enforce even distribution across availability zones and physical hosts:
# k8s/topology-spread.yaml
spec:
topologySpreadConstraints:
# 1. Distribute replicas evenly across cloud availability zones
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: payment-api
# 2. Avoid placing multiple replicas on the same worker node
- maxSkew: 1
topologyKey: "kubernetes.io/hostname"
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: payment-api
Guarding Availability with PodDisruptionBudgets (PDB)
When platform teams upgrade EKS worker nodes, perform cluster autoscaling, or execute kubectl drain, Kubernetes evicts pods to recycle nodes. Without a PodDisruptionBudget, the eviction controller can terminate all replicas simultaneously. A PDB guarantees that a minimum number of healthy replicas remain operational at all times:
# k8s/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payment-api-pdb
namespace: production
spec:
minAvailable: 2 # Alternatively: maxUnavailable: 1
selector:
matchLabels:
app: payment-api
4. Zero-Downtime Rolling Updates: Connection Draining & PreStop Hooks
During a rolling deployment, Kubernetes updates pod replicas incrementally. However, applications frequently drop in-flight user requests during pod termination. Understanding why this happens requires examining the Kubernetes networking control plane race condition.
The Termination Race Condition
When a pod is deleted, two events happen simultaneously in parallel:
- Endpoint Removal: The endpoints controller removes the pod IP from the Service, and
kube-proxyupdatesiptables/ IPVS rules on every worker node. However, this takes 1 to 3 seconds to propagate across the cluster. - Pod Termination: The kubelet immediately sends a
SIGTERMsignal to the container process. If your application terminates immediately, external load balancers and neighboring pods will continue routing new requests to the dying pod for several seconds, resulting in HTTP 502 Bad Gateway errors.
Remediation: PreStop Sleep Hook & Graceful Shutdown
To eliminate 502 errors, inject a preStop lifecycle hook that executes a sleep 15 command before sending SIGTERM. This gives the Kubernetes network control plane ample time to update routing tables and drain active TCP connections before the application stops accepting traffic:
spec:
terminationGracePeriodSeconds: 60 # Must exceed preStop sleep + app shutdown duration
containers:
- name: web
image: devopstribune/core-api:v1.2.0
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
5. Hardening Pod Security Standards (PSS) & Least Privilege
Kubernetes cluster security begins inside the pod specification. Running containers with root privileges or mounting the host filesystem creates massive attack surfaces for container breakout. The native Kubernetes Pod Security Standards (PSS) replace deprecated PodSecurityPolicies with three profiles: Privileged, Baseline, and Restricted.
Production Restricted SecurityContext Specification
All production application pods should adhere to the Restricted PSS profile:
# Production Hardened Security Context
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: api
image: devopstribune/core-api:v1.2.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
volumeMounts:
- name: tmp-volume
mountPath: /tmp
volumes:
- name: tmp-volume
emptyDir: {} # Writable memory-backed scratch space
allowPrivilegeEscalation: false: Prevents child processes from gaining more privileges than their parent (blockssetuidbinary attacks).readOnlyRootFilesystem: true: Prevents malicious binaries or rootkits from being downloaded into the container disk. Temporary files must be written to explicitemptyDirvolume mounts.capabilities.drop: ["ALL"]: Strips all Linux root capabilities (e.g.,CAP_NET_RAW,CAP_SYS_ADMIN), preventing raw socket packet sniffing and kernel manipulation.
6. Zero-Trust Kubernetes Network Policies: Eliminating Flat-Network Vulnerabilities
By default, Kubernetes implements an entirely flat networking model: every pod can communicate with every other pod across all namespaces without restriction. If an attacker gains remote code execution on a publicly exposed frontend web container, the flat network model allows them to immediately probe internal PostgreSQL databases, Redis cache instances, or cluster internal services.
Production Kubernetes clusters must enforce zero-trust network policies via a Container Network Interface (CNI) plugin that supports NetworkPolicy enforcement (such as Cilium, Calico, or AWS VPC CNI with network policy support). The industry-standard approach follows a three-step defense-in-depth model:
- Default Deny All Ingress and Egress: Every production namespace should include a baseline NetworkPolicy that drops all incoming and outgoing network packets by default.
- Explicit Ingress Allowlisting: Workloads only permit incoming connections from designated ingress controllers or authorized client microservices on specific TCP ports.
- Strict Egress Restricting with DNS Access: Pods are prevented from exfiltrating data to arbitrary external IP addresses, while retaining UDP/TCP port 53 access to
kube-dns(CoreDNS) for service discovery.
The following production NetworkPolicy isolates the billing-service workload, allowing ingress exclusively from the ingress controller and restricting egress solely to CoreDNS and the internal database subnet:
# k8s/network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: billing-service-netpol
namespace: production
spec:
podSelector:
matchLabels:
app: billing-service
policyTypes:
- Ingress
- Egress
ingress:
# Rule 1: Allow ingress HTTP traffic exclusively from the NGINX Ingress Controller
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- protocol: TCP
port: 8080
egress:
# Rule 1: Allow internal DNS resolution via CoreDNS in kube-system
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Rule 2: Allow outbound traffic exclusively to internal managed RDS database subnet
- to:
- ipBlock:
cidr: 10.100.16.0/20 # VPC database private subnet
ports:
- protocol: TCP
port: 5432
7. Multi-Tenant Namespace Governance: ResourceQuotas and LimitRanges
In shared clusters, developers or automated CI/CD pipelines can inadvertently deploy unconstrained workloads that exhaust node memory and CPU, triggering cluster-wide node pressure and cascading pod evictions. Kubernetes resolves multi-tenancy noisy-neighbor problems through two governance primitives:
1. LimitRanges: Setting Default Requests and Maximum Ceilings
A LimitRange automatically injects default compute requests and limits into any pod deployed without them, and blocks developers from requesting absurdly large resources that would hog entire bare-metal nodes:
# k8s/limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: default-compute-limits
namespace: production
spec:
limits:
- type: Container
default: # Default limits applied if omitted by user
cpu: "500m"
memory: "512Mi"
defaultRequest: # Default requests applied if omitted
cpu: "100m"
memory: "128Mi"
max: # Hard maximum container ceiling
cpu: "4000m"
memory: "8Gi"
min: # Baseline minimum container floor
cpu: "25m"
memory: "32Mi"
2. ResourceQuotas: Hard Namespace Aggregate Budgets
A ResourceQuota establishes a strict hard ceiling on the aggregate sum of all compute requests, limits, persistent volume claims, and service objects that can exist within a given namespace. When a deployment would cause total memory requests in production to exceed 64Gi, the Kubernetes API server immediately rejects the deployment with an HTTP 403 Forbidden status, protecting other tenants from resource starvation.
# k8s/resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: production-quota
namespace: production
spec:
hard:
requests.cpu: "32"
requests.memory: 64Gi
limits.cpu: "64"
limits.memory: 128Gi
pods: "50"
services.loadbalancers: "2"
requests.storage: 500Gi
8. Zero-Downtime Graceful Pod Lifecycle and Node Draining
During rolling updates, node autoscaling, or security patching via kubectl drain, Kubernetes terminates existing pods and replaces them with new ones. However, a major architectural race condition frequently causes intermittent HTTP 502 Bad Gateway and TCP connection reset errors during these events.
The Underlying Kubernetes Termination Race Condition
When a pod is targeted for termination, Kubernetes initiates two distinct asynchronous processes concurrently:
- Endpoint Controller Deregistration: The endpoint controller removes the pod's IP address from the Service endpoint list. Kube-proxy instances running across every worker node in the cluster must update their local Linux kernel
iptablesor IPVS forwarding rules. Depending on cluster size and etcd load, this propagation delay typically requires 2 to 10 seconds. - Kubelet SIGTERM Execution: Simultaneously, the local kubelet sends a
SIGTERMsignal directly to the container process. If your application immediately halts listening on its TCP socket upon receivingSIGTERM, incoming HTTP requests that were routed by other nodes during the 10-second iptables propagation window will hit a closed socket, producing immediate 502/504 errors for active users.
preStop Sleep Hook
To prevent this race condition, configure a preStop lifecycle hook that sleeps for 10 to 15 seconds before the application handles SIGTERM: lifecycle.preStop.exec.command: ["/bin/sh", "-c", "sleep 15"]. This guarantees that all cluster-wide iptables rules are fully flushed and no new traffic is routed to the pod before the application initiates its in-flight request draining.
Additionally, ensure your pod's terminationGracePeriodSeconds is set high enough (e.g., 45 to 60 seconds) to accommodate the 15-second preStop sleep plus the time required for long-running database transactions or file transfers to finish cleanly.
9. Complete Production-Hardened Kubernetes Deployment Manifest
The following deployment manifest consolidates every operational best practice discussed in this guide into an immutable, production-ready template:
# k8s/production-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-service
namespace: production
labels:
app.kubernetes.io/name: billing-service
app.kubernetes.io/part-of: financial-platform
spec:
replicas: 4
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0 # Zero downtime guarantee
selector:
matchLabels:
app: billing-service
template:
metadata:
labels:
app: billing-service
spec:
terminationGracePeriodSeconds: 45
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: billing-service
containers:
- name: application
image: devopstribune/billing-service:1.4.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: http
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1000m" # Burstable headroom
memory: "512Mi" # Strict OOM ceiling
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 15"]
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 20
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz/liveness
port: 8080
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz/readiness
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 2
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
10. Production Troubleshooting Runbook for Kubernetes Workloads
When operating Kubernetes workloads, use the following operational triage matrix to rapidly isolate common pod failure states:
| Observed Symptom | Root Cause | Immediate SRE Remediation |
|---|---|---|
Exit Code 137 / OOMKilled |
Container memory resident set size exceeded declared limits.memory. |
Inspect application heap dump. Increase limits.memory or configure runtime memory limits (e.g., JVM -XX:MaxRAMPercentage=75). |
| Intermittent HTTP 502 during Deployments | Pod terminated before kube-proxy updated iptables routing rules across worker nodes. | Inject lifecycle.preStop.exec.command: ["sleep", "15"] and increase terminationGracePeriodSeconds. |
| High p99 Latency with Low CPU Usage | Linux CFS quota throttling due to restrictive CPU limits on multi-threaded application. | Inspect container_cpu_cfs_throttled_periods_total. Remove CPU limits or increase limit ceiling 3x above requests. |
| Cluster Node Drain Blocked Forever | Overly restrictive PodDisruptionBudget (e.g., minAvailable: 1 on a single replica deployment). |
Scale deployment to at least 2 replicas before applying PDB, or change PDB to maxUnavailable: 1. |
11. Frequently Asked Questions
Why should I avoid setting CPU limits equal to CPU requests?
Setting CPU limits equal to requests forces the pod into the Guaranteed QoS class but severely restricts multi-threaded performance. Because Linux CFS quotas evaluate CPU time in 100ms slices, a spike in concurrent requests can throttle execution within milliseconds, causing p99 latency regressions. SRE best practice is to set requests based on average load and permit bursting by setting limits significantly higher or omitting CPU limits entirely.
What is the difference between TopologySpreadConstraints and PodAntiAffinity?
PodAntiAffinity is a binary rule (either pods cannot schedule on the same host, or they prefer not to). It often leads to scheduling deadlocks where pods remain in Pending status. TopologySpreadConstraints is a modern, flexible scheduling mechanism that defines a mathematical maximum skew (e.g., maxSkew: 1), ensuring pods are balanced evenly across zones without causing unresolvable scheduling deadlocks.
Why should liveness probes never check external databases?
If an external database experiences high load or a network glitch, a liveness probe that queries the database will fail across all pods simultaneously. Kubernetes will respond by killing and restarting every application container at the exact moment the database is struggling, compounding the outage into a cluster-wide failure cascade. Liveness probes must evaluate only local in-process thread health.