Kubernetes Architecture: Control Plane, Workload Orchestration & Scheduling
An enterprise platform engineering deep dive into Kubernetes architecture: master control plane internals, etcd Raft consensus, kube-scheduler filtering algorithms, worker node subsystems, declarative controllers, and multi-AZ high availability.
Kubernetes has become the de facto distributed operating system of modern cloud computing. However, operating Kubernetes at enterprise scale requires peeling back the abstraction layers of kubectl to understand the underlying distributed systems architecture. Kubernetes is fundamentally a declarative state reconciliation engine: users submit a desired state manifest to a central API, and an array of autonomous, asynchronous control loops continually observe real-world cluster infrastructure, calculating the delta between desired and actual state, and issuing mutations until convergence is reached.
When engineering teams lack a deep architectural understanding of control plane internals—how worker nodes negotiate lease objects, how controllers handle backoff retries, and how the Linux kernel schedules CFS bandwidth——how etcd enforces Raft consensus, how kube-scheduler calculates node scoring, or how kubelet interacts with container runtimes via CRI and CNI plugins—production incidents become mysterious black-box outages. A subtle etcd disk fsync latency bottleneck can freeze API response times globally. An improperly tuned scheduler can concentrate memory-hungry pods onto a single availability zone, inducing widespread cascading node evictions. Conversely, a platform team with mastery over Kubernetes systems architecture designs fault-tolerant multi-master topologies that seamlessly survive physical datacenter outages.
This comprehensive architectural masterclass examines the deep engineering mechanics of Kubernetes. We detail the master control plane stack (kube-apiserver, etcd, kube-scheduler, kube-controller-manager), trace the worker node execution path through kubelet, containerd, and kube-proxy, compare declarative workload primitives (Deployments, StatefulSets, DaemonSets), examine advanced pod scheduling policies, and provide concrete production reference manifests.
1. The Master Control Plane Stack: Distributed Consensus & API Mechanics
The Kubernetes control plane manages global cluster decisions, detects and responds to cluster events, and maintains the authoritative record of all cluster objects. It comprises four primary decoupled services:
1. kube-apiserver: The Front Door and Validation Pipeline
The kube-apiserver is the only component in the entire cluster that communicates directly with etcd. All other control plane and worker node components must communicate through the API server via structured HTTPS REST endpoints. When an incoming request arrives, it undergoes a rigid four-stage sequential pipeline:
- Authentication: Validates client identity via X.509 client certificates, OpenID Connect (OIDC) JWT tokens, or webhook token authenticators. Maps the connection to a User, Group, and Extra metadata.
- Authorization (RBAC): Evaluates whether the authenticated identity has permission to perform the requested verb (
get,list,create,update,delete) against the target resource API group and namespace. - Mutating Admission Webhooks: Intercepts valid requests to modify or inject defaults (e.g., injecting sidecar proxies like Envoy/Istio, or setting default security contexts).
- Validating Admission Webhooks: Enforces strict policy constraints (e.g., rejecting images using the
latesttag, or blocking privileged containers via Kyverno or OPA Gatekeeper).
Only if all four stages succeed is the object written to persistent storage in etcd.
2. etcd: The Distributed Key-Value Store
etcd is a strongly consistent, distributed key-value database implementing the Raft consensus algorithm. It stores the complete specification and status of every Kubernetes resource. Critical architectural constraints include:
- Strict Quorum Arithmetic: An etcd cluster requires a strict majority of nodes to accept writes:
Quorum = floor(N/2) + 1. A 3-node cluster tolerates 1 node failure; a 5-node cluster tolerates 2 node failures. Clusters should always utilize an odd number of members. - Sequential Disk fsync Sensitivity: Every committed write requires a synchronous disk write (
fsync) across quorum members. If disk write latency exceeds 10ms (common on slow network-attached block storage), Raft leader heartbeats time out, triggering disruptive leader re-elections and API paralysis. High-performance NVMe SSDs are mandatory. - Compaction & Defragmentation:
etcdmaintains a complete revision history (Multi-Version Concurrency Control / MVCC). Over time, MVCC revisions exhaust the default 2 GB storage quota, triggering anNOSPACEalarm that locks the cluster into read-only mode. Production clusters must run automated compaction and defragmentation jobs.
3. kube-scheduler: The Placement Engine
The kube-scheduler monitors the API server for newly created pods that have an empty spec.nodeName field. For each unscheduled pod, the scheduler selects the optimal worker node through a two-phase mathematical pipeline:
- Filtering Phase (Predicates): Evaluates all nodes in the cluster against hard feasibility constraints, eliminating nodes that cannot run the pod:
- Does the node have sufficient unallocated CPU and memory requests? (
NodeResourcesFit) - Does the node match the pod's
nodeSelectorandnodeAffinity? - Does the node possess taints that the pod lacks tolerations for? (
NodeTolerations) - Will placing the pod violate port collisions on the host network? (
NodePorts)
- Does the node have sufficient unallocated CPU and memory requests? (
- Scoring Phase (Priorities): Ranks all surviving feasible nodes on a score of 0 to 100 based on soft preference algorithms:
ImageLocality: Prefers nodes that already have the container image layers cached locally to accelerate startup.NodeResourcesBalancedAllocation: Prefers nodes where placing the pod results in the most balanced ratio of CPU and memory utilization.NodeAffinityScoring: Rewards nodes matching soft affinity expressions (preferredDuringSchedulingIgnoredDuringExecution).
The node with the highest cumulative score is selected. The scheduler executes a Binding call to the API server, updating spec.nodeName.
4. kube-controller-manager: The Continuous Reconciliation Loop
The kube-controller-manager consolidates multiple distinct controller loops into a single process. Each controller runs an infinite loop matching this pseudocode:
for {
desiredState := getDesiredStateFromAPIServer()
actualState := getActualStateFromCluster()
if delta := desiredState - actualState; delta != 0 {
reconcile(delta)
}
sleep(reconciliationInterval)
}
Key embedded controllers include: DeploymentController (scales ReplicaSets), NodeController (evicts pods when a worker node stops sending heartbeats for 40 seconds), EndpointSliceController (updates service IP routing endpoints), and NamespaceController (garbage collects resources during namespace deletion).
2. Worker Node Architecture: The Execution Engine
Worker nodes host the actual customer-facing application containers. A production worker node runs three tightly coupled subsystems:
1. kubelet: The Node Daemon
The kubelet is the primary agent running on every worker node. It registers the node with the API server, reports node health metrics (CPU, memory, disk pressure), and ensures containers described in assigned PodSpecs are running healthy. The kubelet uses the Pod Lifecycle Event Generator (PLEG) to continuously poll the local container runtime. If PLEG execution latency spikes due to container runtime locking, the node transitions into NotReady status.
2. Container Runtime Interface (CRI) & containerd
Kubernetes communicates with the local container runtime via gRPC over the CRI socket (typically /run/containerd/containerd.sock). The CRI separates container orchestration from low-level execution: containerd pulls images, manages snapshot storage, and delegates container execution to runc.
3. kube-proxy & CNI Networking
The Container Network Interface (CNI) plugin (such as Cilium, Calico, or AWS VPC CNI) assigns every pod its own unique routable IP address across the cluster. The kube-proxy daemon monitors Services and Endpoints on the API server, translating virtual Service ClusterIPs into actual pod IP addresses. In modern production clusters, kube-proxy should operate in IPVS (IP Virtual Server) mode or be completely replaced by eBPF (Cilium), bypassing the O(N) performance degradation of legacy Linux iptables.
3. Declarative Workload Primitives: Deployments vs StatefulSets vs DaemonSets
Kubernetes provides specialized controllers tailored to distinct workload characteristics:
| Workload Controller | Pod Identity & Naming | Storage Binding | Primary Production Use Cases |
|---|---|---|---|
| Deployment | Stateless, interchangeable, random hashes (e.g. api-7bf89-kx2q) |
Shared read-only or ephemeral storage | Stateless web APIs, microservices, background job consumers |
| StatefulSet | Stable, ordered, deterministic ordinals (e.g. kafka-0, kafka-1) |
Dedicated, persistent PVC per ordinal via volumeClaimTemplates |
Distributed databases (Cassandra, PostgreSQL), Kafka clusters, ZooKeeper, Elasticsearch |
| DaemonSet | Exactly one pod per matching node in the cluster | Host-level bind mounts (/var/log, /var/run) |
Cluster-wide log collectors (Fluentbit), monitoring agents (Node Exporter, Datadog), CNI plugins |
4. Production StatefulSet Manifest: Clustered Distributed Storage
Deploying stateful systems requires deterministic DNS networking and dedicated persistent volume provisioning. The following manifest demonstrates an enterprise clustered database using a Headless Service and volumeClaimTemplates:
# k8s/statefulset-database.yaml
apiVersion: v1
kind: Service
metadata:
name: database-headless
namespace: production
labels:
app: database
spec:
clusterIP: None # Defines a Headless Service for direct pod DNS resolution
ports:
- port: 5432
name: postgres
selector:
app: database
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: database
namespace: production
spec:
serviceName: database-headless
replicas: 3
podManagementPolicy: OrderedReady # Scales 0 -> 1 -> 2 sequentially
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: database
template:
metadata:
labels:
app: database
spec:
terminationGracePeriodSeconds: 60
containers:
- name: postgresql
image: postgres:16-alpine
ports:
- containerPort: 5432
name: postgres
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: gp3-sc # AWS EBS gp3 dynamic storage provisioner
resources:
requests:
storage: 100Gi
With this architecture, pods are assigned immutable network identities: database-0.database-headless.production.svc.cluster.local, allowing replication engines to establish stable primary-replica relationships that persist across container restarts.
5. Advanced Pod Scheduling: Multi-AZ Topology Spread & Anti-Affinity
To withstand the loss of an entire cloud availability zone (AZ), production workloads must enforce mathematical distribution across failure domains. Combining TopologySpreadConstraints with PodAntiAffinity guarantees high availability:
# Pod Spec Snippet: Multi-Zone Resilient Scheduling
topologySpreadConstraints:
# Enforce even spread across physical AWS/GCP Availability Zones
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: DoNotSchedule # Hard scheduling barrier
labelSelector:
matchLabels:
app: core-api
# Enforce even spread across underlying Bare Metal Host Hypervisors
- maxSkew: 1
topologyKey: "kubernetes.io/hostname"
whenUnsatisfiable: ScheduleAnyway # Soft preference across individual nodes
labelSelector:
matchLabels:
app: core-api
6. High-Availability Multi-Master Architecture: Stacked vs External etcd
In enterprise production clusters, a single control plane node represents a fatal single point of failure (SPOF). Production clusters implement High Availability (HA) topologies across three or five master nodes:
Stacked etcd Topology vs External etcd Topology
- Stacked etcd Topology (Recommended for Most Teams):
etcdmembers run directly on the same control plane nodes askube-apiserver, managed as static pods by kubelet. This requires fewer virtual machines (minimum 3 master nodes) and simplifies lifecycle operations usingkubeadm. - External etcd Topology (Maximum Scale):
etcdruns on dedicated, isolated bare-metal or VM nodes separated from the API servers. This isolates etcd disk I/O from API server memory consumption, ideal for clusters exceeding 1,000 worker nodes.
In front of the master nodes, an external layer 4 load balancer (such as HAProxy, AWS NLB, or Keepalived) distributes traffic across all active kube-apiserver instances on port 6443, executing TCP health checks against /livez.
7. Ingress Architecture and the Gateway API Evolution
Routing external HTTP and gRPC traffic into internal Kubernetes pod endpoints has historically relied on the Ingress resource. However, as enterprise multi-tenant clusters scaled, the legacy Ingress specification revealed severe architectural limitations: lack of native support for header-based routing, traffic splitting, TLS pass-through, and vendor-specific fragmentation through hundreds of non-standard annotations.
The Modern Gateway API Architecture
The CNCF Gateway API represents the next-generation standard for service networking in Kubernetes, decoupling operational roles into clean declarative boundaries:
GatewayClass(Infrastructure Provider): Defines the underlying controller implementation (e.g. Envoy Gateway, Cilium, Istio, or cloud load balancers). Managed by cluster administrators.Gateway(Platform / SRE Team): Defines physical or cloud load balancer attachment points, listening ports (HTTP 80, HTTPS 443), and TLS certificate termination boundaries.HTTPRoute/GRPCRoute(Application Developers): Defines routing logic, URI path prefixes, query parameter matching, and canary traffic splitting (e.g. 90% v1, 10% v2) without requiring access to modify central load balancer listeners.
# Gateway API Canary Routing Manifest
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: checkout-route
namespace: production
spec:
parentRefs:
- name: production-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /api/checkout
backendRefs:
- name: checkout-service-v1
port: 8080
weight: 90
- name: checkout-service-v2
port: 8080
weight: 10
8. Storage Architecture: Container Storage Interface (CSI) Lifecycle
Connecting distributed block storage (such as AWS EBS, Ceph RBD, or Google Persistent Disks) to ephemeral containers requires standardizing interactions between the Kubernetes control plane, the host operating system, and external storage APIs. The Container Storage Interface (CSI) executes a three-phase volume mounting lifecycle:
- ControllerPublishVolume (Attachment): The CSI controller issues an API call to the cloud provider (e.g. AWS EC2
AttachVolume) attaching the physical virtual disk to the worker node's VM hypervisor. - NodeStageVolume (Formatting & Global Mount): The CSI node driver running on the worker node formats the raw block device with a filesystem (e.g. ext4 or XFS) and mounts it to a global directory:
/var/lib/kubelet/plugins/kubernetes.io/csi/volume/.... - NodePublishVolume (Bind Mount to Pod): The kubelet executes a bind mount, projecting the staged filesystem directly into the container's private MNT namespace:
/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/....
This strict three-stage decoupling ensures that node crashes or container restarts safely unmount filesystems before releasing cloud storage attachments, preventing data corruption.
9. The Operator Pattern & Custom Resource Definitions (CRDs)
The true power of Kubernetes architecture lies in its extensibility. Through Custom Resource Definitions (CRDs), platform teams extend the Kubernetes API with proprietary domain primitives (such as KafkaCluster, VaultSecret, or PostgresCluster). An Operator implements an active control loop using the controller-runtime library that observes these custom resources, calculates state differences, and continuously orchestrates low-level Deployments, ConfigMaps, and cloud storage volumes to achieve convergence.
The Status Subresource and Optimistic Concurrency Control
Production Operators must always declare a dedicated /status subresource in their CRD manifest. Separating the user-managed spec from the controller-managed status prevents write race conditions:
// Example Reconcile Loop in Go (controller-runtime)
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var db customv1.DatabaseCluster
if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// 1. Ensure underlying StatefulSet matches desired replicas
desiredStatefulSet := r.constructStatefulSet(&db)
if err := r.reconcileStatefulSet(ctx, desiredStatefulSet); err != nil {
return ctrl.Result{RequeueAfter: time.Second * 5}, err
}
// 2. Update status subresource using optimistic concurrency
db.Status.ReadyReplicas = desiredStatefulSet.Status.ReadyReplicas
db.Status.Phase = customv1.PhaseRunning
return ctrl.Result{RequeueAfter: time.Minute * 10}, r.Status().Update(ctx, &db)
}
By leveraging Kubernetes API server resourceVersion validation, optimistic locking guarantees that two concurrent reconcilers will never corrupt cluster metadata or trigger duplicate provisioning runs.
10. Production Troubleshooting Runbook for Kubernetes Cluster Nodes
When operating Kubernetes clusters, use the following operational triage matrix to diagnose cluster-level infrastructure failures:
| Observed Node Condition | Root Cause | Immediate SRE Remediation |
|---|---|---|
NodeNotReady |
Kubelet daemon crashed, host network split, or PLEG execution latency timeout. | SSH to node. Inspect journalctl -u kubelet -n 100. Verify containerd socket response via crictl info. |
DiskPressure |
Host root filesystem or container image storage (/var/lib/containerd) exceeded 85% capacity. |
Kubelet begins evicting BestEffort pods. Execute crictl rmi --prune to remove unreferenced layers. Increase disk volume size. |
MemoryPressure |
Host unallocated memory dropped below kubelet eviction threshold (typically 100Mi). | Kubelet immediately evicts unconstrained pods. Identify runaway memory consumers with kubectl top nodes and enforce namespace ResourceQuotas. |
etcd server: leader lost |
etcd member disk fsync latency exceeded heartbeat timeout (typically 100ms) or network partitioning between quorum members. | Inspect disk I/O metrics (fdatasync_duration_seconds). Move etcd data directory to dedicated high-IOPS NVMe storage. |
11. Frequently Asked Questions
What happens when the Kubernetes control plane is down?
If all control plane nodes or etcd instances become temporarily unavailable, already running application containers continue executing normally on worker nodes. Worker node kubelets and CNI routing tables continue forwarding traffic. However, all control plane automation ceases: you cannot deploy new pods, scaling controllers cannot adjust replicas, and if a worker node crashes, its pods cannot be rescheduled onto surviving nodes until the control plane recovers.
Why should etcd clusters always contain an odd number of nodes?
etcd uses the Raft distributed consensus protocol requiring a strict majority quorum: floor(N/2) + 1. A 3-node cluster requires 2 nodes for quorum (tolerates 1 failure). A 4-node cluster requires 3 nodes for quorum (still tolerates only 1 failure). Adding a 4th node increases network synchronization overhead without providing any additional failure tolerance. Therefore, etcd clusters always scale in odd increments: 3, 5, or 7 members.
What is the difference between IPVS and iptables in kube-proxy?
iptables is a sequential packet filtering framework inside the Linux netfilter subsystem. As a cluster scales to thousands of services, iptables rules grow to tens of thousands of lines, creating an O(N) packet inspection bottleneck where every network packet must traverse the entire list sequentially. IPVS (IP Virtual Server) is an in-kernel Layer 4 transport load balancer utilizing hash tables, providing O(1) constant-time routing performance regardless of cluster scale.