DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD
CONTAINERS & CLUSTERS · 25 MIN READ
Kubernetes official logo

Kubernetes Architecture: Control Plane, Workload Orchestration & Scheduling

An enterprise platform engineering deep dive into Kubernetes architecture: master control plane internals, etcd Raft consensus, kube-scheduler filtering algorithms, worker node subsystems, declarative controllers, and multi-AZ high availability.

By DevOps Tribune Engineering Staff · Published November 17, 2026

Kubernetes has become the de facto distributed operating system of modern cloud computing. However, operating Kubernetes at enterprise scale requires peeling back the abstraction layers of kubectl to understand the underlying distributed systems architecture. Kubernetes is fundamentally a declarative state reconciliation engine: users submit a desired state manifest to a central API, and an array of autonomous, asynchronous control loops continually observe real-world cluster infrastructure, calculating the delta between desired and actual state, and issuing mutations until convergence is reached.

When engineering teams lack a deep architectural understanding of control plane internals—how worker nodes negotiate lease objects, how controllers handle backoff retries, and how the Linux kernel schedules CFS bandwidth——how etcd enforces Raft consensus, how kube-scheduler calculates node scoring, or how kubelet interacts with container runtimes via CRI and CNI plugins—production incidents become mysterious black-box outages. A subtle etcd disk fsync latency bottleneck can freeze API response times globally. An improperly tuned scheduler can concentrate memory-hungry pods onto a single availability zone, inducing widespread cascading node evictions. Conversely, a platform team with mastery over Kubernetes systems architecture designs fault-tolerant multi-master topologies that seamlessly survive physical datacenter outages.

This comprehensive architectural masterclass examines the deep engineering mechanics of Kubernetes. We detail the master control plane stack (kube-apiserver, etcd, kube-scheduler, kube-controller-manager), trace the worker node execution path through kubelet, containerd, and kube-proxy, compare declarative workload primitives (Deployments, StatefulSets, DaemonSets), examine advanced pod scheduling policies, and provide concrete production reference manifests.


1. The Master Control Plane Stack: Distributed Consensus & API Mechanics

The Kubernetes control plane manages global cluster decisions, detects and responds to cluster events, and maintains the authoritative record of all cluster objects. It comprises four primary decoupled services:

1. kube-apiserver: The Front Door and Validation Pipeline

The kube-apiserver is the only component in the entire cluster that communicates directly with etcd. All other control plane and worker node components must communicate through the API server via structured HTTPS REST endpoints. When an incoming request arrives, it undergoes a rigid four-stage sequential pipeline:

  1. Authentication: Validates client identity via X.509 client certificates, OpenID Connect (OIDC) JWT tokens, or webhook token authenticators. Maps the connection to a User, Group, and Extra metadata.
  2. Authorization (RBAC): Evaluates whether the authenticated identity has permission to perform the requested verb (get, list, create, update, delete) against the target resource API group and namespace.
  3. Mutating Admission Webhooks: Intercepts valid requests to modify or inject defaults (e.g., injecting sidecar proxies like Envoy/Istio, or setting default security contexts).
  4. Validating Admission Webhooks: Enforces strict policy constraints (e.g., rejecting images using the latest tag, or blocking privileged containers via Kyverno or OPA Gatekeeper).

Only if all four stages succeed is the object written to persistent storage in etcd.

2. etcd: The Distributed Key-Value Store

etcd is a strongly consistent, distributed key-value database implementing the Raft consensus algorithm. It stores the complete specification and status of every Kubernetes resource. Critical architectural constraints include:

  • Strict Quorum Arithmetic: An etcd cluster requires a strict majority of nodes to accept writes: Quorum = floor(N/2) + 1. A 3-node cluster tolerates 1 node failure; a 5-node cluster tolerates 2 node failures. Clusters should always utilize an odd number of members.
  • Sequential Disk fsync Sensitivity: Every committed write requires a synchronous disk write (fsync) across quorum members. If disk write latency exceeds 10ms (common on slow network-attached block storage), Raft leader heartbeats time out, triggering disruptive leader re-elections and API paralysis. High-performance NVMe SSDs are mandatory.
  • Compaction & Defragmentation: etcd maintains a complete revision history (Multi-Version Concurrency Control / MVCC). Over time, MVCC revisions exhaust the default 2 GB storage quota, triggering an NOSPACE alarm that locks the cluster into read-only mode. Production clusters must run automated compaction and defragmentation jobs.

3. kube-scheduler: The Placement Engine

The kube-scheduler monitors the API server for newly created pods that have an empty spec.nodeName field. For each unscheduled pod, the scheduler selects the optimal worker node through a two-phase mathematical pipeline:

  1. Filtering Phase (Predicates): Evaluates all nodes in the cluster against hard feasibility constraints, eliminating nodes that cannot run the pod:
    • Does the node have sufficient unallocated CPU and memory requests? (NodeResourcesFit)
    • Does the node match the pod's nodeSelector and nodeAffinity?
    • Does the node possess taints that the pod lacks tolerations for? (NodeTolerations)
    • Will placing the pod violate port collisions on the host network? (NodePorts)
  2. Scoring Phase (Priorities): Ranks all surviving feasible nodes on a score of 0 to 100 based on soft preference algorithms:
    • ImageLocality: Prefers nodes that already have the container image layers cached locally to accelerate startup.
    • NodeResourcesBalancedAllocation: Prefers nodes where placing the pod results in the most balanced ratio of CPU and memory utilization.
    • NodeAffinityScoring: Rewards nodes matching soft affinity expressions (preferredDuringSchedulingIgnoredDuringExecution).

The node with the highest cumulative score is selected. The scheduler executes a Binding call to the API server, updating spec.nodeName.

4. kube-controller-manager: The Continuous Reconciliation Loop

The kube-controller-manager consolidates multiple distinct controller loops into a single process. Each controller runs an infinite loop matching this pseudocode:

for {
    desiredState := getDesiredStateFromAPIServer()
    actualState := getActualStateFromCluster()
    if delta := desiredState - actualState; delta != 0 {
        reconcile(delta)
    }
    sleep(reconciliationInterval)
}

Key embedded controllers include: DeploymentController (scales ReplicaSets), NodeController (evicts pods when a worker node stops sending heartbeats for 40 seconds), EndpointSliceController (updates service IP routing endpoints), and NamespaceController (garbage collects resources during namespace deletion).


2. Worker Node Architecture: The Execution Engine

Worker nodes host the actual customer-facing application containers. A production worker node runs three tightly coupled subsystems:

1. kubelet: The Node Daemon

The kubelet is the primary agent running on every worker node. It registers the node with the API server, reports node health metrics (CPU, memory, disk pressure), and ensures containers described in assigned PodSpecs are running healthy. The kubelet uses the Pod Lifecycle Event Generator (PLEG) to continuously poll the local container runtime. If PLEG execution latency spikes due to container runtime locking, the node transitions into NotReady status.

2. Container Runtime Interface (CRI) & containerd

Kubernetes communicates with the local container runtime via gRPC over the CRI socket (typically /run/containerd/containerd.sock). The CRI separates container orchestration from low-level execution: containerd pulls images, manages snapshot storage, and delegates container execution to runc.

3. kube-proxy & CNI Networking

The Container Network Interface (CNI) plugin (such as Cilium, Calico, or AWS VPC CNI) assigns every pod its own unique routable IP address across the cluster. The kube-proxy daemon monitors Services and Endpoints on the API server, translating virtual Service ClusterIPs into actual pod IP addresses. In modern production clusters, kube-proxy should operate in IPVS (IP Virtual Server) mode or be completely replaced by eBPF (Cilium), bypassing the O(N) performance degradation of legacy Linux iptables.


3. Declarative Workload Primitives: Deployments vs StatefulSets vs DaemonSets

Kubernetes provides specialized controllers tailored to distinct workload characteristics:

Workload Controller Pod Identity & Naming Storage Binding Primary Production Use Cases
Deployment Stateless, interchangeable, random hashes (e.g. api-7bf89-kx2q) Shared read-only or ephemeral storage Stateless web APIs, microservices, background job consumers
StatefulSet Stable, ordered, deterministic ordinals (e.g. kafka-0, kafka-1) Dedicated, persistent PVC per ordinal via volumeClaimTemplates Distributed databases (Cassandra, PostgreSQL), Kafka clusters, ZooKeeper, Elasticsearch
DaemonSet Exactly one pod per matching node in the cluster Host-level bind mounts (/var/log, /var/run) Cluster-wide log collectors (Fluentbit), monitoring agents (Node Exporter, Datadog), CNI plugins

4. Production StatefulSet Manifest: Clustered Distributed Storage

Deploying stateful systems requires deterministic DNS networking and dedicated persistent volume provisioning. The following manifest demonstrates an enterprise clustered database using a Headless Service and volumeClaimTemplates:

# k8s/statefulset-database.yaml
apiVersion: v1
kind: Service
metadata:
  name: database-headless
  namespace: production
  labels:
    app: database
spec:
  clusterIP: None # Defines a Headless Service for direct pod DNS resolution
  ports:
    - port: 5432
      name: postgres
  selector:
    app: database
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
  namespace: production
spec:
  serviceName: database-headless
  replicas: 3
  podManagementPolicy: OrderedReady # Scales 0 -> 1 -> 2 sequentially
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgresql
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
              name: postgres
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
          resources:
            requests:
              cpu: "1000m"
              memory: "2Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: gp3-sc # AWS EBS gp3 dynamic storage provisioner
        resources:
          requests:
            storage: 100Gi

With this architecture, pods are assigned immutable network identities: database-0.database-headless.production.svc.cluster.local, allowing replication engines to establish stable primary-replica relationships that persist across container restarts.


5. Advanced Pod Scheduling: Multi-AZ Topology Spread & Anti-Affinity

To withstand the loss of an entire cloud availability zone (AZ), production workloads must enforce mathematical distribution across failure domains. Combining TopologySpreadConstraints with PodAntiAffinity guarantees high availability:

# Pod Spec Snippet: Multi-Zone Resilient Scheduling
topologySpreadConstraints:
  # Enforce even spread across physical AWS/GCP Availability Zones
  - maxSkew: 1
    topologyKey: "topology.kubernetes.io/zone"
    whenUnsatisfiable: DoNotSchedule # Hard scheduling barrier
    labelSelector:
      matchLabels:
        app: core-api

  # Enforce even spread across underlying Bare Metal Host Hypervisors
  - maxSkew: 1
    topologyKey: "kubernetes.io/hostname"
    whenUnsatisfiable: ScheduleAnyway # Soft preference across individual nodes
    labelSelector:
      matchLabels:
        app: core-api

6. High-Availability Multi-Master Architecture: Stacked vs External etcd

In enterprise production clusters, a single control plane node represents a fatal single point of failure (SPOF). Production clusters implement High Availability (HA) topologies across three or five master nodes:

Stacked etcd Topology vs External etcd Topology

  • Stacked etcd Topology (Recommended for Most Teams): etcd members run directly on the same control plane nodes as kube-apiserver, managed as static pods by kubelet. This requires fewer virtual machines (minimum 3 master nodes) and simplifies lifecycle operations using kubeadm.
  • External etcd Topology (Maximum Scale): etcd runs on dedicated, isolated bare-metal or VM nodes separated from the API servers. This isolates etcd disk I/O from API server memory consumption, ideal for clusters exceeding 1,000 worker nodes.

In front of the master nodes, an external layer 4 load balancer (such as HAProxy, AWS NLB, or Keepalived) distributes traffic across all active kube-apiserver instances on port 6443, executing TCP health checks against /livez.


7. Ingress Architecture and the Gateway API Evolution

Routing external HTTP and gRPC traffic into internal Kubernetes pod endpoints has historically relied on the Ingress resource. However, as enterprise multi-tenant clusters scaled, the legacy Ingress specification revealed severe architectural limitations: lack of native support for header-based routing, traffic splitting, TLS pass-through, and vendor-specific fragmentation through hundreds of non-standard annotations.

The Modern Gateway API Architecture

The CNCF Gateway API represents the next-generation standard for service networking in Kubernetes, decoupling operational roles into clean declarative boundaries:

  • GatewayClass (Infrastructure Provider): Defines the underlying controller implementation (e.g. Envoy Gateway, Cilium, Istio, or cloud load balancers). Managed by cluster administrators.
  • Gateway (Platform / SRE Team): Defines physical or cloud load balancer attachment points, listening ports (HTTP 80, HTTPS 443), and TLS certificate termination boundaries.
  • HTTPRoute / GRPCRoute (Application Developers): Defines routing logic, URI path prefixes, query parameter matching, and canary traffic splitting (e.g. 90% v1, 10% v2) without requiring access to modify central load balancer listeners.
# Gateway API Canary Routing Manifest
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-route
  namespace: production
spec:
  parentRefs:
    - name: production-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/checkout
      backendRefs:
        - name: checkout-service-v1
          port: 8080
          weight: 90
        - name: checkout-service-v2
          port: 8080
          weight: 10

8. Storage Architecture: Container Storage Interface (CSI) Lifecycle

Connecting distributed block storage (such as AWS EBS, Ceph RBD, or Google Persistent Disks) to ephemeral containers requires standardizing interactions between the Kubernetes control plane, the host operating system, and external storage APIs. The Container Storage Interface (CSI) executes a three-phase volume mounting lifecycle:

  1. ControllerPublishVolume (Attachment): The CSI controller issues an API call to the cloud provider (e.g. AWS EC2 AttachVolume) attaching the physical virtual disk to the worker node's VM hypervisor.
  2. NodeStageVolume (Formatting & Global Mount): The CSI node driver running on the worker node formats the raw block device with a filesystem (e.g. ext4 or XFS) and mounts it to a global directory: /var/lib/kubelet/plugins/kubernetes.io/csi/volume/....
  3. NodePublishVolume (Bind Mount to Pod): The kubelet executes a bind mount, projecting the staged filesystem directly into the container's private MNT namespace: /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/....

This strict three-stage decoupling ensures that node crashes or container restarts safely unmount filesystems before releasing cloud storage attachments, preventing data corruption.


9. The Operator Pattern & Custom Resource Definitions (CRDs)

The true power of Kubernetes architecture lies in its extensibility. Through Custom Resource Definitions (CRDs), platform teams extend the Kubernetes API with proprietary domain primitives (such as KafkaCluster, VaultSecret, or PostgresCluster). An Operator implements an active control loop using the controller-runtime library that observes these custom resources, calculates state differences, and continuously orchestrates low-level Deployments, ConfigMaps, and cloud storage volumes to achieve convergence.

The Status Subresource and Optimistic Concurrency Control

Production Operators must always declare a dedicated /status subresource in their CRD manifest. Separating the user-managed spec from the controller-managed status prevents write race conditions:

// Example Reconcile Loop in Go (controller-runtime)
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var db customv1.DatabaseCluster
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 1. Ensure underlying StatefulSet matches desired replicas
    desiredStatefulSet := r.constructStatefulSet(&db)
    if err := r.reconcileStatefulSet(ctx, desiredStatefulSet); err != nil {
        return ctrl.Result{RequeueAfter: time.Second * 5}, err
    }

    // 2. Update status subresource using optimistic concurrency
    db.Status.ReadyReplicas = desiredStatefulSet.Status.ReadyReplicas
    db.Status.Phase = customv1.PhaseRunning
    return ctrl.Result{RequeueAfter: time.Minute * 10}, r.Status().Update(ctx, &db)
}

By leveraging Kubernetes API server resourceVersion validation, optimistic locking guarantees that two concurrent reconcilers will never corrupt cluster metadata or trigger duplicate provisioning runs.


10. Production Troubleshooting Runbook for Kubernetes Cluster Nodes

When operating Kubernetes clusters, use the following operational triage matrix to diagnose cluster-level infrastructure failures:

Observed Node Condition Root Cause Immediate SRE Remediation
NodeNotReady Kubelet daemon crashed, host network split, or PLEG execution latency timeout. SSH to node. Inspect journalctl -u kubelet -n 100. Verify containerd socket response via crictl info.
DiskPressure Host root filesystem or container image storage (/var/lib/containerd) exceeded 85% capacity. Kubelet begins evicting BestEffort pods. Execute crictl rmi --prune to remove unreferenced layers. Increase disk volume size.
MemoryPressure Host unallocated memory dropped below kubelet eviction threshold (typically 100Mi). Kubelet immediately evicts unconstrained pods. Identify runaway memory consumers with kubectl top nodes and enforce namespace ResourceQuotas.
etcd server: leader lost etcd member disk fsync latency exceeded heartbeat timeout (typically 100ms) or network partitioning between quorum members. Inspect disk I/O metrics (fdatasync_duration_seconds). Move etcd data directory to dedicated high-IOPS NVMe storage.

11. Frequently Asked Questions

What happens when the Kubernetes control plane is down?

If all control plane nodes or etcd instances become temporarily unavailable, already running application containers continue executing normally on worker nodes. Worker node kubelets and CNI routing tables continue forwarding traffic. However, all control plane automation ceases: you cannot deploy new pods, scaling controllers cannot adjust replicas, and if a worker node crashes, its pods cannot be rescheduled onto surviving nodes until the control plane recovers.

Why should etcd clusters always contain an odd number of nodes?

etcd uses the Raft distributed consensus protocol requiring a strict majority quorum: floor(N/2) + 1. A 3-node cluster requires 2 nodes for quorum (tolerates 1 failure). A 4-node cluster requires 3 nodes for quorum (still tolerates only 1 failure). Adding a 4th node increases network synchronization overhead without providing any additional failure tolerance. Therefore, etcd clusters always scale in odd increments: 3, 5, or 7 members.

What is the difference between IPVS and iptables in kube-proxy?

iptables is a sequential packet filtering framework inside the Linux netfilter subsystem. As a cluster scales to thousands of services, iptables rules grow to tens of thousands of lines, creating an O(N) packet inspection bottleneck where every network packet must traverse the entire list sequentially. IPVS (IP Virtual Server) is an in-kernel Layer 4 transport load balancer utilizing hash tables, providing O(1) constant-time routing performance regardless of cluster scale.

Related Kubernetes Operational & SRE Guides