Most interview prep material regurgitates basic definitions: "What is Docker?" or "Explain the difference between Continuous Delivery and Continuous Deployment." In real enterprise technical screens for Senior DevOps, Staff Platform, and SRE roles, hiring managers assess your troubleshooting instincts, mental models, and architectural judgement under pressure.

This guide compiles 25 rigorous scenario-based questions organized into five critical architectural pillars: Linux kernel diagnostics, Kubernetes control plane and pod internals, CI/CD supply-chain security, Infrastructure as Code reliability, and Site Reliability Engineering incident resolution.


Pillar 1: Linux Kernel, Memory & System Performance

Scenario 1: High Load Average with Low CPU Utilization

The Question: "You run uptime on a Linux production node and see a 1-minute load average of 38.0 on an 8-core server. However, running top shows CPU idle time is 92%. What does this indicate, and how do you isolate the root cause?"

Senior Answer: Linux load average does not measure CPU usage alone; it counts processes in the Task Run state (R) plus processes in the Uninterruptible Sleep state (D). When CPU utilization is near zero while load average spikes, tasks are blocked waiting on uninterruptible I/O—typically disk subsystems, network storage (NFS/EFS), or kernel lock contention.

Triage steps:

  1. Check disk I/O wait with vmstat 1 5 or iostat -xz 1. Inspect the %iowait and %util columns to determine if a block device is saturated.
  2. Identify processes stuck in state TASK_UNINTERRUPTIBLE (D) using ps -eo pid,user,stat,wchan:20,comm | grep ' D'. The wchan field identifies the kernel function waiting on I/O.
  3. Inspect kernel traces directly via cat /proc/<PID>/stack to see the exact call stack (e.g., btrfs_wait_ordered_extents or nfs_wait_on_request).
  4. Inspect kernel ring buffer via dmesg -T | grep -E 'hung_task|scsi|I/O error' for disk hardware failures or controller timeouts.

Scenario 2: The Difference Between Memory Buffers, Page Cache, and Slab

The Question: "A monitoring alert triggers stating that a production host has only 4% 'Free' memory remaining. Should you immediately page the team to restart services or add RAM?"

Senior Answer: No. Linux follows a fundamental design principle: unused RAM is wasted RAM. The kernel dynamically repurposes idle memory for the Page Cache (caching file-backed disk reads) and Buffers (caching raw disk block metadata). When an application requests heap memory, the kernel instantly reclaims clean page cache with near-zero latency.

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           64120       12400        1840         320       49880       50960
Swap:              0           0           0

Always evaluate available memory, not free. In the example above, while free is only 1,840 MB (~3%), available is 50,960 MB (~79%). Paging should only trigger when available drops below critical thresholds or when system memory pressure causes aggressive swap thrashing or high vmpressure kernel events.

Scenario 3: Resolving Socket Exhaustion & TCP TIME_WAIT Spikes

The Question: "An API gateway microservice handling 15,000 requests per second begins throwing connect: cannot assign requested address (EADDRNOTAVAIL) errors when dialing downstream backends. Why is this happening, and how do you fix it?"

Senior Answer: This is classic ephemeral port exhaustion caused by accumulated sockets in the TIME_WAIT state. By default, after a TCP connection closes, the active closer retains the 4-tuple (source IP, source port, dest IP, dest port) in TIME_WAIT for 2 MSL (typically 60 seconds) to ensure delayed duplicate packets from the previous connection do not corrupt a new connection.

When clients or proxies establish short-lived HTTP/1.0 or un-pooled HTTP/1.1 connections, all ephemeral ports in net.ipv4.ip_local_port_range (typically 32768 to 60999 = 28,231 ports) are consumed within 30 seconds.

Architectural Solutions:

  • HTTP Keep-Alive & Connection Pooling (Primary Fix): Configure client HTTP connection pools with persistent keep-alive connections to reuse established sockets.
  • Kernel Tuning: Enable net.ipv4.tcp_tw_reuse = 1 in sysctl. This allows the kernel to safely reuse TIME_WAIT sockets for outgoing connections when timestamp extensions (RFC 1323) confirm monotonic sequences.
  • Port Range Expansion: Expand local ephemeral port range: sysctl -w net.ipv4.ip_local_port_range="1024 65535".

Pillar 2: Kubernetes Control Plane & Workload Troubleshooting

Scenario 4: Etcd Split-Brain & Raft Quorum Failure

The Question: "In a 3-node self-managed etcd control plane, network connectivity between Node 1 and Node 2/3 drops. Can the cluster still process read and write requests?"

Senior Answer: Etcd uses the Raft consensus algorithm, requiring a strict majority quorum formula: Q = floor(N/2) + 1. For a 3-node cluster, quorum is 2.

  • Partition with Node 2 & Node 3: Has 2 of 3 members (meets quorum). Node 2 and Node 3 elect a leader, continue accepting write transactions, update state, and commit to disk.
  • Partition with Node 1: Has only 1 member (below quorum). Node 1 cannot elect a leader or commit writes. All write requests to kube-apiserver communicating with Node 1 will fail with etcdserver: leader lost or 500 Internal Server Error.

If two nodes permanently fail in a 3-node cluster, the remaining single node cannot establish quorum. Recovery requires forcing a new single-node cluster state via etcdctl snapshot restore --skip-hash-check or reconfiguring cluster membership via etcdctl member remove.

Scenario 5: Admission Webhook Outage Freezing the Control Plane

The Question: "An engineer deletes a namespace running a security policy controller (e.g., Kyverno or OPA Gatekeeper). Suddenly, no team can deploy or delete pods in any namespace across the cluster. Why is this happening, and how do you restore operations?"

Senior Answer: The security controller registered a cluster-wide ValidatingWebhookConfiguration or MutatingWebhookConfiguration with failurePolicy: Fail and a broad namespaceSelector. When kube-apiserver receives a pod creation request, it attempts to call the webhook's Service endpoint. Because the underlying pods and namespace were terminated, the webhook endpoint returns connection refused or times out after 10 seconds.

Because failurePolicy was set to Fail (rather than Ignore), the apiserver blocks the request, rejecting all workload operations across the cluster.

Immediate Recovery:

# Bypass webhook validation by deleting the blocking configuration directly
$ kubectl get validatingwebhookconfigurations
$ kubectl delete validatingwebhookconfiguration <faulty-webhook-name>
$ kubectl delete mutatingwebhookconfiguration <faulty-webhook-name>

Architectural Safeguard: Always configure webhook configurations with explicit namespace exclusions (e.g. kube-system, security controller's own namespace) and establish emergency break-glass RBAC contexts.

Scenario 6: Java / Go Runtime OOMKilled vs Heap Sizing

The Question: "A Java microservice in Kubernetes has an application heap limit set to -Xmx2g and a pod memory limit of 2Gi. The pod repeatedly terminates with Exit Code 137. Why is it failing if heap is capped at 2GB?"

Senior Answer: -Xmx2g sets the upper bound only for the JVM Heap. Total container memory consumption is defined by:

Total JVM Memory = Heap + Metaspace + Native Threads (Stack * Threads) + Direct Byte Buffers + JIT Code Cache + Garbage Collector Overhead + Cgo/OS Libraries

If 200 threads are active, with -Xss1m (1MB stack per thread), thread stacks alone consume 200MB outside the heap. Metaspace and netty buffers can add another 300MB. As soon as total resident set size (RSS) hits 2,048 MB, the Linux kernel cgroup v2 controller triggers an out-of-memory killer (OOM) signal (SIGKILL, exit code 128 + 9 = 137).

Correction: Apply the 75% rule: if the container memory limit is 2Gi, set -XX:MaxRAMPercentage=70.0 or cap -Xmx1536m to leave sufficient headroom for off-heap allocations.


Pillar 3: CI/CD Supply Chain Security & Identity

Scenario 7: Eliminating Static AWS IAM Secret Keys in CI/CD

The Question: "Security audits require removing all long-lived AWS Access Keys and Secret Keys stored in GitHub Secrets. How do you design keyless authentication for GitHub Actions to deploy infrastructure via Terraform?"

Senior Answer: Implement OpenID Connect (OIDC) Federated Identity. GitHub Actions acts as an OIDC Identity Provider (IdP) that issues short-lived, cryptographically signed JSON Web Tokens (JWTs) for every workflow job.

  1. Configure an IAM OIDC Identity Provider in AWS pointing to https://token.actions.githubusercontent.com with client ID sts.amazonaws.com.
  2. Create an IAM Role with an AssumeRoleWithWebIdentity trust policy restricted by exact repository claims:
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:my-org/my-infra-repo:ref:refs/heads/main"
      }
    }
  3. In the GitHub Actions workflow, use aws-actions/configure-aws-credentials@v4 with role-to-assume. The runner exchanges the short-lived GitHub JWT for temporary AWS STS credentials (valid for 1 hour max), completely eliminating static secrets.

Scenario 8: Cryptographic Container Image Verification with Cosign

The Question: "How do you ensure that only container images built and scanned in your verified CI pipeline can run on production Kubernetes clusters, blocking unauthorized developer or third-party images?"

Senior Answer: Implement automated container signing and admission verification via Sigstore Cosign and a Kubernetes admission controller (such as Kyverno or Ratify):

  1. Build & Sign in CI: After building the container and verifying vulnerability scans (Trivy/Grype), the pipeline runs:
    cosign sign --key kms://aws/kms/... $REGISTRY/$IMAGE:$TAG
    The digital signature and Software Bill of Materials (SBOM) are stored in the OCI registry alongside the image manifest.
  2. Enforce at Admission: Deploy a Kyverno ClusterPolicy that intercepts every pod creation. The policy verifies that the image digest has a valid signature signed by the platform team's KMS key. Unsigned or tampered images are rejected at admission before containerd attempts to pull them.

Pillar 4: Infrastructure as Code & Cloud Reliability

Scenario 9: Resolving Deadlocked Terraform State Locks

The Question: "A continuous integration pipeline applying Terraform crashed mid-execution. Subsequent runs fail with: Error: Error acquiring the state lock: ConditionalCheckFailedException. How do you safely resolve this?"

Senior Answer: Terraform uses DynamoDB (on AWS) or Consul to enforce state locking and prevent concurrent mutating modifications from corrupting state files.

  1. Verify No Running Processes: Confirm that the crashed CI job is genuinely dead and no other engineer is executing a local apply. Check AWS CloudTrail or CI runner logs.
  2. Extract Lock ID: The error output provides an explicit Lock Info: ID: 52a12b....
  3. Release the Lock: Run terraform force-unlock <LOCK_ID>. Never delete the DynamoDB lock row manually through the AWS console unless force-unlock fails due to credential or backend issues.
  4. Run Refresh / Plan: Immediately run terraform plan to reconcile any state discrepancies caused by the interrupted apply.

Scenario 10: Zero-Downtime Database Migrations (Expand/Contract Pattern)

The Question: "You need to rename a column in a PostgreSQL table with 80 million rows serving 10,000 active transactions per second. How do you deploy this without table locks or downtime?"

Senior Answer: Executing ALTER TABLE users RENAME COLUMN phone TO mobile; acquires an ACCESS EXCLUSIVE lock. Under high traffic, this blocks all incoming queries, creates connection pool backpressure, and brings down the application.

Instead, follow the Expand and Contract Pattern across multiple releases:

  1. Phase 1 (Expand): Add the new column as nullable: ALTER TABLE users ADD COLUMN mobile VARCHAR(20);. (In modern Postgres, adding a nullable column without a volatile default takes microseconds).
  2. Phase 2 (Dual-Write): Deploy an updated application version that writes to both phone and mobile, but still reads from phone.
  3. Phase 3 (Backfill): Run an asynchronous, batched background migration to copy historic data from phone to mobile in batches of 5,000 rows with pg_sleep intervals to avoid replica lag and I/O spikes.
  4. Phase 4 (Switch Reads): Deploy application version reading from mobile and writing to mobile.
  5. Phase 5 (Contract): Drop old column phone safely in a maintenance window.

Pillar 5: SRE, Observability & Incident Response

Scenario 11: Multi-Window Multi-Burn-Rate Alerting

The Question: "Traditional static alerts on 'Error rate > 1%' generate constant noise or miss rapid outages. How does Site Reliability Engineering design modern burn rate alerting based on Service Level Objectives (SLOs)?"

Senior Answer: Google SRE introduced Multi-Window Multi-Burn-Rate Alerting. Instead of alerting on an instantaneous threshold, you monitor how fast an incident consumes your 30-day Error Budget.

  • 14.4x Burn Rate (Paging Alert): An outage consuming 2% of the total monthly error budget within 1 hour. Requires a 1-hour short window AND a 5-minute confirmation window to avoid flapping.
  • 6x Burn Rate (Ticket/Warning): Consumes 5% of error budget in 6 hours. Requires a 6-hour long window AND a 30-minute confirmation window.

This dual-window approach guarantees that alerts trigger only when an incident is both statistically severe and sustained, completely eliminating false positives from brief transient network spikes.

Scenario 12: Preventing Cascading Failures with Decorrelated Jitter

The Question: "A downstream payment service suffers a 2-minute outage. When it recovers, upstream microservices immediately crash it again in an endless loop. What caused this, and how is it prevented?"

Senior Answer: This is a Thundering Herd (Retry Storm). When the service recovered, hundreds of client instances executed retries at identical fixed intervals, generating an instantaneous traffic surge that overwhelmed connection pools and thread workers.

Engineering Controls:

  1. Exponential Backoff with Full / Decorrelated Jitter: Never retry at fixed intervals. Add randomized jitter:
    sleep = min(cap, random_between(base, sleep * 3))
    Jitter smooths retries across a uniform timeline, preventing resonant spikes.
  2. Circuit Breakers: Implement Envoy or resilience4j circuit breakers that trip to an OPEN state after a consecutive failure threshold, failing fast immediately without transmitting network traffic downstream until health probes succeed.
  3. Server-Side Rate Limiting & Shedding: Apply token bucket algorithms or concurrency limits (e.g. Envoy adaptive concurrency limits) to shed excess load with HTTP 429 / 503 instead of crashing.

Summary: The Senior Platform Engineer Mindset

In high-bar interviews, interviewers are listening for how you think about trade-offs, how you systematically isolate layers of the OSI stack, and whether your operational instincts minimize blast radius. Never jump to restarting a service before gathering forensics; understand the system, inspect the telemetry, and implement durable architectural controls.