Production Monitoring & Alerting Troubleshooting: SRE Triage Runbook
An in-depth SRE runbook for diagnosing and repairing production observability failures: resolving silent alert black holes, Prometheus TSDB ingestion bottlenecks, high-cardinality crashes, and log backpressure.
In modern high-scale distributed systems, the monitoring and alerting pipeline is the vital nervous system of the infrastructure. When production services experience degradation, platform teams depend entirely on real-time metrics, distributed traces, and aggregated log streams to detect anomalies, quantify business impact, and identify root causes. However, the observability infrastructure itself is a complex, distributed, stateful software system—and it is equally vulnerable to catastrophic failure. There is no worse scenario for an engineering organization than discovering a major customer-facing outage not from an automated PagerDuty alert, but from an angry tweet or an escalated executive email because the monitoring pipeline silently failed hours earlier.
Production monitoring failures typically fall into two destructive extremes: silent alert failures, where critical thresholds breach but alerts are swallowed by misconfigured Alertmanager routing trees or dead man's snitch black holes; and alert fatigue, where thousands of noisy, non-actionable notifications bombard on-call engineers until real incidents are ignored. Behind these symptoms lie deep systems bottlenecks: Prometheus Time Series Database (TSDB) write stalls caused by high-cardinality label explosions, disk I/O starvation on Write-Ahead Logs (WAL), asynchronous distributed trace context stripping across thread boundaries, and log aggregation buffer drops under sudden traffic spikes.
This masterclass provides an exhaustive SRE troubleshooting runbook for debugging and hardening production monitoring and alerting systems. We dissect telemetry ingestion architectures, resolve Alertmanager routing failures, optimize high-cardinality TSDB memory footprints, implement Google SRE multi-window multi-burn-rate error budget alerting, and deliver verified production configurations to ensure your monitoring systems remain resilient under extreme operational pressure.
1. The Anatomy of Telemetry Failures: Metrics, Logs, and Traces
Observability relies on the three pillars of telemetry, each with distinct failure modes that require specific diagnostic approaches:
| Telemetry Pillar | Ingestion Mechanism | Primary Failure Symptom | Root System Bottleneck |
|---|---|---|---|
| Metrics (Prometheus / VictoriaMetrics) | Pull-based HTTP scrapes or OTLP push | Scrape gaps, delayed dashboards, OOM kills | High-cardinality label explosion; Head chunk compaction I/O wait. |
| Alerts (Alertmanager / PagerDuty) | Push webhooks from rule evaluation engines | Silent failure during outages; notification storm | Mismatched receiver matchers; missing Watchdog heartbeat; flapping thresholds. |
| Logs (Grafana Loki / Fluent Bit / Vector) | Agent log tailing & HTTP push batches | Missing incident logs, rate-limit 429 drops | Buffer disk overflow; dynamic stream label cardinality; ingestion ring quorum loss. |
| Traces (OpenTelemetry / Jaeger / Tempo) | In-process SDK instrumentation & OTLP collector | Broken trace spans; missing root context | W3C header propagation stripped by proxy; aggressive head-based sampling drops. |
2. Triage Category 1: Silent Alert Failures & Alertmanager Routing Traps
A silent alert failure occurs when a Prometheus rule successfully enters the FIRING state, but zero notifications arrive in Slack, PagerDuty, or email. The failure resides inside the Alertmanager routing tree or notification dispatch pipeline.
1. The Subtree Early-Exit Trap
Alertmanager routes alerts hierarchically starting at the root route. By default, when an alert matches a child route, Alertmanager stops evaluation and does not evaluate subsequent sibling routes unless continue: true is explicitly configured:
# Alertmanager Production Routing Definition
route:
receiver: 'slack-default'
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
# Trap: If an alert matches 'team: platform', it sends to Slack,
# but NEVER reaches PagerDuty unless continue is set to true!
- match:
team: platform
receiver: 'slack-platform-channel'
continue: true # Mandatory to allow critical alerts to reach PagerDuty
- match:
severity: critical
receiver: 'pagerduty-high-urgency'
2. Dead Man's Snitch / Watchdog Heartbeat Monitoring
How do you know if your monitoring system is dead? If Prometheus stops running or Alertmanager crashes, it cannot send an alert saying it has crashed. Resilient SRE architectures mandate a Watchdog alert (Dead Man's Snitch):
- Configure an alert rule that is permanently firing 24/7/365:
vector(1) > 0withalertname: Watchdog. - Route this alert to an external heartbeat monitoring service (such as Healthchecks.io, Dead Man's Snitch, or PagerDuty Heartbeats).
- The external service expects a webhook ping every 60 seconds. If a ping fails to arrive within 2 minutes, the external third-party service directly pages the primary SRE on-call team, alerting them that the entire internal observability cluster is offline.
3. Triage Category 2: Prometheus TSDB Memory Explosions & High Cardinality
Prometheus stores active time series in RAM inside memory chunks before flushing them to immutable two-hour blocks on disk. The memory footprint of Prometheus is directly proportional to the number of active time series. High cardinality occurs when a developer inadvertently injects high-entropy labels into a metric:
# Catastrophic Anti-Pattern: High-Cardinality User ID and Order ID Labels
# If 100,000 users place orders, this creates 100,000 distinct time series in RAM!
http_requests_total{method="POST", path="/checkout", user_id="u_98472", order_id="ord_12894"} 1
Within minutes, the Prometheus memory usage spikes from 4 GB to 32 GB, triggering Linux kernel Out-Of-Memory (OOM) killer terminations (Exit Code 137). When Prometheus restarts, it must replay the Write-Ahead Log (WAL), which consumes massive memory, immediately triggering another OOM kill in a continuous crash loop.
Investigating TSDB High Cardinality via API
Query the Prometheus TSDB runtime status directly to identify the top offending metric names and label values:
# Query top 10 series with highest memory footprint
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[0:10]'
# Query labels with highest number of unique values
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName[0:10]'
Emergency Triage and Remediation
- Drop Labels at Scrape Time via metric_relabel_configs: Drop offending high-cardinality labels before they enter the TSDB:
metric_relabel_configs: - regex: "(user_id|order_id|session_token)" action: labeldrop - Recovering from WAL Replay OOM Crashes: If Prometheus is trapped in an OOM restart loop, temporarily allocate swap space or run Prometheus with
--storage.tsdb.wal-compressionand increase pod memory limits to allow the replay to finish cleanly. In extreme emergency disasters where historical data can be sacrificed to restore production alerting, rename or truncate the corruptedwal/directory.
4. Triage Category 3: Distributed Tracing & Missing Context Propagation
In microservice architectures, an incoming user HTTP request may traverse an API gateway, authentication service, order service, payment gateway, and database. Distributed tracing connects these isolated operations into a single directed graph through trace context propagation.
The Broken Trace Problem
When engineers inspect a trace in Jaeger or Grafana Tempo and observe that child spans are disconnected or missing entirely, the cause is almost always context stripping:
- Unpropagated HTTP Headers: Under the W3C TraceContext standard, distributed tracing relies on the
traceparentheader (e.g.,traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01). If an intermediate reverse proxy (like an older NGINX ingress or API gateway) strips unrecognized headers, downstream services generate a brand-new trace ID, breaking the distributed trace graph. - Asynchronous Thread Handoffs: In languages like Java, Go, or Node.js, if a request handler spawns an asynchronous background goroutine, thread pool worker, or message queue publisher without passing the trace
context.Context, the active span is detached from the background job.
Verification Procedure
Verify trace context headers directly using curl with the traceparent header against your gateway:
curl -v -H "traceparent: 00-11111111111111111111111111111111-2222222222222222-01" https://api.devopstribune.com/orders
Inspect backend container logs to ensure the trace ID 11111111111111111111111111111111 is preserved across all microservices.
5. Triage Category 4: Log Aggregation Backpressure & Buffer Overflows
Log forwarding agents (such as Fluent Bit, Vector, or Logstash) tail log files from container stdout/stderr. When an application encounters a sudden database outage, it often begins logging hundreds of thousands of exceptions per second. If the central log storage (e.g., Grafana Loki or Elasticsearch) experiences write latency, the log forwarding pipeline faces extreme backpressure.
Preventing Memory Exhaustion and Log Loss
If log agents buffer messages exclusively in memory, sudden ingestion spikes cause the log agent to crash with an OOM error, dropping all logs generated during the critical minutes of the outage. Production log forwarders must configure disk-backed buffering:
# Fluent Bit Production Disk Buffer Configuration
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 50MB # Memory ceiling
Buffer_Chunk_Size 512k
Buffer_Max_Size 5M
Storage.type filesystem # Spill over to host disk when memory ceiling is reached
[OUTPUT]
Name loki
Match *
Host loki-gateway.monitoring.svc.cluster.local
Port 3100
Retry_Limit 10
Workers 4
With filesystem storage enabled, if Loki throttles incoming requests with HTTP 429 Rate Limit Exceeded, Fluent Bit safely buffers logs to local disk storage on the worker node, draining the buffer smoothly once downstream storage recovers.
6. Multi-Window Multi-Burn-Rate SLO Alerting (Google SRE Standard)
Traditional alerting asserts simple static rules: "Page on-call if error rate > 2% for 5 minutes." This approach is severely flawed: a 5-minute transient spike causes an unnecessary wake-up page, while a slow 0.5% leak that drains 100% of the monthly error budget over 3 days never triggers an alert.
Google SRE standardized Multi-Window Multi-Burn-Rate Alerting. The burn rate represents how fast the service is consuming its allowable Service Level Objective (SLO) error budget:
- Burn Rate 1: Consumes 100% of the 30-day budget in exactly 30 days.
- Burn Rate 14.4: Consumes 100% of the budget in 2 days (2% of budget consumed per hour). Action: Page on-call immediately!
- Burn Rate 6: Consumes 100% of the budget in 5 days (5% consumed in 6 hours). Action: Page during business hours.
- Burn Rate 1: Consumes budget in 30 days. Action: Generate a ticket for sprint planning.
Production Multi-Burn-Rate PromQL Alert Rule
To avoid false positives, the alert requires two concurrent evaluation windows: a long window (to ensure sufficient budget consumption) and a short window (to verify the outage is actively continuing):
# Multi-Window Multi-Burn-Rate Alert Rule (14.4x Burn Rate on 99.9% SLO)
- alert: HighErrorBudgetBurnRate
expr: |
(
# Short window: 5 minutes (verifies error is actively firing)
sum(rate(http_requests_total{status=~"5.*"}[5m]))
/
sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)
and
(
# Long window: 1 hour (verifies significant budget consumption)
sum(rate(http_requests_total{status=~"5.*"}[1h]))
/
sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
)
for: 2m
labels:
severity: critical
team: platform
annotations:
summary: "High error budget burn rate (14.4x) on API service"
description: "Service has consumed more than 2% of its monthly error budget in the past hour."
7. Observability High Availability & Multi-Cluster Architecture (Thanos & Mimir)
A single Prometheus server constitutes a single point of failure (SPOF) and is fundamentally bounded by the memory and disk capacity of a single physical or virtual machine. When managing hundreds of Kubernetes clusters across multiple cloud regions, platform teams must evolve from isolated Prometheus instances to a federated, highly available architecture using Thanos or Grafana Mimir.
1. Thanos Architecture: Decentralized Long-Term Object Storage
Thanos runs alongside Prometheus as modular microservices, seamlessly extending Prometheus with global querying and infinite historical retention:
- Thanos Sidecar: Deployed as a sidecar container in the Prometheus pod. It intercepts PromQL queries from the central querier and continuously ships immutable 2-hour TSDB blocks directly to Amazon S3 or Google Cloud Storage.
- Thanos Store Gateway: Acts as an efficient read proxy over the historical metric blocks in object storage, caching index headers in memory to answer long-term PromQL range queries (e.g., year-over-year capacity forecasts) in sub-seconds.
- Thanos Compactor: A singleton background worker that periodically merges historical 2-hour blocks into larger chunks and performs mathematical downsampling (aggregating raw 15-second data into 5-minute and 1-hour resolution blocks), slashing object storage retrieval costs by over 90%.
- Thanos Querier: Exposes a standard Prometheus v1 API endpoint that deduplicates concurrent time series scraped by redundant Prometheus replicas (Replica A and Replica B), providing a single pane of glass for global multi-cluster dashboards.
8. Eradicating Alert Fatigue: Inhibition Rules & Flapping Suppression
Alert fatigue is the silent killer of engineering operational velocity. When an on-call engineer is awakened at 3:00 AM by 40 distinct PagerDuty notifications for the same underlying issue, cognitive overload occurs, leading to delayed incident response times and burnout. Production Alertmanager configurations eliminate alert noise through two critical mechanisms:
1. Alertmanager Inhibition Rules
Inhibition rules suppress lower-severity alerts when a higher-severity alert for the same system or failure domain is already actively firing. For example, if an entire Kubernetes node crashes (NodeDown), Alertmanager should not page engineers for 20 individual pod readiness probe failures running on that node:
# Production Alertmanager Inhibition Rule
inhibit_rules:
# Suppress all pod-level alerts if the parent node is down
- source_match:
alertname: 'NodeDown'
target_match_re:
alertname: '(KubePodCrashLooping|KubePodNotReady|TargetMissing)'
equal: ['node', 'cluster']
# Suppress warning alerts if a critical alert is already firing for the same service
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'service', 'namespace']
2. Flapping Alert Suppression
An alert that fires, resolves, fires again, and resolves within minutes is considered flapping. Flapping alerts are typically caused by volatile metric evaluations hovering around a sharp threshold without hysteresis (e.g., CPU utilization bouncing between 79.8% and 80.2%). Resolve flapping by implementing a sustained evaluation window via the for: clause (e.g., for: 10m) and adding hysteresis: fire when metric > 85%, but only resolve when metric drops below 75% for 15 minutes.
9. Step-by-Step Observability Incident Triage Runbook
When the monitoring system itself fails during an active production outage, follow this structured emergency recovery protocol:
- Step 1: Verify Scrape Target Health: Navigate to
http://prometheus:9090/targetsor query the API:curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'. Verify whether targets are failing due to DNS resolution, network timeouts, or authorization headers. - Step 2: Inspect Prometheus Head Chunks & WAL: Check if disk I/O bottlenecks are stalling memory chunk compaction:
iostat -xz 1 5. If%utilis 100%, the underlying EBS volume or SSD is IOPS-throttled. Temporarily provision higher IOPS (e.g., AWS gp3 to io2). - Step 3: Test Alert Dispatch Path: Use the
amtoolcommand-line utility to inject a synthetic test alert through Alertmanager to verify that external notification webhooks (Slack, PagerDuty, Opsgenie) are operational. - Step 4: Check OpenTelemetry Collector Buffers: Inspect collector metrics:
otelcol_exporter_enqueue_failed_spansandotelcol_exporter_send_failed_spans. If numbers are increasing, scale the collector horizontal replicas and increase memory batch buffers.
10. Complete Production OpenTelemetry Collector Pipeline Configuration
The OpenTelemetry Collector is the vendor-agnostic proxy that receives, processes, and exports telemetry across distributed infrastructure. To prevent memory exhaustion and backpressure drops, the collector must be configured with a memory limiter processor, a batch processor, and persistent retry queues:
# Production OpenTelemetry Collector Configuration
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
# Memory Limiter: Drops or throttles data before process memory hits pod limits
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
# Batch Processor: Groups spans and metrics to reduce downstream HTTP connection overhead
batch:
send_batch_size: 8192
timeout: 10s
send_batch_max_size: 10240
# Resource Processor: Normalizes cloud and Kubernetes metadata attributes
resourcedetection:
detectors: [env, gcp, ecs, ec2]
timeout: 2s
exporters:
# Prometheus Remote Write Exporter for Metrics
prometheusremotewrite:
endpoint: "http://thanos-receive.monitoring.svc.cluster.local:19291/api/v1/receive"
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 5m
# OTLP gRPC Exporter for Distributed Tracing (Tempo)
otlp/tempo:
endpoint: "tempo-distributor.monitoring.svc.cluster.local:4317"
tls:
insecure: true
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite]
telemetry:
metrics:
address: 0.0.0.0:8888
This pipeline configuration guarantees that if Thanos or Tempo slows down during a massive traffic surge, the memory limiter prevents the OpenTelemetry collector daemon from exceeding its Linux cgroup memory quota, preserving uninterrupted telemetry flow across the platform.
11. Production SRE Diagnostic Toolkit & CLI Commands
When troubleshooting live observability infrastructure, execute these standardized SRE commands:
# 1. Inspect active Alertmanager cluster mesh and peer states
curl -s http://localhost:9093/api/v2/status | jq '.cluster'
# 2. Query all currently firing alerts across the cluster
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels'
# 3. Verify Prometheus rule evaluation health and parse errors
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.health != "ok")'
# 4. Check Prometheus WAL disk write duration and head chunks
curl -s http://localhost:9090/metrics | grep "prometheus_tsdb_wal_fsync_duration_seconds"
# 5. Test Alertmanager routing resolution for a synthetic alert
amtool config routes test --config.file=/etc/alertmanager/config.yml \
--labels="alertname=OrderServiceLatency,team=platform,severity=critical"
12. Frequently Asked Questions
What causes Prometheus to enter an OOM crash loop during startup?
During startup, Prometheus replays its Write-Ahead Log (WAL) to restore unwritten in-memory metrics. If high-cardinality churn bloated the WAL before a crash, replaying it can exceed pod memory limits, causing repeated OOM terminations. Temporarily increasing memory limits or dropping high-cardinality labels resolves the loop.
Why is a Watchdog / Dead Man's Snitch alert mandatory in production?
If the monitoring system itself crashes or network connectivity is completely severed, it cannot send an alert announcing its own failure. A Watchdog alert fires constantly to an external third-party service; if the heartbeat stops arriving, the external service pages the on-call team directly.
What is high cardinality in metrics and why is it dangerous?
High cardinality occurs when metric labels have hundreds of thousands of unique values (like user IDs, IP addresses, or order IDs). Each unique combination creates a distinct time series in Prometheus RAM, leading to massive memory consumption, slow queries, and cluster crashes.
13. Verifiable Sources & Further Reading
- Prometheus Alertmanager Documentation — Official routing tree, inhibition, and receiver specifications.
- Google SRE Workbook: Alerting on SLOs — Mathematical foundation of multi-window multi-burn-rate alerting.
- OpenTelemetry Tracing Specification — W3C TraceContext standards and context propagation guidelines.
- Grafana Loki Architecture & Best Practices — Stream labeling, chunk storage, and log backpressure management.