DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD
INCIDENT RESOLUTION · 24 MIN READ
Production monitoring logo

Production Monitoring & Alerting Troubleshooting: SRE Triage Runbook

An in-depth SRE runbook for diagnosing and repairing production observability failures: resolving silent alert black holes, Prometheus TSDB ingestion bottlenecks, high-cardinality crashes, and log backpressure.

By DevOps Tribune Engineering Staff · Published November 17, 2026

In modern high-scale distributed systems, the monitoring and alerting pipeline is the vital nervous system of the infrastructure. When production services experience degradation, platform teams depend entirely on real-time metrics, distributed traces, and aggregated log streams to detect anomalies, quantify business impact, and identify root causes. However, the observability infrastructure itself is a complex, distributed, stateful software system—and it is equally vulnerable to catastrophic failure. There is no worse scenario for an engineering organization than discovering a major customer-facing outage not from an automated PagerDuty alert, but from an angry tweet or an escalated executive email because the monitoring pipeline silently failed hours earlier.

Production monitoring failures typically fall into two destructive extremes: silent alert failures, where critical thresholds breach but alerts are swallowed by misconfigured Alertmanager routing trees or dead man's snitch black holes; and alert fatigue, where thousands of noisy, non-actionable notifications bombard on-call engineers until real incidents are ignored. Behind these symptoms lie deep systems bottlenecks: Prometheus Time Series Database (TSDB) write stalls caused by high-cardinality label explosions, disk I/O starvation on Write-Ahead Logs (WAL), asynchronous distributed trace context stripping across thread boundaries, and log aggregation buffer drops under sudden traffic spikes.

This masterclass provides an exhaustive SRE troubleshooting runbook for debugging and hardening production monitoring and alerting systems. We dissect telemetry ingestion architectures, resolve Alertmanager routing failures, optimize high-cardinality TSDB memory footprints, implement Google SRE multi-window multi-burn-rate error budget alerting, and deliver verified production configurations to ensure your monitoring systems remain resilient under extreme operational pressure.


1. The Anatomy of Telemetry Failures: Metrics, Logs, and Traces

Observability relies on the three pillars of telemetry, each with distinct failure modes that require specific diagnostic approaches:

Telemetry Pillar Ingestion Mechanism Primary Failure Symptom Root System Bottleneck
Metrics (Prometheus / VictoriaMetrics) Pull-based HTTP scrapes or OTLP push Scrape gaps, delayed dashboards, OOM kills High-cardinality label explosion; Head chunk compaction I/O wait.
Alerts (Alertmanager / PagerDuty) Push webhooks from rule evaluation engines Silent failure during outages; notification storm Mismatched receiver matchers; missing Watchdog heartbeat; flapping thresholds.
Logs (Grafana Loki / Fluent Bit / Vector) Agent log tailing & HTTP push batches Missing incident logs, rate-limit 429 drops Buffer disk overflow; dynamic stream label cardinality; ingestion ring quorum loss.
Traces (OpenTelemetry / Jaeger / Tempo) In-process SDK instrumentation & OTLP collector Broken trace spans; missing root context W3C header propagation stripped by proxy; aggressive head-based sampling drops.

2. Triage Category 1: Silent Alert Failures & Alertmanager Routing Traps

A silent alert failure occurs when a Prometheus rule successfully enters the FIRING state, but zero notifications arrive in Slack, PagerDuty, or email. The failure resides inside the Alertmanager routing tree or notification dispatch pipeline.

1. The Subtree Early-Exit Trap

Alertmanager routes alerts hierarchically starting at the root route. By default, when an alert matches a child route, Alertmanager stops evaluation and does not evaluate subsequent sibling routes unless continue: true is explicitly configured:

# Alertmanager Production Routing Definition
route:
  receiver: 'slack-default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Trap: If an alert matches 'team: platform', it sends to Slack,
    # but NEVER reaches PagerDuty unless continue is set to true!
    - match:
        team: platform
      receiver: 'slack-platform-channel'
      continue: true  # Mandatory to allow critical alerts to reach PagerDuty

    - match:
        severity: critical
      receiver: 'pagerduty-high-urgency'

2. Dead Man's Snitch / Watchdog Heartbeat Monitoring

How do you know if your monitoring system is dead? If Prometheus stops running or Alertmanager crashes, it cannot send an alert saying it has crashed. Resilient SRE architectures mandate a Watchdog alert (Dead Man's Snitch):

  • Configure an alert rule that is permanently firing 24/7/365: vector(1) > 0 with alertname: Watchdog.
  • Route this alert to an external heartbeat monitoring service (such as Healthchecks.io, Dead Man's Snitch, or PagerDuty Heartbeats).
  • The external service expects a webhook ping every 60 seconds. If a ping fails to arrive within 2 minutes, the external third-party service directly pages the primary SRE on-call team, alerting them that the entire internal observability cluster is offline.

3. Triage Category 2: Prometheus TSDB Memory Explosions & High Cardinality

Prometheus stores active time series in RAM inside memory chunks before flushing them to immutable two-hour blocks on disk. The memory footprint of Prometheus is directly proportional to the number of active time series. High cardinality occurs when a developer inadvertently injects high-entropy labels into a metric:

# Catastrophic Anti-Pattern: High-Cardinality User ID and Order ID Labels
# If 100,000 users place orders, this creates 100,000 distinct time series in RAM!
http_requests_total{method="POST", path="/checkout", user_id="u_98472", order_id="ord_12894"} 1

Within minutes, the Prometheus memory usage spikes from 4 GB to 32 GB, triggering Linux kernel Out-Of-Memory (OOM) killer terminations (Exit Code 137). When Prometheus restarts, it must replay the Write-Ahead Log (WAL), which consumes massive memory, immediately triggering another OOM kill in a continuous crash loop.

Investigating TSDB High Cardinality via API

Query the Prometheus TSDB runtime status directly to identify the top offending metric names and label values:

# Query top 10 series with highest memory footprint
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[0:10]'

# Query labels with highest number of unique values
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName[0:10]'

Emergency Triage and Remediation

  1. Drop Labels at Scrape Time via metric_relabel_configs: Drop offending high-cardinality labels before they enter the TSDB:
    metric_relabel_configs:
      - regex: "(user_id|order_id|session_token)"
        action: labeldrop
  2. Recovering from WAL Replay OOM Crashes: If Prometheus is trapped in an OOM restart loop, temporarily allocate swap space or run Prometheus with --storage.tsdb.wal-compression and increase pod memory limits to allow the replay to finish cleanly. In extreme emergency disasters where historical data can be sacrificed to restore production alerting, rename or truncate the corrupted wal/ directory.

4. Triage Category 3: Distributed Tracing & Missing Context Propagation

In microservice architectures, an incoming user HTTP request may traverse an API gateway, authentication service, order service, payment gateway, and database. Distributed tracing connects these isolated operations into a single directed graph through trace context propagation.

The Broken Trace Problem

When engineers inspect a trace in Jaeger or Grafana Tempo and observe that child spans are disconnected or missing entirely, the cause is almost always context stripping:

  • Unpropagated HTTP Headers: Under the W3C TraceContext standard, distributed tracing relies on the traceparent header (e.g., traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01). If an intermediate reverse proxy (like an older NGINX ingress or API gateway) strips unrecognized headers, downstream services generate a brand-new trace ID, breaking the distributed trace graph.
  • Asynchronous Thread Handoffs: In languages like Java, Go, or Node.js, if a request handler spawns an asynchronous background goroutine, thread pool worker, or message queue publisher without passing the trace context.Context, the active span is detached from the background job.

Verification Procedure

Verify trace context headers directly using curl with the traceparent header against your gateway:

curl -v -H "traceparent: 00-11111111111111111111111111111111-2222222222222222-01" https://api.devopstribune.com/orders

Inspect backend container logs to ensure the trace ID 11111111111111111111111111111111 is preserved across all microservices.


5. Triage Category 4: Log Aggregation Backpressure & Buffer Overflows

Log forwarding agents (such as Fluent Bit, Vector, or Logstash) tail log files from container stdout/stderr. When an application encounters a sudden database outage, it often begins logging hundreds of thousands of exceptions per second. If the central log storage (e.g., Grafana Loki or Elasticsearch) experiences write latency, the log forwarding pipeline faces extreme backpressure.

Preventing Memory Exhaustion and Log Loss

If log agents buffer messages exclusively in memory, sudden ingestion spikes cause the log agent to crash with an OOM error, dropping all logs generated during the critical minutes of the outage. Production log forwarders must configure disk-backed buffering:

# Fluent Bit Production Disk Buffer Configuration
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*
    Mem_Buf_Limit     50MB       # Memory ceiling
    Buffer_Chunk_Size 512k
    Buffer_Max_Size   5M
    Storage.type      filesystem # Spill over to host disk when memory ceiling is reached

[OUTPUT]
    Name              loki
    Match             *
    Host              loki-gateway.monitoring.svc.cluster.local
    Port              3100
    Retry_Limit       10
    Workers           4

With filesystem storage enabled, if Loki throttles incoming requests with HTTP 429 Rate Limit Exceeded, Fluent Bit safely buffers logs to local disk storage on the worker node, draining the buffer smoothly once downstream storage recovers.


6. Multi-Window Multi-Burn-Rate SLO Alerting (Google SRE Standard)

Traditional alerting asserts simple static rules: "Page on-call if error rate > 2% for 5 minutes." This approach is severely flawed: a 5-minute transient spike causes an unnecessary wake-up page, while a slow 0.5% leak that drains 100% of the monthly error budget over 3 days never triggers an alert.

Google SRE standardized Multi-Window Multi-Burn-Rate Alerting. The burn rate represents how fast the service is consuming its allowable Service Level Objective (SLO) error budget:

  • Burn Rate 1: Consumes 100% of the 30-day budget in exactly 30 days.
  • Burn Rate 14.4: Consumes 100% of the budget in 2 days (2% of budget consumed per hour). Action: Page on-call immediately!
  • Burn Rate 6: Consumes 100% of the budget in 5 days (5% consumed in 6 hours). Action: Page during business hours.
  • Burn Rate 1: Consumes budget in 30 days. Action: Generate a ticket for sprint planning.

Production Multi-Burn-Rate PromQL Alert Rule

To avoid false positives, the alert requires two concurrent evaluation windows: a long window (to ensure sufficient budget consumption) and a short window (to verify the outage is actively continuing):

# Multi-Window Multi-Burn-Rate Alert Rule (14.4x Burn Rate on 99.9% SLO)
- alert: HighErrorBudgetBurnRate
  expr: |
    (
      # Short window: 5 minutes (verifies error is actively firing)
      sum(rate(http_requests_total{status=~"5.*"}[5m]))
      /
      sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
    )
    and
    (
      # Long window: 1 hour (verifies significant budget consumption)
      sum(rate(http_requests_total{status=~"5.*"}[1h]))
      /
      sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
    )
  for: 2m
  labels:
    severity: critical
    team: platform
  annotations:
    summary: "High error budget burn rate (14.4x) on API service"
    description: "Service has consumed more than 2% of its monthly error budget in the past hour."


7. Observability High Availability & Multi-Cluster Architecture (Thanos & Mimir)

A single Prometheus server constitutes a single point of failure (SPOF) and is fundamentally bounded by the memory and disk capacity of a single physical or virtual machine. When managing hundreds of Kubernetes clusters across multiple cloud regions, platform teams must evolve from isolated Prometheus instances to a federated, highly available architecture using Thanos or Grafana Mimir.

1. Thanos Architecture: Decentralized Long-Term Object Storage

Thanos runs alongside Prometheus as modular microservices, seamlessly extending Prometheus with global querying and infinite historical retention:

  • Thanos Sidecar: Deployed as a sidecar container in the Prometheus pod. It intercepts PromQL queries from the central querier and continuously ships immutable 2-hour TSDB blocks directly to Amazon S3 or Google Cloud Storage.
  • Thanos Store Gateway: Acts as an efficient read proxy over the historical metric blocks in object storage, caching index headers in memory to answer long-term PromQL range queries (e.g., year-over-year capacity forecasts) in sub-seconds.
  • Thanos Compactor: A singleton background worker that periodically merges historical 2-hour blocks into larger chunks and performs mathematical downsampling (aggregating raw 15-second data into 5-minute and 1-hour resolution blocks), slashing object storage retrieval costs by over 90%.
  • Thanos Querier: Exposes a standard Prometheus v1 API endpoint that deduplicates concurrent time series scraped by redundant Prometheus replicas (Replica A and Replica B), providing a single pane of glass for global multi-cluster dashboards.

8. Eradicating Alert Fatigue: Inhibition Rules & Flapping Suppression

Alert fatigue is the silent killer of engineering operational velocity. When an on-call engineer is awakened at 3:00 AM by 40 distinct PagerDuty notifications for the same underlying issue, cognitive overload occurs, leading to delayed incident response times and burnout. Production Alertmanager configurations eliminate alert noise through two critical mechanisms:

1. Alertmanager Inhibition Rules

Inhibition rules suppress lower-severity alerts when a higher-severity alert for the same system or failure domain is already actively firing. For example, if an entire Kubernetes node crashes (NodeDown), Alertmanager should not page engineers for 20 individual pod readiness probe failures running on that node:

# Production Alertmanager Inhibition Rule
inhibit_rules:
  # Suppress all pod-level alerts if the parent node is down
  - source_match:
      alertname: 'NodeDown'
    target_match_re:
      alertname: '(KubePodCrashLooping|KubePodNotReady|TargetMissing)'
    equal: ['node', 'cluster']

  # Suppress warning alerts if a critical alert is already firing for the same service
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'service', 'namespace']

2. Flapping Alert Suppression

An alert that fires, resolves, fires again, and resolves within minutes is considered flapping. Flapping alerts are typically caused by volatile metric evaluations hovering around a sharp threshold without hysteresis (e.g., CPU utilization bouncing between 79.8% and 80.2%). Resolve flapping by implementing a sustained evaluation window via the for: clause (e.g., for: 10m) and adding hysteresis: fire when metric > 85%, but only resolve when metric drops below 75% for 15 minutes.


9. Step-by-Step Observability Incident Triage Runbook

When the monitoring system itself fails during an active production outage, follow this structured emergency recovery protocol:

  1. Step 1: Verify Scrape Target Health: Navigate to http://prometheus:9090/targets or query the API: curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health != "up")'. Verify whether targets are failing due to DNS resolution, network timeouts, or authorization headers.
  2. Step 2: Inspect Prometheus Head Chunks & WAL: Check if disk I/O bottlenecks are stalling memory chunk compaction: iostat -xz 1 5. If %util is 100%, the underlying EBS volume or SSD is IOPS-throttled. Temporarily provision higher IOPS (e.g., AWS gp3 to io2).
  3. Step 3: Test Alert Dispatch Path: Use the amtool command-line utility to inject a synthetic test alert through Alertmanager to verify that external notification webhooks (Slack, PagerDuty, Opsgenie) are operational.
  4. Step 4: Check OpenTelemetry Collector Buffers: Inspect collector metrics: otelcol_exporter_enqueue_failed_spans and otelcol_exporter_send_failed_spans. If numbers are increasing, scale the collector horizontal replicas and increase memory batch buffers.

10. Complete Production OpenTelemetry Collector Pipeline Configuration

The OpenTelemetry Collector is the vendor-agnostic proxy that receives, processes, and exports telemetry across distributed infrastructure. To prevent memory exhaustion and backpressure drops, the collector must be configured with a memory limiter processor, a batch processor, and persistent retry queues:

# Production OpenTelemetry Collector Configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Memory Limiter: Drops or throttles data before process memory hits pod limits
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  # Batch Processor: Groups spans and metrics to reduce downstream HTTP connection overhead
  batch:
    send_batch_size: 8192
    timeout: 10s
    send_batch_max_size: 10240

  # Resource Processor: Normalizes cloud and Kubernetes metadata attributes
  resourcedetection:
    detectors: [env, gcp, ecs, ec2]
    timeout: 2s

exporters:
  # Prometheus Remote Write Exporter for Metrics
  prometheusremotewrite:
    endpoint: "http://thanos-receive.monitoring.svc.cluster.local:19291/api/v1/receive"
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m

  # OTLP gRPC Exporter for Distributed Tracing (Tempo)
  otlp/tempo:
    endpoint: "tempo-distributor.monitoring.svc.cluster.local:4317"
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
  telemetry:
    metrics:
      address: 0.0.0.0:8888

This pipeline configuration guarantees that if Thanos or Tempo slows down during a massive traffic surge, the memory limiter prevents the OpenTelemetry collector daemon from exceeding its Linux cgroup memory quota, preserving uninterrupted telemetry flow across the platform.

11. Production SRE Diagnostic Toolkit & CLI Commands

When troubleshooting live observability infrastructure, execute these standardized SRE commands:

# 1. Inspect active Alertmanager cluster mesh and peer states
curl -s http://localhost:9093/api/v2/status | jq '.cluster'

# 2. Query all currently firing alerts across the cluster
curl -s http://localhost:9093/api/v2/alerts | jq '.[].labels'

# 3. Verify Prometheus rule evaluation health and parse errors
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.health != "ok")'

# 4. Check Prometheus WAL disk write duration and head chunks
curl -s http://localhost:9090/metrics | grep "prometheus_tsdb_wal_fsync_duration_seconds"

# 5. Test Alertmanager routing resolution for a synthetic alert
amtool config routes test --config.file=/etc/alertmanager/config.yml \
  --labels="alertname=OrderServiceLatency,team=platform,severity=critical"

12. Frequently Asked Questions

What causes Prometheus to enter an OOM crash loop during startup?

During startup, Prometheus replays its Write-Ahead Log (WAL) to restore unwritten in-memory metrics. If high-cardinality churn bloated the WAL before a crash, replaying it can exceed pod memory limits, causing repeated OOM terminations. Temporarily increasing memory limits or dropping high-cardinality labels resolves the loop.

Why is a Watchdog / Dead Man's Snitch alert mandatory in production?

If the monitoring system itself crashes or network connectivity is completely severed, it cannot send an alert announcing its own failure. A Watchdog alert fires constantly to an external third-party service; if the heartbeat stops arriving, the external service pages the on-call team directly.

What is high cardinality in metrics and why is it dangerous?

High cardinality occurs when metric labels have hundreds of thousands of unique values (like user IDs, IP addresses, or order IDs). Each unique combination creates a distinct time series in Prometheus RAM, leading to massive memory consumption, slow queries, and cluster crashes.


13. Verifiable Sources & Further Reading