A prevalent misconception in modern cloud engineering is that platform teams must choose between OpenTelemetry and Prometheus. In high-scale production systems, they are not mutually exclusive competitors; they operate at complementary layers of the telemetry lifecycle.

OpenTelemetry (OTel) is a standardized specification, SDK ecosystem, and collector pipeline designed for generating, processing, and routing multi-signal telemetry (metrics, distributed traces, and logs). Prometheus is a specialized time-series database (TSDB), scraping engine, and querying ecosystem (PromQL) optimized for storing, indexing, and evaluating alerts against numerical metrics.


1. Architectural Comparison Matrix

Capability OpenTelemetry (OTel) Prometheus
Primary Function Telemetry generation, SDK instrumentation, pipeline routing Time-series metric storage, scraping engine, PromQL alerting
Data Signals Multi-signal: Traces, Metrics, Logs, Baggage Metrics only (Time-series floats)
Transmission Model Push by default (OTLP over gRPC or HTTP/protobuf) Pull by default (Scrapes HTTP /metrics endpoints)
Storage Backend None built-in (routes to Prometheus, ClickHouse, Datadog) Built-in optimized local TSDB / Remote Write support
Query Language No query engine · Relies on backend destinations PromQL (The industry standard for metric aggregation)
Alerting Engine No alerting engine · Outsourced to downstream systems Integrated Alertmanager with routing and silencing

2. Push vs. Pull: The Operational Trade-Off

The Prometheus Pull Advantage

Prometheus uses service discovery to automatically find application targets (e.g., Kubernetes Pods, EC2 instances) and pulls metrics at regular intervals. This provides two significant operational advantages:

  • Built-In Liveness Detection: If a target fails to respond to a scrape, Prometheus immediately sets the up{job="api"} metric to 0, triggering alerts without requiring the dying service to send a notification.
  • Backpressure Protection: The monitoring server controls ingestion rate. High traffic on an application cannot overwhelm the monitoring system with a flood of metric events.

Where Pull Breaks Down & Push Wins

The pull model struggles in serverless architectures, ephemeral batch jobs, and edge IoT devices. When an AWS Lambda function or Kubernetes CronJob executes in 400 milliseconds and terminates, a Prometheus scraper pulling every 15 seconds will miss the execution entirely. OpenTelemetry’s push-based OTLP architecture allows short-lived workloads to transmit metrics immediately before process exit.


3. The Unified Production Architecture (Pattern: OTel to Prometheus)

The prevailing enterprise pattern instruments applications with OpenTelemetry SDKs, transmits data via OTLP to a cluster-local OpenTelemetry Collector, and exports metrics to Prometheus or a Prometheus-compatible distributed TSDB (Thanos, Cortex, or VictoriaMetrics):

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape legacy infrastructure targets exposing /metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-nodes'
          scrape_interval: 15s
          static_configs:
            - targets: ['node-exporter.monitoring:9100']

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  # Export metrics to Prometheus via remote write
  prometheusremotewrite:
    endpoint: "http://prometheus-server.monitoring:9090/api/v1/write"
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

4. Metric Temporality: Cumulative vs. Delta Pitfalls

A critical engineering nuance when translating OTel metrics to Prometheus is temporality:

  • Cumulative Counters (Prometheus standard): A monotonically increasing value from process startup (e.g., total requests handled = 1,420). PromQL functions like rate() and increase() require cumulative counters to detect process restarts and calculate derivatives.
  • Delta Counters (StatsD / CloudWatch standard): Represents the change since the last report (e.g., +15 requests in the last second).

When configuring the OpenTelemetry Collector’s prometheusremotewrite exporter, always ensure counter temporality is configured as Cumulative. Pushing delta counters to Prometheus will cause rate() calculations to return incorrect flatlines or wild spikes.


5. Strategic Recommendations: How to Choose

  • For Application Instrumentation: Standardize on OpenTelemetry SDKs. Never hardcode proprietary vendor libraries or Prometheus client libraries into microservice source code. OTel protects you against future vendor lock-in.
  • For Telemetry Routing: Deploy the OpenTelemetry Collector as a Kubernetes DaemonSet or sidecar. Use it to scrub PII, aggregate batches, and fan out metrics to multiple storage backends.
  • For Metric Storage & Alerting: Maintain Prometheus or compatible TSDBs (Mimir/Thanos). PromQL remains the industry standard for alerting rules and Grafana dashboards.

6. Related Guides & Runbooks