Pathway Monitoring and Observability Tools: OpenTelemetry, Prometheus, and Grafana Integration

Pathway natively supports OpenTelemetry (OTLP), Prometheus metrics endpoints, and Grafana dashboards out-of-the-box, enabling comprehensive pipeline observability with zero additional instrumentation.

The pathwaycom/pathway repository provides built-in monitoring and observability tools compatible with Pathway that allow you to capture traces, metrics, and logs from your data pipelines. By leveraging standard protocols like OpenTelemetry and Prometheus, Pathway eliminates the need for custom instrumentation while ensuring compatibility with the most popular open-source and commercial observability platforms.

Core Observability Protocols in Pathway

Pathway implements two primary telemetry protocols that serve as the foundation for all monitoring integrations.

OpenTelemetry (OTLP) Support

The framework emits traces, metrics, and logs via the OpenTelemetry Protocol (OTLP). In src/engine/telemetry.rs, the Telemetry struct initializes an OTLP exporter that pushes data to a configurable collector endpoint. This implementation supports both gRPC and HTTP transport, with gRPC being the default for high-throughput scenarios.

The OTLP integration captures:

• Pipeline execution traces showing data flow through operators
• System metrics including CPU and memory utilization
• Structured logs from the Rust engine and Python API

Prometheus Metrics Endpoint

For environments preferring pull-based monitoring, Pathway exposes a Prometheus-compatible /metrics endpoint implemented in src/engine/http_server.rs. This lightweight HTTP server uses the prometheus_client crate to serve metrics in OpenMetrics text format.

By default, the endpoint listens on port 8001 (configurable via PATHWAY_MONITORING_HTTP_PORT), making it compatible with existing Prometheus scrape configurations without requiring additional exporters.

Supported Monitoring Backends and Tools

Pathway's protocol-native approach ensures compatibility with a wide ecosystem of observability platforms.

Grafana Stack (Loki, Tempo, Prometheus)

The repository includes complete configuration for the Grafana observability stack in examples/projects/monitoring/. This integration provides:

• Grafana Loki for log aggregation via OTLP forwarding
• Grafana Tempo for distributed trace storage and querying
• Prometheus for time-series metrics storage
• Pre-built dashboards (grafana-dashboard.json) visualizing pipeline health, operator performance, and system resource usage

The example docker-compose.yaml orchestrates the entire stack, including an OpenTelemetry collector configured to route data to each backend.

Third-Party OTLP-Compatible Platforms

Because Pathway uses standard OTLP, you can forward telemetry to any platform supporting the protocol without code changes:

• Datadog via OTLP ingestion
• New Relic OpenTelemetry integration
• Splunk Observability (formerly SignalFx)
• Lightstep (now part of ServiceNow)
• AWS X-Ray via the OpenTelemetry collector

Simply configure the collector's exporter to point to your platform's OTLP endpoint using the appropriate authentication headers.

Implementation Guide: Configuring Pathway Monitoring

Enabling OpenTelemetry from Python

Configure telemetry emission directly in your Pathway application using set_monitoring_config:

import pathway as pw

# Send metrics, traces and logs to an OpenTelemetry collector running on localhost:4317

pw.set_monitoring_config(
    server_endpoint="http://localhost:4317",   # OTLP gRPC endpoint

    detailed_metrics_dir="./metrics",          # optional local folder for CSV metrics

    metrics_reader_interval_secs=5            # how often to push metrics

)

# Example pipeline (replace with your own logic)

@pw.udf
def double(x: int) -> int:
    return x * 2

source = pw.io.kafka.read(topic="input", bootstrap_servers="localhost:9092")
result = source.select(x=double(pw.this.value))
pw.run(source=result, monitoring_level=pw.MonitoringLevel.ALL)

This configuration is exposed via python_api.rs, which bridges the Python interface to the Rust telemetry implementation.

Scraping Prometheus Metrics

For pull-based monitoring, start your pipeline and scrape the metrics endpoint:

# Default port is 8001, configurable via PATHWAY_MONITORING_HTTP_PORT

curl http://localhost:8001/metrics

The output follows the OpenMetrics text format and includes counters for rows processed, operator latency histograms, and system resource gauges. This endpoint is implemented in src/engine/http_server.rs using the metrics_from_stats function to translate internal statistics into Prometheus format.

Deploying the OpenTelemetry Collector

Use the provided Docker Compose configuration to deploy a complete observability stack:

# docker-compose.yaml excerpt

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol-contrib/config.yaml"]
    volumes:
      - ./config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC

The collector configuration (config.yaml) routes telemetry to multiple backends simultaneously:

# config.yaml excerpt (collector)

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheusremotewrite:
    endpoint: ${PROMETHEUS_URL}
    auth:
      authenticator: basicauth/grafana_cloud_prometheus
  loki:
    endpoint: ${LOKI_URL}
    auth:
      authenticator: basicauth/grafana_cloud_loki
  otlp/tempo:
    endpoint: ${TEMPO_URL}
    auth:
      authenticator: basicauth/grafana_cloud_tempo

Pre-Built Dashboards and Configuration

The repository includes production-ready monitoring assets in examples/projects/monitoring/:

• grafana-dashboard.json: Visualizes pipeline throughput, operator performance, memory usage, and error rates
• docker-compose.yaml: Complete stack including Pathway, collector, Loki, Tempo, and Prometheus
• config.yaml: Collector routing configuration for Grafana Cloud

Import the dashboard into Grafana to immediately gain insights into:

• System-level metrics (CPU, memory, GC pressure)
• Pipeline operator counters (rows processed, latency percentiles)
• Distributed traces showing data flow through the computation graph

Key Source Files and Architecture

Understanding the implementation helps troubleshoot integration issues:

File	Purpose	Location
telemetry.rs	Initializes OTLP exporters and meter providers	src/engine/telemetry.rs
exporter.rs	Implements OTLP export logic for traces, metrics, logs	src/engine/telemetry/exporter.rs
http_server.rs	Serves Prometheus-compatible metrics on /metrics	src/engine/http_server.rs
python_api.rs	Exposes set_monitoring_config to Python applications	src/python_api.rs
50.pathway-monitoring.md	Official deployment guide for monitoring setup	docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md

The architecture follows a dual-path approach: push via OTLP to collectors, and pull via Prometheus endpoints, ensuring compatibility with existing infrastructure regardless of whether you use cloud-native or traditional monitoring stacks.

Summary

• Pathway provides native OpenTelemetry (OTLP) support for traces, metrics, and logs via src/engine/telemetry.rs, enabling push-based observability to any OTLP-compatible backend.
• A Prometheus-compatible /metrics endpoint exposed by src/engine/http_server.rs allows pull-based monitoring without additional exporters.
• The framework integrates seamlessly with the Grafana stack (Loki, Tempo, Prometheus) through provided Docker Compose configurations and pre-built dashboards in examples/projects/monitoring/.
• Any third-party platform supporting OTLP (Datadog, New Relic, Splunk, Lightstep) can receive Pathway telemetry by configuring the OpenTelemetry collector exporter.
• Configuration requires only a single Python call to pw.set_monitoring_config() or environment variable setup for the Prometheus endpoint.

Frequently Asked Questions

How do I enable monitoring in a Pathway pipeline without modifying code?

You can enable the Prometheus metrics endpoint by setting the environment variable PATHWAY_MONITORING_HTTP_PORT (defaults to 8001) before starting your pipeline. This exposes the /metrics endpoint without requiring any code changes. For full OpenTelemetry support including traces and logs, you must use pw.set_monitoring_config() in your Python code to specify the collector endpoint.

Can I use Pathway with Datadog or New Relic instead of Grafana?

Yes. Because Pathway uses the standard OpenTelemetry Protocol (OTLP), you can route telemetry to any OTLP-compatible platform. Configure the OpenTelemetry collector's exporter to point to Datadog's OTLP intake endpoint (e.g., https://http-intake.logs.datadoghq.com) or New Relic's OTLP endpoint using the appropriate API keys. The collector configuration in examples/projects/monitoring/config.yaml demonstrates the pattern for routing to multiple backends.

What metrics does Pathway expose by default?

Pathway exposes system-level metrics (CPU usage, memory consumption, garbage collection pressure) and pipeline-specific metrics including row processing counters per operator, latency histograms for data transformations, and error rates. When using the Prometheus endpoint (/metrics), these appear in OpenMetrics text format. Through OTLP, you additionally receive distributed traces showing the data flow graph through your pipeline operators.

Where can I find the complete monitoring stack configuration?

The repository provides a production-ready monitoring stack in examples/projects/monitoring/. This directory contains a docker-compose.yaml that orchestrates Pathway, the OpenTelemetry collector, Grafana Loki (logs), Grafana Tempo (traces), and Prometheus (metrics). It also includes config.yaml for collector routing rules and grafana-dashboard.json for visualizing pipeline health. The official deployment guide at docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md provides step-by-step setup instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}