# Pathway Monitoring and Observability Tools: OpenTelemetry, Prometheus, and Grafana Integration

> Explore Pathway's seamless integration with OpenTelemetry Prometheus and Grafana for powerful pipeline observability with zero extra instrumentation. Gain deep insights now.

- Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway)
- Tags: how-to-guide
- Published: 2026-03-06

---

**Pathway natively supports OpenTelemetry (OTLP), Prometheus metrics endpoints, and Grafana dashboards out-of-the-box, enabling comprehensive pipeline observability with zero additional instrumentation.**

The `pathwaycom/pathway` repository provides built-in monitoring and observability tools compatible with Pathway that allow you to capture traces, metrics, and logs from your data pipelines. By leveraging standard protocols like OpenTelemetry and Prometheus, Pathway eliminates the need for custom instrumentation while ensuring compatibility with the most popular open-source and commercial observability platforms.

## Core Observability Protocols in Pathway

Pathway implements two primary telemetry protocols that serve as the foundation for all monitoring integrations.

### OpenTelemetry (OTLP) Support

The framework emits **traces, metrics, and logs** via the OpenTelemetry Protocol (OTLP). In [`src/engine/telemetry.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/telemetry.rs), the `Telemetry` struct initializes an OTLP exporter that pushes data to a configurable collector endpoint. This implementation supports both gRPC and HTTP transport, with gRPC being the default for high-throughput scenarios.

The OTLP integration captures:
- Pipeline execution traces showing data flow through operators
- System metrics including CPU and memory utilization
- Structured logs from the Rust engine and Python API

### Prometheus Metrics Endpoint

For environments preferring pull-based monitoring, Pathway exposes a **Prometheus-compatible `/metrics` endpoint** implemented in [`src/engine/http_server.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/http_server.rs). This lightweight HTTP server uses the `prometheus_client` crate to serve metrics in OpenMetrics text format.

By default, the endpoint listens on port `8001` (configurable via `PATHWAY_MONITORING_HTTP_PORT`), making it compatible with existing Prometheus scrape configurations without requiring additional exporters.

## Supported Monitoring Backends and Tools

Pathway's protocol-native approach ensures compatibility with a wide ecosystem of observability platforms.

### Grafana Stack (Loki, Tempo, Prometheus)

The repository includes complete configuration for the **Grafana observability stack** in `examples/projects/monitoring/`. This integration provides:

- **Grafana Loki** for log aggregation via OTLP forwarding
- **Grafana Tempo** for distributed trace storage and querying
- **Prometheus** for time-series metrics storage
- **Pre-built dashboards** ([`grafana-dashboard.json`](https://github.com/pathwaycom/pathway/blob/main/grafana-dashboard.json)) visualizing pipeline health, operator performance, and system resource usage

The example [`docker-compose.yaml`](https://github.com/pathwaycom/pathway/blob/main/docker-compose.yaml) orchestrates the entire stack, including an OpenTelemetry collector configured to route data to each backend.

### Third-Party OTLP-Compatible Platforms

Because Pathway uses standard OTLP, you can forward telemetry to any platform supporting the protocol without code changes:

- **Datadog** via OTLP ingestion
- **New Relic** OpenTelemetry integration
- **Splunk Observability** (formerly SignalFx)
- **Lightstep** (now part of ServiceNow)
- **AWS X-Ray** via the OpenTelemetry collector

Simply configure the collector's exporter to point to your platform's OTLP endpoint using the appropriate authentication headers.

## Implementation Guide: Configuring Pathway Monitoring

### Enabling OpenTelemetry from Python

Configure telemetry emission directly in your Pathway application using `set_monitoring_config`:

```python
import pathway as pw

# Send metrics, traces and logs to an OpenTelemetry collector running on localhost:4317

pw.set_monitoring_config(
    server_endpoint="http://localhost:4317",   # OTLP gRPC endpoint

    detailed_metrics_dir="./metrics",          # optional local folder for CSV metrics

    metrics_reader_interval_secs=5            # how often to push metrics

)

# Example pipeline (replace with your own logic)

@pw.udf
def double(x: int) -> int:
    return x * 2

source = pw.io.kafka.read(topic="input", bootstrap_servers="localhost:9092")
result = source.select(x=double(pw.this.value))
pw.run(source=result, monitoring_level=pw.MonitoringLevel.ALL)

```

This configuration is exposed via [`python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/python_api.rs), which bridges the Python interface to the Rust telemetry implementation.

### Scraping Prometheus Metrics

For pull-based monitoring, start your pipeline and scrape the metrics endpoint:

```bash

# Default port is 8001, configurable via PATHWAY_MONITORING_HTTP_PORT

curl http://localhost:8001/metrics

```

The output follows the OpenMetrics text format and includes counters for rows processed, operator latency histograms, and system resource gauges. This endpoint is implemented in [`src/engine/http_server.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/http_server.rs) using the `metrics_from_stats` function to translate internal statistics into Prometheus format.

### Deploying the OpenTelemetry Collector

Use the provided Docker Compose configuration to deploy a complete observability stack:

```yaml

# docker-compose.yaml excerpt

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol-contrib/config.yaml"]
    volumes:
      - ./config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC

```

The collector configuration ([`config.yaml`](https://github.com/pathwaycom/pathway/blob/main/config.yaml)) routes telemetry to multiple backends simultaneously:

```yaml

# config.yaml excerpt (collector)

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  prometheusremotewrite:
    endpoint: ${PROMETHEUS_URL}
    auth:
      authenticator: basicauth/grafana_cloud_prometheus
  loki:
    endpoint: ${LOKI_URL}
    auth:
      authenticator: basicauth/grafana_cloud_loki
  otlp/tempo:
    endpoint: ${TEMPO_URL}
    auth:
      authenticator: basicauth/grafana_cloud_tempo

```

## Pre-Built Dashboards and Configuration

The repository includes production-ready monitoring assets in `examples/projects/monitoring/`:

- **grafana-dashboard.json**: Visualizes pipeline throughput, operator performance, memory usage, and error rates
- **docker-compose.yaml**: Complete stack including Pathway, collector, Loki, Tempo, and Prometheus
- **config.yaml**: Collector routing configuration for Grafana Cloud

Import the dashboard into Grafana to immediately gain insights into:
- System-level metrics (CPU, memory, GC pressure)
- Pipeline operator counters (rows processed, latency percentiles)
- Distributed traces showing data flow through the computation graph

## Key Source Files and Architecture

Understanding the implementation helps troubleshoot integration issues:

| File | Purpose | Location |
|------|---------|----------|
| [`telemetry.rs`](https://github.com/pathwaycom/pathway/blob/main/telemetry.rs) | Initializes OTLP exporters and meter providers | [`src/engine/telemetry.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/telemetry.rs) |
| [`exporter.rs`](https://github.com/pathwaycom/pathway/blob/main/exporter.rs) | Implements OTLP export logic for traces, metrics, logs | [`src/engine/telemetry/exporter.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/telemetry/exporter.rs) |
| [`http_server.rs`](https://github.com/pathwaycom/pathway/blob/main/http_server.rs) | Serves Prometheus-compatible metrics on `/metrics` | [`src/engine/http_server.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/http_server.rs) |
| [`python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/python_api.rs) | Exposes `set_monitoring_config` to Python applications | [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) |
| [`50.pathway-monitoring.md`](https://github.com/pathwaycom/pathway/blob/main/50.pathway-monitoring.md) | Official deployment guide for monitoring setup | [`docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md`](https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md) |

The architecture follows a dual-path approach: **push** via OTLP to collectors, and **pull** via Prometheus endpoints, ensuring compatibility with existing infrastructure regardless of whether you use cloud-native or traditional monitoring stacks.

## Summary

- Pathway provides **native OpenTelemetry (OTLP)** support for traces, metrics, and logs via [`src/engine/telemetry.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/telemetry.rs), enabling push-based observability to any OTLP-compatible backend.
- A **Prometheus-compatible `/metrics` endpoint** exposed by [`src/engine/http_server.rs`](https://github.com/pathwaycom/pathway/blob/main/src/engine/http_server.rs) allows pull-based monitoring without additional exporters.
- The framework integrates seamlessly with the **Grafana stack** (Loki, Tempo, Prometheus) through provided Docker Compose configurations and pre-built dashboards in `examples/projects/monitoring/`.
- Any third-party platform supporting OTLP (Datadog, New Relic, Splunk, Lightstep) can receive Pathway telemetry by configuring the OpenTelemetry collector exporter.
- Configuration requires only a single Python call to `pw.set_monitoring_config()` or environment variable setup for the Prometheus endpoint.

## Frequently Asked Questions

### How do I enable monitoring in a Pathway pipeline without modifying code?

You can enable the Prometheus metrics endpoint by setting the environment variable `PATHWAY_MONITORING_HTTP_PORT` (defaults to 8001) before starting your pipeline. This exposes the `/metrics` endpoint without requiring any code changes. For full OpenTelemetry support including traces and logs, you must use `pw.set_monitoring_config()` in your Python code to specify the collector endpoint.

### Can I use Pathway with Datadog or New Relic instead of Grafana?

Yes. Because Pathway uses the standard OpenTelemetry Protocol (OTLP), you can route telemetry to any OTLP-compatible platform. Configure the OpenTelemetry collector's exporter to point to Datadog's OTLP intake endpoint (e.g., `https://http-intake.logs.datadoghq.com`) or New Relic's OTLP endpoint using the appropriate API keys. The collector configuration in [`examples/projects/monitoring/config.yaml`](https://github.com/pathwaycom/pathway/blob/main/examples/projects/monitoring/config.yaml) demonstrates the pattern for routing to multiple backends.

### What metrics does Pathway expose by default?

Pathway exposes system-level metrics (CPU usage, memory consumption, garbage collection pressure) and pipeline-specific metrics including row processing counters per operator, latency histograms for data transformations, and error rates. When using the Prometheus endpoint (`/metrics`), these appear in OpenMetrics text format. Through OTLP, you additionally receive distributed traces showing the data flow graph through your pipeline operators.

### Where can I find the complete monitoring stack configuration?

The repository provides a production-ready monitoring stack in `examples/projects/monitoring/`. This directory contains a [`docker-compose.yaml`](https://github.com/pathwaycom/pathway/blob/main/docker-compose.yaml) that orchestrates Pathway, the OpenTelemetry collector, Grafana Loki (logs), Grafana Tempo (traces), and Prometheus (metrics). It also includes [`config.yaml`](https://github.com/pathwaycom/pathway/blob/main/config.yaml) for collector routing rules and [`grafana-dashboard.json`](https://github.com/pathwaycom/pathway/blob/main/grafana-dashboard.json) for visualizing pipeline health. The official deployment guide at [`docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md`](https://github.com/pathwaycom/pathway/blob/main/docs/2.developers/4.user-guide/60.deployment/50.pathway-monitoring.md) provides step-by-step setup instructions.