# How to Monitor PentAGI Performance: OpenTelemetry Setup and Grafana Dashboards

> Easily monitor PentAGI performance using its built-in OpenTelemetry stack. Visualize metrics, logs, and traces with pre-configured Grafana dashboards for real-time insights.

- Repository: [VXControl/pentagi](https://github.com/vxcontrol/pentagi)
- Tags: how-to-guide
- Published: 2026-03-21

---

**PentAGI ships with a built-in OpenTelemetry observability stack that exports metrics, logs, and traces to any OTLP-compatible backend, complete with pre-configured Grafana dashboards for real-time performance visualization.**

The vxcontrol/pentagi repository includes a comprehensive observability layer located in `backend/pkg/observability` that automatically instruments the application for production monitoring. This guide explains how to enable and configure the OpenTelemetry pipeline to effectively monitor PentAGI performance, from process-level resource consumption to custom business metrics.

## PentAGI Observability Architecture

### The Observer Singleton

In [`backend/pkg/observability/obs.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/obs.go), the `Observer` struct serves as a global singleton that coordinates tracing, logging, metrics, and optional Langfuse hooks. The `InitObserver` function (lines 78-86) initializes this singleton, which is automatically invoked via the `init()` function (lines 44-48) when the package loads. This design ensures that every component of PentAGI shares a unified telemetry pipeline without manual dependency injection.

### OTLP Telemetry Client

The actual export mechanism resides in [`backend/pkg/observability/otelclient.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/otelclient.go). The `NewTelemetryClient` function (lines 90-108) constructs an OTLP gRPC client that batches and transmits telemetry data to your designated collector. This client handles log records, metric aggregations, and trace spans through a single efficient pipeline, converting internal PentAGI events into standardized OpenTelemetry signals.

### Built-in Metric Collectors

Process and runtime instrumentation is implemented in [`backend/pkg/observability/collector.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/collector.go). The `StartProcessMetricCollect` and `StartGoRuntimeMetricCollect` functions (lines 19-84) register observable gauges that sample:
- `process_resident_memory_bytes`
- `process_virtual_memory_bytes`
- `process_cpu_usage_percent`
- `go_goroutines`
- `go_heap_objects_bytes`

These gauges invoke underlying OS and Go runtime APIs (lines 24-71) to capture resource utilization without requiring manual instrumentation of your service code.

### Server Bootstrap

The observability stack activates in [`backend/cmd/pentagi/main.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/cmd/pentagi/main.go) (lines 62-71), where the server entry point explicitly calls `obs.InitObserver` followed by the metric collection starters. This bootstrap sequence ensures the telemetry pipeline is live and exporting before the application begins processing requests.

## Configuring OpenTelemetry Monitoring

Enabling PentAGI performance monitoring requires minimal configuration:

1. **Set the OTLP endpoint** via environment variable:

   ```dotenv
   OTEL_HOST=otel-collector:4317
   ```

2. **Start the binary** — the observer initializes automatically due to the `init()` hook in [`obs.go`](https://github.com/vxcontrol/pentagi/blob/main/obs.go).

3. **Import Grafana dashboards** from [`observability/grafana/dashboards/components/pentagi_service.json`](https://github.com/vxcontrol/pentagi/blob/main/observability/grafana/dashboards/components/pentagi_service.json) (for application metrics) and [`victoriametrics.json`](https://github.com/vxcontrol/pentagi/blob/main/victoriametrics.json) (for storage metrics) into your Grafana instance via Dashboards > Import.

## Key Performance Metrics

Once configured, the following metrics are automatically exported via the OTLP pipeline defined in [`collector.go`](https://github.com/vxcontrol/pentagi/blob/main/collector.go):
- **Process metrics**: Resident memory, virtual memory, and CPU utilization percentages sampled from `/proc` filesystem data.
- **Go runtime metrics**: Active goroutines, heap allocations, GC pause times, and scheduler statistics exposed by the Go standard library.

These dimensions provide complete visibility into PentAGI's resource footprint and runtime health, with data updating continuously through the observable gauge pattern.

## Instrumenting Custom Metrics

For application-specific telemetry, use the `Observer`'s factory methods exposed in [`backend/pkg/observability/obs.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/obs.go). The singleton provides methods like `NewInt64Counter` for creating custom instruments that flow through the same OTLP exporter as system metrics:

```go
import (
    obs "pentagi/pkg/observability"
    "go.opentelemetry.io/otel/attribute"
    otelmetric "go.opentelemetry.io/otel/metric"
)

func init() {
    counter, err := obs.Observer.NewInt64Counter(
        "pentagi_flows_completed",
    )
    if err != nil {
        log.Printf("failed to create counter: %v", err)
        return
    }
    
    // Increment after a flow finishes
    counter.Add(context.Background(), 1,
        otelmetric.WithAttributes(attribute.String("component", "flow")))
}

```

Custom counters created this way inherit the same batching, retry, and export configuration as the built-in system collectors.

## Summary

- PentAGI performance monitoring is built on OpenTelemetry standards with OTLP exporters implemented in [`backend/pkg/observability/otelclient.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/otelclient.go).
- The `Observer` singleton in [`obs.go`](https://github.com/vxcontrol/pentagi/blob/main/obs.go) coordinates all telemetry signals including optional Langfuse LLM tracing.
- Enable monitoring by setting the `OTEL_HOST` environment variable and deploying pre-built dashboards from `observability/grafana/dashboards/`.
- Built-in collectors in [`collector.go`](https://github.com/vxcontrol/pentagi/blob/main/collector.go) capture process and Go runtime metrics without requiring code changes.
- Extend monitoring using `NewInt64Counter` and other metric factory methods for business-specific KPIs.

## Frequently Asked Questions

### What backend systems does PentAGI support for performance monitoring?

PentAGI exports standard OTLP (OpenTelemetry Protocol) data via gRPC, making it compatible with Grafana Tempo, Prometheus, VictoriaMetrics, Loki, and any other OTLP-compatible observability backend. The `OTEL_HOST` environment variable should point to your OpenTelemetry Collector or direct ingestion endpoint.

### Where are the Grafana dashboards located in the repository?

Pre-configured dashboards reside in `observability/grafana/dashboards/components/`, including [`pentagi_service.json`](https://github.com/vxcontrol/pentagi/blob/main/pentagi_service.json) for application-specific metrics and [`victoriametrics.json`](https://github.com/vxcontrol/pentagi/blob/main/victoriametrics.json) for storage-layer visualization. Import these JSON files via Grafana's Dashboards > Import interface to immediately visualize PentAGI health.

### Does PentAGI support distributed tracing for LLM operations?

Yes. The observability stack includes optional Langfuse integration implemented in [`backend/pkg/observability/langfuse/observer.go`](https://github.com/vxcontrol/pentagi/blob/main/backend/pkg/observability/langfuse/observer.go). When configured with Langfuse credentials, this captures detailed traces of LLM-generated events alongside standard performance metrics, providing full request visibility through the unified `Observer` interface.

### How do I verify that metrics are being collected correctly?

After starting the server with `OTEL_HOST` configured, check the "PentAGI Service" dashboard in Grafana for active data points on `process_cpu_usage_percent` and `go_goroutines`. The [`collector.go`](https://github.com/vxcontrol/pentagi/blob/main/collector.go) implementation uses observable gauges that update automatically every collection interval, so missing data indicates a connection issue with your OTLP endpoint rather than a collection failure.