How to Monitor PentAGI Performance: OpenTelemetry Setup and Grafana Dashboards

PentAGI ships with a built-in OpenTelemetry observability stack that exports metrics, logs, and traces to any OTLP-compatible backend, complete with pre-configured Grafana dashboards for real-time performance visualization.

The vxcontrol/pentagi repository includes a comprehensive observability layer located in backend/pkg/observability that automatically instruments the application for production monitoring. This guide explains how to enable and configure the OpenTelemetry pipeline to effectively monitor PentAGI performance, from process-level resource consumption to custom business metrics.

PentAGI Observability Architecture

The Observer Singleton

In backend/pkg/observability/obs.go, the Observer struct serves as a global singleton that coordinates tracing, logging, metrics, and optional Langfuse hooks. The InitObserver function (lines 78-86) initializes this singleton, which is automatically invoked via the init() function (lines 44-48) when the package loads. This design ensures that every component of PentAGI shares a unified telemetry pipeline without manual dependency injection.

OTLP Telemetry Client

The actual export mechanism resides in backend/pkg/observability/otelclient.go. The NewTelemetryClient function (lines 90-108) constructs an OTLP gRPC client that batches and transmits telemetry data to your designated collector. This client handles log records, metric aggregations, and trace spans through a single efficient pipeline, converting internal PentAGI events into standardized OpenTelemetry signals.

Built-in Metric Collectors

Process and runtime instrumentation is implemented in backend/pkg/observability/collector.go. The StartProcessMetricCollect and StartGoRuntimeMetricCollect functions (lines 19-84) register observable gauges that sample:

• process_resident_memory_bytes
• process_virtual_memory_bytes
• process_cpu_usage_percent
• go_goroutines
• go_heap_objects_bytes

These gauges invoke underlying OS and Go runtime APIs (lines 24-71) to capture resource utilization without requiring manual instrumentation of your service code.

Server Bootstrap

The observability stack activates in backend/cmd/pentagi/main.go (lines 62-71), where the server entry point explicitly calls obs.InitObserver followed by the metric collection starters. This bootstrap sequence ensures the telemetry pipeline is live and exporting before the application begins processing requests.

Configuring OpenTelemetry Monitoring

Enabling PentAGI performance monitoring requires minimal configuration:

1. Set the OTLP endpoint via environment variable:

   OTEL_HOST=otel-collector:4317

2. Start the binary — the observer initializes automatically due to the init() hook in obs.go.

3. Import Grafana dashboards from observability/grafana/dashboards/components/pentagi_service.json (for application metrics) and victoriametrics.json (for storage metrics) into your Grafana instance via Dashboards > Import.

Key Performance Metrics

Once configured, the following metrics are automatically exported via the OTLP pipeline defined in collector.go:

• Process metrics: Resident memory, virtual memory, and CPU utilization percentages sampled from /proc filesystem data.
• Go runtime metrics: Active goroutines, heap allocations, GC pause times, and scheduler statistics exposed by the Go standard library.

These dimensions provide complete visibility into PentAGI's resource footprint and runtime health, with data updating continuously through the observable gauge pattern.

Instrumenting Custom Metrics

For application-specific telemetry, use the Observer's factory methods exposed in backend/pkg/observability/obs.go. The singleton provides methods like NewInt64Counter for creating custom instruments that flow through the same OTLP exporter as system metrics:

import (
    obs "pentagi/pkg/observability"
    "go.opentelemetry.io/otel/attribute"
    otelmetric "go.opentelemetry.io/otel/metric"
)

func init() {
    counter, err := obs.Observer.NewInt64Counter(
        "pentagi_flows_completed",
    )
    if err != nil {
        log.Printf("failed to create counter: %v", err)
        return
    }
    
    // Increment after a flow finishes
    counter.Add(context.Background(), 1,
        otelmetric.WithAttributes(attribute.String("component", "flow")))
}

Custom counters created this way inherit the same batching, retry, and export configuration as the built-in system collectors.

Summary

• PentAGI performance monitoring is built on OpenTelemetry standards with OTLP exporters implemented in backend/pkg/observability/otelclient.go.
• The Observer singleton in obs.go coordinates all telemetry signals including optional Langfuse LLM tracing.
• Enable monitoring by setting the OTEL_HOST environment variable and deploying pre-built dashboards from observability/grafana/dashboards/.
• Built-in collectors in collector.go capture process and Go runtime metrics without requiring code changes.
• Extend monitoring using NewInt64Counter and other metric factory methods for business-specific KPIs.

Frequently Asked Questions

What backend systems does PentAGI support for performance monitoring?

PentAGI exports standard OTLP (OpenTelemetry Protocol) data via gRPC, making it compatible with Grafana Tempo, Prometheus, VictoriaMetrics, Loki, and any other OTLP-compatible observability backend. The OTEL_HOST environment variable should point to your OpenTelemetry Collector or direct ingestion endpoint.

Where are the Grafana dashboards located in the repository?

Pre-configured dashboards reside in observability/grafana/dashboards/components/, including pentagi_service.json for application-specific metrics and victoriametrics.json for storage-layer visualization. Import these JSON files via Grafana's Dashboards > Import interface to immediately visualize PentAGI health.

Does PentAGI support distributed tracing for LLM operations?

Yes. The observability stack includes optional Langfuse integration implemented in backend/pkg/observability/langfuse/observer.go. When configured with Langfuse credentials, this captures detailed traces of LLM-generated events alongside standard performance metrics, providing full request visibility through the unified Observer interface.

How do I verify that metrics are being collected correctly?

After starting the server with OTEL_HOST configured, check the "PentAGI Service" dashboard in Grafana for active data points on process_cpu_usage_percent and go_goroutines. The collector.go implementation uses observable gauges that update automatically every collection interval, so missing data indicates a connection issue with your OTLP endpoint rather than a collection failure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}