# How to Monitor TencentDB Agent Memory Status Using OpenTelemetry and ClickHouse

> Learn how to monitor TencentDB Agent memory status in real time. Discover how OpenTelemetry and ClickHouse provide insights into performance metrics for better observability.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-08-25

---

**TencentDB Agent Memory exposes request-level telemetry through an OpenTelemetry pipeline that exports latency, session metadata, and error metrics to ClickHouse and Langfuse, allowing operators to observe knowledge retrieval performance in real time.**

The TencentCloud/TencentDB-Agent-Memory repository instruments its knowledge engine with built-in observability hooks. By configuring the telemetry initialiser and telemetry sinks, you can monitor memory agent status without deploying external APM agents, using only the existing `MemoryKnowledge` and `MemoryCore` modules.

## Core Telemetry Components

### OpenTelemetry Initialiser

The telemetry stack boots from [`MemoryKnowledge/src/telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/telemetry.ts). The `initTelemetry()` function instantiates an OpenTelemetry `NodeSDK`, registers a `TracerProvider`, and conditionally enables the Langfuse exporter based on the `LANGFUSE_SECRET_KEY` environment variable. This file also exports the `withSpan` helper used throughout the codebase to wrap asynchronous operations.

### ClickHouse Telemetry Sink

Persistent metrics are handled by [`MemoryKnowledge/src/clickhouse-telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/clickhouse-telemetry.ts). This module implements a custom exporter that buffers spans and writes one row per request into a ClickHouse table named `telemetry`. The schema includes columns for `event`, `session_key`, `turn_seq`, `duration_ms`, and `event_time`, enabling time-series analysis of agent interactions.

### Span Instrumentation

Individual components, such as the Wiki engine in [`MemoryKnowledge/src/engines/wiki/manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/manager.ts), instrument their handlers by calling `withSpan()`. This function creates a child span that inherits the trace context, automatically capturing start and end timestamps for every knowledge retrieval operation.

### Environment Configuration

Telemetry toggles and backend URLs are loaded via [`MemoryCore/src/utils/env.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/env.ts) and validated against the schema in [`MemoryCore/src/utils/env-config.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/env-config.ts). The `createLogger` utility defined in [`env.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/env.ts) reports high-level health information and initialization failures to stdout, providing immediate feedback during startup.

## Monitoring Workflow

1.  **Agent Startup** – When the Knowledge service starts via [`MemoryKnowledge/src/server.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/server.ts), it invokes `initTelemetry()` to configure the global tracer and optionally connect to ClickHouse.

2.  **Request Wrapping** – Each incoming request is wrapped with `withSpan()`, which records the session key, turn sequence, and operation type as span attributes.

3.  **LLM Instrumentation** – When the underlying LLM SDK is invoked with `experimental_telemetry: { isEnabled: true }`, it emits child spans that attach to the parent request span, capturing token usage and model latency.

4.  **Data Export** – At the end of the request lifecycle, the span is ended. The exporter flush logic (managed by [`MemoryCore/src/utils/managed-timer.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/managed-timer.ts)) writes the buffered data to ClickHouse and/or Langfuse based on your environment configuration.

## Enabling Telemetry in Your Deployment

Set the following environment variables before starting the agent:

```bash
export LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxx      # Optional; enables Langfuse tracing

export KNOWLEDGE_CLICKHOUSE_URL=http://localhost:8123
export KNOWLEDGE_CLICKHOUSE_ENABLED=true

```

The agent reads these values during initialization in [`MemoryCore/src/utils/env.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/env.ts). If `KNOWLEDGE_CLICKHOUSE_ENABLED` is unset or false, the ClickHouse exporter is skipped, but OpenTelemetry spans remain active for other collectors.

## Querying and Analyzing Metrics

### ClickHouse SQL Examples

To analyze recent latency trends, connect to your ClickHouse instance and query the `telemetry` table:

```sql
SELECT
  session_key,
  turn_seq,
  duration_ms,
  event,
  toDateTime(event_time) AS ts
FROM telemetry
WHERE event = 'usage'
ORDER BY ts DESC
LIMIT 100;

```

This returns the last 100 usage events with their execution duration, allowing you to identify slow sessions or spikes in processing time.

### Viewing Traces in Langfuse

When `LANGFUSE_SECRET_KEY` is configured, spans are forwarded to Langfuse automatically. Navigate to your Langfuse project dashboard (e.g., `https://app.langfuse.com` or a self-hosted instance) and filter by the service name `tencentdb-agent-memory`. The UI displays a waterfall view of each request, showing nested spans for database queries, LLM calls, and retrieval steps.

## Adding Custom Instrumentation

To capture domain-specific metrics, wrap new functions with `withSpan` and attach custom attributes:

```typescript
import { withSpan } from "./telemetry.js";

async function customRetrieval(query: string) {
  return await withSpan("retrieval.custom", async (span) => {
    span.setAttribute("query.length", query.length);
    span.setAttribute("retrieval.source", "vector_store");

    const results = await fetchResults(query);
    span.setAttribute("results.count", results.length);
    return results;
  });
}

```

These attributes appear as columns in ClickHouse (if mapped) or as metadata in Langfuse, enabling granular filtering by query characteristics.

## Summary

-   **Initialization** occurs in [`MemoryKnowledge/src/telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/telemetry.ts) via `initTelemetry()`, which configures the OpenTelemetry SDK and optional Langfuse exporter.
-   **Data Persistence** is handled by [`MemoryKnowledge/src/clickhouse-telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/clickhouse-telemetry.ts), writing structured rows to the `telemetry` table for long-term storage.
-   **Instrumentation** relies on the `withSpan()` helper to capture timing and context from [`MemoryKnowledge/src/engines/wiki/manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/manager.ts) and other modules.
-   **Configuration** is driven by environment variables defined in [`MemoryCore/src/utils/env.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/env.ts), including `KNOWLEDGE_CLICKHOUSE_ENABLED` and `LANGFUSE_SECRET_KEY`.
-   **Querying** can be performed via raw SQL against ClickHouse or through the Langfuse UI for distributed trace visualization.

## Frequently Asked Questions

### How do I enable ClickHouse telemetry without Langfuse?

Set `KNOWLEDGE_CLICKHOUSE_ENABLED=true` and provide a valid `KNOWLEDGE_CLICKHOUSE_URL`, but leave `LANGFUSE_SECRET_KEY` unset or empty. The initialiser in [`MemoryKnowledge/src/telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/telemetry.ts) checks for the secret key before registering the Langfuse exporter, so omitting it disables that sink while keeping the ClickHouse exporter active.

### What database schema does the ClickHouse telemetry sink use?

The sink writes to a table named `telemetry` with columns including `event`, `session_key`, `turn_seq`, `duration_ms`, and `event_time`. The exact DDL is managed by the ingestion logic in [`MemoryKnowledge/src/clickhouse-telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/clickhouse-telemetry.ts), which handles buffering and batch inserts to optimize write performance.

### Where is the telemetry system initialized in the application lifecycle?

The `initTelemetry()` function is invoked during server startup in [`MemoryKnowledge/src/server.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/server.ts). This ensures the `TracerProvider` is registered before any request handlers are mounted, guaranteeing that all subsequent calls to `withSpan()` capture complete traces from the beginning of the request lifecycle.

### How can I add custom attributes to existing spans?

Import `withSpan` from [`MemoryKnowledge/src/telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/telemetry.ts) and use the span object passed to your callback. Call `span.setAttribute(key, value)` with string or numeric values; these attributes are serialized into the ClickHouse row or forwarded to Langfuse depending on your configured exporters.