How to Monitor TencentDB Agent Memory Status Using OpenTelemetry and ClickHouse

TencentDB Agent Memory exposes request-level telemetry through an OpenTelemetry pipeline that exports latency, session metadata, and error metrics to ClickHouse and Langfuse, allowing operators to observe knowledge retrieval performance in real time.

The TencentCloud/TencentDB-Agent-Memory repository instruments its knowledge engine with built-in observability hooks. By configuring the telemetry initialiser and telemetry sinks, you can monitor memory agent status without deploying external APM agents, using only the existing MemoryKnowledge and MemoryCore modules.

Core Telemetry Components

OpenTelemetry Initialiser

The telemetry stack boots from MemoryKnowledge/src/telemetry.ts. The initTelemetry() function instantiates an OpenTelemetry NodeSDK, registers a TracerProvider, and conditionally enables the Langfuse exporter based on the LANGFUSE_SECRET_KEY environment variable. This file also exports the withSpan helper used throughout the codebase to wrap asynchronous operations.

ClickHouse Telemetry Sink

Persistent metrics are handled by MemoryKnowledge/src/clickhouse-telemetry.ts. This module implements a custom exporter that buffers spans and writes one row per request into a ClickHouse table named telemetry. The schema includes columns for event, session_key, turn_seq, duration_ms, and event_time, enabling time-series analysis of agent interactions.

Span Instrumentation

Individual components, such as the Wiki engine in MemoryKnowledge/src/engines/wiki/manager.ts, instrument their handlers by calling withSpan(). This function creates a child span that inherits the trace context, automatically capturing start and end timestamps for every knowledge retrieval operation.

Environment Configuration

Telemetry toggles and backend URLs are loaded via MemoryCore/src/utils/env.ts and validated against the schema in MemoryCore/src/utils/env-config.ts. The createLogger utility defined in env.ts reports high-level health information and initialization failures to stdout, providing immediate feedback during startup.

Monitoring Workflow

1. Agent Startup – When the Knowledge service starts via MemoryKnowledge/src/server.ts, it invokes initTelemetry() to configure the global tracer and optionally connect to ClickHouse.

2. Request Wrapping – Each incoming request is wrapped with withSpan(), which records the session key, turn sequence, and operation type as span attributes.

3. LLM Instrumentation – When the underlying LLM SDK is invoked with experimental_telemetry: { isEnabled: true }, it emits child spans that attach to the parent request span, capturing token usage and model latency.

4. Data Export – At the end of the request lifecycle, the span is ended. The exporter flush logic (managed by MemoryCore/src/utils/managed-timer.ts) writes the buffered data to ClickHouse and/or Langfuse based on your environment configuration.

Enabling Telemetry in Your Deployment

Set the following environment variables before starting the agent:

export LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxx      # Optional; enables Langfuse tracing

export KNOWLEDGE_CLICKHOUSE_URL=http://localhost:8123
export KNOWLEDGE_CLICKHOUSE_ENABLED=true

The agent reads these values during initialization in MemoryCore/src/utils/env.ts. If KNOWLEDGE_CLICKHOUSE_ENABLED is unset or false, the ClickHouse exporter is skipped, but OpenTelemetry spans remain active for other collectors.

Querying and Analyzing Metrics

ClickHouse SQL Examples

To analyze recent latency trends, connect to your ClickHouse instance and query the telemetry table:

SELECT
  session_key,
  turn_seq,
  duration_ms,
  event,
  toDateTime(event_time) AS ts
FROM telemetry
WHERE event = 'usage'
ORDER BY ts DESC
LIMIT 100;

This returns the last 100 usage events with their execution duration, allowing you to identify slow sessions or spikes in processing time.

Viewing Traces in Langfuse

When LANGFUSE_SECRET_KEY is configured, spans are forwarded to Langfuse automatically. Navigate to your Langfuse project dashboard (e.g., https://app.langfuse.com or a self-hosted instance) and filter by the service name tencentdb-agent-memory. The UI displays a waterfall view of each request, showing nested spans for database queries, LLM calls, and retrieval steps.

Adding Custom Instrumentation

To capture domain-specific metrics, wrap new functions with withSpan and attach custom attributes:

import { withSpan } from "./telemetry.js";

async function customRetrieval(query: string) {
  return await withSpan("retrieval.custom", async (span) => {
    span.setAttribute("query.length", query.length);
    span.setAttribute("retrieval.source", "vector_store");

    const results = await fetchResults(query);
    span.setAttribute("results.count", results.length);
    return results;
  });
}

These attributes appear as columns in ClickHouse (if mapped) or as metadata in Langfuse, enabling granular filtering by query characteristics.

Summary

• Initialization occurs in MemoryKnowledge/src/telemetry.ts via initTelemetry(), which configures the OpenTelemetry SDK and optional Langfuse exporter.
• Data Persistence is handled by MemoryKnowledge/src/clickhouse-telemetry.ts, writing structured rows to the telemetry table for long-term storage.
• Instrumentation relies on the withSpan() helper to capture timing and context from MemoryKnowledge/src/engines/wiki/manager.ts and other modules.
• Configuration is driven by environment variables defined in MemoryCore/src/utils/env.ts, including KNOWLEDGE_CLICKHOUSE_ENABLED and LANGFUSE_SECRET_KEY.
• Querying can be performed via raw SQL against ClickHouse or through the Langfuse UI for distributed trace visualization.

Frequently Asked Questions

How do I enable ClickHouse telemetry without Langfuse?

Set KNOWLEDGE_CLICKHOUSE_ENABLED=true and provide a valid KNOWLEDGE_CLICKHOUSE_URL, but leave LANGFUSE_SECRET_KEY unset or empty. The initialiser in MemoryKnowledge/src/telemetry.ts checks for the secret key before registering the Langfuse exporter, so omitting it disables that sink while keeping the ClickHouse exporter active.

What database schema does the ClickHouse telemetry sink use?

The sink writes to a table named telemetry with columns including event, session_key, turn_seq, duration_ms, and event_time. The exact DDL is managed by the ingestion logic in MemoryKnowledge/src/clickhouse-telemetry.ts, which handles buffering and batch inserts to optimize write performance.

Where is the telemetry system initialized in the application lifecycle?

The initTelemetry() function is invoked during server startup in MemoryKnowledge/src/server.ts. This ensures the TracerProvider is registered before any request handlers are mounted, guaranteeing that all subsequent calls to withSpan() capture complete traces from the beginning of the request lifecycle.

How can I add custom attributes to existing spans?

Import withSpan from MemoryKnowledge/src/telemetry.ts and use the span object passed to your callback. Call span.setAttribute(key, value) with string or numeric values; these attributes are serialized into the ClickHouse row or forwarded to Langfuse depending on your configured exporters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}