How to Configure LLM Routing with Proxy and BYO Modes in TencentDB Agent Memory

TencentDB Agent Memory supports two mutually exclusive LLM routing modes: proxy (default), which routes all agent requests through the Memory Proxy service at port 8096, and custom (BYO), which bypasses the proxy for direct LLM calls using the memory gateway configuration.

When you configure LLM routing with proxy and BYO modes, you determine whether AI agent requests flow through an intermediary proxy layer or communicate directly with your chosen LLM provider. This architecture, implemented in the TencentCloud/TencentDB-Agent-Memory repository, uses the LLM_MODE environment variable to control request flow between the Panel UI, Agent SDKs, and upstream language models.

Understanding the Two LLM Routing Modes

TencentDB Agent Memory separates routing logic from memory services through distinct operational modes defined in the deployment configuration.

Proxy Mode (Default) routes all LLM calls from agents through the Memory Proxy (tdai-proxy). According to deploy/panel-knowledge-combined/start-combined.sh (lines 82-96), when LLM_MODE=proxy, the system writes a proxy_endpoint field into metadata-instances.json, causing client SDKs to target http://<host>:8096. The proxy then forwards requests to an upstream LLM defined by PROXY_UPSTREAM_URL, PROXY_UPSTREAM_API_KEY, and PROXY_UPSTREAM_MODEL, while injecting context assets and enforcing authentication logic implemented in MemoryProxy/src/handler.ts.

BYO (Custom) Mode bypasses the proxy entirely. When LLM_MODE=custom, the Panel omits the proxy_endpoint field (falling back to gateway_endpoint per lines 26-40 of start-combined.sh), and agents call the LLM directly using the memory group configuration (MEMORY_UPSTREAM_URL, MEMORY_UPSTREAM_API_KEY, MEMORY_UPSTREAM_MODEL). This mode eliminates proxy-level features like session initialization and cost guarding in favor of lower latency direct connections.

Configuration File Structure

Before implementing either mode, understand the key source files controlling the routing behavior:

• deploy/panel-knowledge-combined/start-combined.sh – Sets the LLM_MODE default value and injects routing endpoints into the panel configuration JSON.
• deploy/global-images/.env.example – Defines the complete schema for environment variables across both routing modes.
• MemoryProxy/src/handler.ts – Contains the core request-forwarding logic that processes proxy-mode LLM calls.
• deploy/global-images/start-proxy.sh – Generates the proxy's config.yaml from environment variables (lines 46-78).

Step-by-Step Configuration Guide

Initialize Environment Variables

Begin by copying the example environment file in the global-images directory:

cd deploy/global-images
cp .env.example .env

Edit .env to set your base LLM credentials. These variables serve as fallbacks or primary configuration depending on your selected mode.

Configure Proxy Mode (Default)

To enable the default routing through Memory Proxy:

1. Set the mode identifier:

   LLM_MODE=proxy

2. Define the client-facing proxy address:

   REMOTE_INSTANCE_PROXY_URL=http://localhost:8096

3. Configure the upstream LLM that the proxy will call:

   PROXY_UPSTREAM_URL=https://api.openai.com/v1
   PROXY_UPSTREAM_API_KEY=sk-your-openai-key
   PROXY_UPSTREAM_MODEL=gpt-4o

According to the README (lines 90-95), the Panel UI will display this REMOTE_INSTANCE_PROXY_URL value in the "客户端接入地址" (Client Access Address) card, ensuring agents connect through the managed proxy endpoint.

Configure BYO (Custom) Mode

To bypass the proxy and enable direct LLM communication:

1. Set the mode to custom:

   LLM_MODE=custom

2. Remove or comment out REMOTE_INSTANCE_PROXY_URL. The Panel will automatically fall back to the gateway_endpoint.

3. Configure the memory group variables for direct access:

   MEMORY_UPSTREAM_URL=https://api.deepseek.com/v1
   MEMORY_UPSTREAM_API_KEY=sk-your-deepseek-key
   MEMORY_UPSTREAM_MODEL=deepseek-coder-v2

In this configuration, the deploy/global-images/start-all.sh script (lines 52-66) orchestrates container startup without initializing the proxy service dependency, and agents connect directly to the memory-core gateway at port 8125.

Launch the Complete Stack

After configuring .env, execute the orchestration script:

./start-all.sh

This script validates the LLM_MODE variable and spins up the appropriate services (memory-core, memory-hub, and optionally tdai-proxy).

SDK Implementation Examples

Using Proxy Mode in TypeScript

When operating in proxy mode, point the SDK to the proxy port (8096). The proxy handles authentication with the upstream provider:

import { MemoryClient } from '@tencentdb-agent-memory/memory-core';

const client = new MemoryClient({
  baseURL: 'http://localhost:8096/v3',
  apiKey: '' // Authentication handled by proxy configuration
});

await client.chatMemory.create({
  messages: [{ role: 'user', content: 'Hello' }]
});

Using BYO Mode in TypeScript

For custom mode, target the memory-hub gateway directly and provide your LLM API key in the client constructor:

import { MemoryClient } from '@tencentdb-agent-memory/memory-core';

const client = new MemoryClient({
  baseURL: 'http://localhost:8125/v3',
  apiKey: process.env.MEMORY_UPSTREAM_API_KEY
});

await client.chatMemory.create({
  messages: [{ role: 'user', content: 'Hello' }]
});

Architecture and Source Code References

The routing decision logic resides in multiple coordinated components. In MemoryProxy/src/handler.ts, the proxy evaluates incoming requests against its configuration before forwarding them to the URL specified in PROXY_UPSTREAM_URL.

The environment resolution happens at runtime in deploy/panel-knowledge-combined/start-combined.sh. Lines 82-96 explicitly export LLM_MODE with a default value of proxy, while lines 26-40 conditionally inject the proxy_endpoint field into the panel's metadata JSON only when REMOTE_INSTANCE_PROXY_URL is non-empty.

For API specifications, consult MemoryProxy/v3-api-memoryproxy-doc.md when implementing proxy-mode clients, and MemoryCore/v3-api-memorycore-doc.md for BYO-mode direct gateway integrations. These documents define the /v3 namespace endpoints that both routing modes ultimately expose to agent applications.

Summary

• Proxy Mode (LLM_MODE=proxy) is the default configuration that routes agent requests through tdai-proxy at port 8096, enabling centralized authentication, context injection, and cost control.
• BYO Mode (LLM_MODE=custom) bypasses the proxy for direct LLM calls using MEMORY_UPSTREAM_* variables, reducing latency but losing proxy-layer features.
• The REMOTE_INSTANCE_PROXY_URL environment variable determines whether the Panel UI exposes a proxy endpoint or falls back to the direct gateway address.
• Configuration changes take effect by restarting the stack via deploy/global-images/start-all.sh, which enforces the correct service topology based on LLM_MODE.

Frequently Asked Questions

What is the default LLM routing mode in TencentDB Agent Memory?

Proxy mode is the default. According to lines 82-96 of deploy/panel-knowledge-combined/start-combined.sh, the LLM_MODE variable defaults to proxy if not explicitly set, causing all agent traffic to route through the Memory Proxy service at port 8096.

Can I use different LLM providers for the proxy and the memory core?

Yes. The PROXY_UPSTREAM_* and MEMORY_UPSTREAM_* variable groups are independent. You can configure the proxy to call OpenAI (PROXY_UPSTREAM_URL=https://api.openai.com/v1) while the memory core retains different credentials for internal operations, though typically these reference the same provider unless implementing complex failover scenarios.

How do I verify which routing mode is currently active?

Check the Panel UI or environment variables. Navigate to http://localhost:8125 and inspect the team's client access address card. If it displays the value of REMOTE_INSTANCE_PROXY_URL, proxy mode is active. If it shows a gateway address, BYO mode is enabled. Alternatively, inspect the running containers: active proxy mode requires the tdai-proxy container to be healthy on port 8096.

Does BYO mode support proxy-specific features like session initialization?

No. BYO mode bypasses MemoryProxy/src/handler.ts entirely, which means features like automatic session initialization, authentication enforcement, and context asset injection managed by the proxy layer are not applied. When using LLM_MODE=custom, you must implement these concerns in your client application or upstream LLM gateway.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}