How TencentDB Agent Memory Handles Context Offload When Token Limits Are Exceeded

TencentDB Agent Memory uses a three-stage offload server architecture that detects token limit violations via finish_reason="length" flags, automatically escalates requests through L1, L1.5, and L2 processing tiers with adaptive max_tokens allocation, and persists intermediate fragments to storage for final assembly.

When processing large language model requests, the TencentCloud/TencentDB-Agent-Memory repository implements a dedicated offload mechanism to prevent token budget violations while preserving complete conversation context. The system divides processing into configurable stages with progressive token ceilings, ensuring that oversized payloads are automatically segmented and processed across multiple LLM invocations without data loss.

Three-Stage Token Management Pipeline

Stage-Specific Configuration

The offload pipeline organizes processing into three distinct tiers—L1, L1.5, and L2—each governed by independent token limits specified in MemoryCore/src/offload_server/config/*.yaml. These configuration files define l1MaxTokens, l15MaxTokens, and l2MaxTokens parameters that establish progressively higher capacity ceilings. This tiered structure allows the system to handle contexts ranging from small queries to extensive conversation histories without violating model context windows.

The Offload Execution Flow

Token Budget Validation and Trimming

When a request arrives at the /v2/offload/ingest endpoint defined in MemoryCore/src/offload_server/router.ts, the executor immediately measures the total payload size including system prompts and user messages. If the token count exceeds the current stage's limit, the system trims the payload to the allowable size while recording the original dimensions in a usage object. This validation occurs before any LLM invocation, ensuring that outbound requests never exceed the configured budget for that processing tier.

Detecting LLM Truncation via finish_reason

The core detection mechanism resides in MemoryCore/src/offload_server/offload-task-executor.ts. After each LLM call, the executor inspects the response's finish_reason field. When this value equals "length", the code recognizes that the model stopped generating output due to hitting the max_tokens limit, triggering the adaptive escalation logic.

Adaptive Token Allocation and Stage Escalation

Upon detecting a truncation event, the executor implements an exponential backoff strategy for token limits. The system calculates a new limit using Math.min(currentConfig.max_tokens * 2, HARD_MAX), updates the runtime configuration, and invokes scheduleNextStage(task, newLimit) to progress the request to the next tier (L1 → L1.5 → L2). This progressive escalation ensures that larger contexts are accommodated through successive attempts rather than request rejection, with each stage offering approximately double the token capacity of the previous tier.

Persistent Storage and Final Assembly

Each offload stage writes its partial results to the offload/{sessionId}/ directory structure, as implemented in the storage handlers within MemoryCore/src/offload_server/router.ts. The task-transition.ts state machine manages the lifecycle of these distributed tasks, ensuring that fragments from interrupted or multi-stage processes remain available. Once all stages complete or the token budget is satisfied, the /v2/offload/compact endpoint retrieves these stored fragments, concatenates them into a coherent response, and returns the complete result to the client while reporting aggregated token usage statistics (prompt_tokens, completion_tokens, total_tokens).

Implementation Examples

Client API Integration

The following TypeScript example demonstrates how client applications interact with the offload endpoint:

import axios from "axios";

async function askOffload(sessionId: string, messages: any[]) {
  const resp = await axios.post(
    `https://<offload-host>/v2/offload/ingest`,
    {
      sessionId,
      messages,
      max_tokens: 2048,
    }
  );

  return resp.data;
}

Internal Truncation Handling Logic

This excerpt from offload-task-executor.ts illustrates the detection and adaptation mechanism:

if (llmResp.finish_reason === "length") {
  const newLimit = Math.min(currentConfig.max_tokens * 2, HARD_MAX);
  this.config.max_tokens = newLimit;
  scheduleNextStage(task, newLimit);
}

Summary

• Three-stage pipeline: The system processes requests through L1, L1.5, and L2 tiers with independently configurable token ceilings (l1MaxTokens, l15MaxTokens, l2MaxTokens) defined in YAML configuration files.
• Truncation detection: The offload-task-executor.ts module monitors LLM responses for finish_reason === "length" to identify token limit violations immediately.
• Adaptive escalation: Upon detection, the system automatically increases max_tokens by doubling the current limit (bounded by HARD_MAX) and escalates the request to the next processing stage via scheduleNextStage().
• Data persistence: Intermediate results are stored at offload/{sessionId}/ paths and later assembled by the compact endpoint to ensure no conversation context is lost during multi-stage processing.
• Progressive processing: Rather than rejecting oversized contexts, the architecture leverages multiple LLM invocations with expanding token budgets to handle arbitrary conversation lengths.

Frequently Asked Questions

What triggers a context offload in TencentDB Agent Memory?

A context offload initiates when the combined token count of the system prompt and user messages exceeds the current stage's configured limit (l1MaxTokens, l15MaxTokens, or l2MaxTokens). The system detects this condition in offload-task-executor.ts and automatically routes the request through the escalation pipeline rather than truncating the conversation history.

How does the system prevent data loss during multi-stage processing?

The system writes partial LLM responses to persistent storage at offload/{sessionId}/ directories after each stage completion. The task-transition.ts state machine manages progression between L1, L1.5, and L2 stages, while the /v2/offload/compact endpoint retrieves and concatenates all fragments into the final response, ensuring complete preservation of conversation data across multiple round-trips.

What is the maximum token limit for each offload stage?

The maximum token limits are configurable via YAML files in MemoryCore/src/offload_server/config/, with defaults set for l1MaxTokens, l15MaxTokens, and l2MaxTokens. Each successive stage offers approximately double the capacity of the previous tier, and the system enforces a HARD_MAX upper bound during the Math.min(currentConfig.max_tokens * 2, HARD_MAX) calculation to prevent resource exhaustion.

How can developers configure custom token thresholds?

Developers modify the token ceilings by editing the YAML configuration files located in MemoryCore/src/offload_server/config/*.yaml. These settings directly control the l1MaxTokens, l15MaxTokens, and l2MaxTokens parameters that determine when the system escalates requests from one processing stage to the next, allowing customization based on specific LLM context window sizes and memory constraints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}