# How TencentDB Agent Memory Handles Context Offload When Token Limits Are Exceeded

> Discover how TencentDB Agent Memory handles context offload when token limits are exceeded using a three-stage architecture and adaptive max_tokens allocation for seamless request processing.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-09-04

---

**TencentDB Agent Memory uses a three-stage offload server architecture that detects token limit violations via `finish_reason="length"` flags, automatically escalates requests through L1, L1.5, and L2 processing tiers with adaptive `max_tokens` allocation, and persists intermediate fragments to storage for final assembly.**

When processing large language model requests, the TencentCloud/TencentDB-Agent-Memory repository implements a dedicated offload mechanism to prevent token budget violations while preserving complete conversation context. The system divides processing into configurable stages with progressive token ceilings, ensuring that oversized payloads are automatically segmented and processed across multiple LLM invocations without data loss.

## Three-Stage Token Management Pipeline

### Stage-Specific Configuration

The offload pipeline organizes processing into three distinct tiers—**L1**, **L1.5**, and **L2**—each governed by independent token limits specified in `MemoryCore/src/offload_server/config/*.yaml`. These configuration files define `l1MaxTokens`, `l15MaxTokens`, and `l2MaxTokens` parameters that establish progressively higher capacity ceilings. This tiered structure allows the system to handle contexts ranging from small queries to extensive conversation histories without violating model context windows.

## The Offload Execution Flow

### Token Budget Validation and Trimming

When a request arrives at the `/v2/offload/ingest` endpoint defined in [`MemoryCore/src/offload_server/router.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/offload_server/router.ts), the executor immediately measures the total payload size including system prompts and user messages. If the token count exceeds the current stage's limit, the system trims the payload to the allowable size while recording the original dimensions in a `usage` object. This validation occurs before any LLM invocation, ensuring that outbound requests never exceed the configured budget for that processing tier.

### Detecting LLM Truncation via finish_reason

The core detection mechanism resides in [`MemoryCore/src/offload_server/offload-task-executor.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/offload_server/offload-task-executor.ts). After each LLM call, the executor inspects the response's `finish_reason` field. When this value equals `"length"`, the code recognizes that the model stopped generating output due to hitting the `max_tokens` limit, triggering the adaptive escalation logic.

### Adaptive Token Allocation and Stage Escalation

Upon detecting a truncation event, the executor implements an exponential backoff strategy for token limits. The system calculates a new limit using `Math.min(currentConfig.max_tokens * 2, HARD_MAX)`, updates the runtime configuration, and invokes `scheduleNextStage(task, newLimit)` to progress the request to the next tier (L1 → L1.5 → L2). This progressive escalation ensures that larger contexts are accommodated through successive attempts rather than request rejection, with each stage offering approximately double the token capacity of the previous tier.

## Persistent Storage and Final Assembly

Each offload stage writes its partial results to the `offload/{sessionId}/` directory structure, as implemented in the storage handlers within [`MemoryCore/src/offload_server/router.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/offload_server/router.ts). The [`task-transition.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/task-transition.ts) state machine manages the lifecycle of these distributed tasks, ensuring that fragments from interrupted or multi-stage processes remain available. Once all stages complete or the token budget is satisfied, the `/v2/offload/compact` endpoint retrieves these stored fragments, concatenates them into a coherent response, and returns the complete result to the client while reporting aggregated token usage statistics (`prompt_tokens`, `completion_tokens`, `total_tokens`).

## Implementation Examples

### Client API Integration

The following TypeScript example demonstrates how client applications interact with the offload endpoint:

```typescript
import axios from "axios";

async function askOffload(sessionId: string, messages: any[]) {
  const resp = await axios.post(
    `https://<offload-host>/v2/offload/ingest`,
    {
      sessionId,
      messages,
      max_tokens: 2048,
    }
  );

  return resp.data;
}

```

### Internal Truncation Handling Logic

This excerpt from [`offload-task-executor.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/offload-task-executor.ts) illustrates the detection and adaptation mechanism:

```typescript
if (llmResp.finish_reason === "length") {
  const newLimit = Math.min(currentConfig.max_tokens * 2, HARD_MAX);
  this.config.max_tokens = newLimit;
  scheduleNextStage(task, newLimit);
}

```

## Summary

- **Three-stage pipeline**: The system processes requests through L1, L1.5, and L2 tiers with independently configurable token ceilings (`l1MaxTokens`, `l15MaxTokens`, `l2MaxTokens`) defined in YAML configuration files.
- **Truncation detection**: The [`offload-task-executor.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/offload-task-executor.ts) module monitors LLM responses for `finish_reason === "length"` to identify token limit violations immediately.
- **Adaptive escalation**: Upon detection, the system automatically increases `max_tokens` by doubling the current limit (bounded by `HARD_MAX`) and escalates the request to the next processing stage via `scheduleNextStage()`.
- **Data persistence**: Intermediate results are stored at `offload/{sessionId}/` paths and later assembled by the compact endpoint to ensure no conversation context is lost during multi-stage processing.
- **Progressive processing**: Rather than rejecting oversized contexts, the architecture leverages multiple LLM invocations with expanding token budgets to handle arbitrary conversation lengths.

## Frequently Asked Questions

### What triggers a context offload in TencentDB Agent Memory?

A context offload initiates when the combined token count of the system prompt and user messages exceeds the current stage's configured limit (`l1MaxTokens`, `l15MaxTokens`, or `l2MaxTokens`). The system detects this condition in [`offload-task-executor.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/offload-task-executor.ts) and automatically routes the request through the escalation pipeline rather than truncating the conversation history.

### How does the system prevent data loss during multi-stage processing?

The system writes partial LLM responses to persistent storage at `offload/{sessionId}/` directories after each stage completion. The [`task-transition.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/task-transition.ts) state machine manages progression between L1, L1.5, and L2 stages, while the `/v2/offload/compact` endpoint retrieves and concatenates all fragments into the final response, ensuring complete preservation of conversation data across multiple round-trips.

### What is the maximum token limit for each offload stage?

The maximum token limits are configurable via YAML files in `MemoryCore/src/offload_server/config/`, with defaults set for `l1MaxTokens`, `l15MaxTokens`, and `l2MaxTokens`. Each successive stage offers approximately double the capacity of the previous tier, and the system enforces a `HARD_MAX` upper bound during the `Math.min(currentConfig.max_tokens * 2, HARD_MAX)` calculation to prevent resource exhaustion.

### How can developers configure custom token thresholds?

Developers modify the token ceilings by editing the YAML configuration files located in `MemoryCore/src/offload_server/config/*.yaml`. These settings directly control the `l1MaxTokens`, `l15MaxTokens`, and `l2MaxTokens` parameters that determine when the system escalates requests from one processing stage to the next, allowing customization based on specific LLM context window sizes and memory constraints.