How Automatic Context Cutoff Works in ChocolateLMLite: Token and Time-Based Trimming

ChocolateLMLite automatically trims conversation history using a two-stage process that enforces token budgets and optional time windows before every LLM request.

Managing long conversation histories in LLM applications requires careful token management to avoid exceeding model limits. In the open-source project gpsnmeajp/chocolatelmlite, the automatic context cutoff system ensures that each request stays within budget by intelligently pruning older messages. This mechanism combines token-based truncation with optional time-based filtering to maintain context relevance while preventing API errors.

Two-Stage Context Trimming Pipeline

The cutoff operates in two coordinated stages managed by the FileManager and Tokens classes.

Stage 1: Time-Based Cutoff (Optional)

If TalkHistoryCutoffByPastHours is greater than zero, the system removes entries older than the specified hour limit. When TalkHistoryCutoffBeforeSummary is enabled, the algorithm protects messages preceding the latest persona summary from removal. This logic resides in FileManager.ApplyTalkHistoryTimeCutoff (lines 1381–1413 in src/FileManager.cs).

Stage 2: Token-Budget Cutoff

After time filtering, Tokens.TrimTalkTokens (lines 42–65 in src/Tokens.cs) processes the remaining history. Starting from the newest entry, it accumulates messages until the total token count—system prompt plus approximately 200 tokens of overhead—would exceed TalkHistoryCutoffThreshold. The resulting trimmed list becomes the final context sent to the LLM.

Configuration Settings for Context Management

The cutoff behavior is controlled through FileManager.GeneralSettings:

• TalkHistoryCutoffThreshold: Hard token ceiling (default: 40,000 tokens)
• TalkHistoryCutoffByPastHours: Hours to retain (default: 0, disabled)
• TalkHistoryCutoffBeforeSummary: Protects pre-summary messages when true (default: false)

These settings are deserialized from the persona JSON definition in src/Persona.cs (around line 749).

Implementation Details and Code Flow

The LLM driver orchestrates the pipeline in src/LLM.cs (line 198). It first applies the time-based filter, then invokes the token trimmer. If the final message count equals zero, the system aborts with an error indicating the token limit is too low.

Practical example of the flow:

// Retrieve cached history
var allMessages = fileManager.GetAllTalkHistoryAllFromActivePersonaCached();

// Apply time-based filtering
int archivedByTime;
var timeFiltered = fileManager.GetTalkHistoryWithTimeCutoffFromActivePersona(
    activePersonaSettings.TalkHistoryCutoffByPastHours,
    activePersonaSettings.TalkHistoryCutoffBeforeSummary,
    activePersonaSummary?.Timestamp ?? 0,
    out archivedByTime);

// Enforce token budget
var trimmed = Tokens.TrimTalkTokens(
    systemPrompt,
    timeFiltered,
    fileManager.GeneralSettings.TalkHistoryCutoffThreshold);

Runtime configuration adjustments:

// Increase budget to 80K tokens
fileManager.GeneralSettings.TalkHistoryCutoffThreshold = 80 * 1000;

// Enable 12-hour retention with summary protection
fileManager.GeneralSettings.TalkHistoryCutoffByPastHours = 12;
fileManager.GeneralSettings.TalkHistoryCutoffBeforeSummary = true;

Error Handling When Context Is Fully Trimmed

When TalkHistoryCutoffThreshold is too restrictive, Tokens.TrimTalkTokens returns an empty collection. The LLM layer detects this condition and throws an InvalidOperationException with a message containing "会話履歴が全てカットオフ" (all conversation history cut off).

Handle this scenario as follows:

try
{
    var reply = await llm.GetResponseAsync(...);
}
catch (InvalidOperationException ex) when (ex.Message.Contains("会話履歴が全てカットオフ"))
{
    // Prompt user to increase threshold or reduce system prompt
    Console.WriteLine("Context fully trimmed. Raise TalkHistoryCutoffThreshold or reduce system prompt size.");
}

Summary

• ChocolateLMLite implements automatic context cutoff through two sequential stages: optional time-based filtering followed by mandatory token-budget trimming.
• The FileManager.ApplyTalkHistoryTimeCutoff method in src/FileManager.cs handles temporal pruning, while Tokens.TrimTalkTokens in src/Tokens.cs enforces the token ceiling.
• Configuration occurs via TalkHistoryCutoffThreshold, TalkHistoryCutoffByPastHours, and TalkHistoryCutoffBeforeSummary in FileManager.GeneralSettings.
• The LLM pipeline in src/LLM.cs coordinates both stages and throws descriptive errors when the context is completely eliminated.

Frequently Asked Questions

What happens if the token limit is set too low in ChocolateLMLite?

If TalkHistoryCutoffThreshold is insufficient to accommodate even the system prompt plus overhead, Tokens.TrimTalkTokens returns an empty message list. The LLM driver in src/LLM.cs (around line 204) detects this condition and raises an exception with the message "会話履歴が全てカットオフ", indicating that no conversation history remains for the model to process.

How does the time-based cutoff interact with persona summaries?

When TalkHistoryCutoffBeforeSummary is enabled, the time-based cutoff algorithm in FileManager.ApplyTalkHistoryTimeCutoff compares entry timestamps against the latest summary timestamp. Messages created before the summary are retained regardless of the TalkHistoryCutoffByPastHours setting, ensuring that the persona's accumulated context remains intact.

Can I disable the automatic context cutoff entirely?

You cannot completely disable token-based trimming, as it prevents API errors from exceeding model limits. However, you can effectively disable time-based filtering by setting TalkHistoryCutoffByPastHours to 0 (the default). To minimize truncation, set TalkHistoryCutoffThreshold to a high value appropriate for your LLM's context window.

Where is the token counting logic implemented?

The core token estimation and trimming logic resides in src/Tokens.cs, specifically within the TrimTalkTokens method (lines 42–65). This method estimates token counts including approximately 200 tokens of overhead for tools and auto-inserted content, then truncates the conversation history from the oldest entries outward while preserving the newest messages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}