How Continue Handles Token Counting and Context Window Management

Continue stays within model limits by counting tokens for every prompt component using model-specific encoders, then pruning the oldest or least-critical content until the total fits inside the context window while preserving system prompts, tool definitions, and recent exchanges.

Continue, the open-source AI coding assistant, implements sophisticated token counting and context window management to ensure requests never exceed a model's maximum capacity. According to the continuedev/continue source code, the system combines accurate token encoding with intelligent pruning strategies to maximize available context while dynamically adjusting for each model's specific tokenizer and safety requirements.

Token Counting Implementation in core/llm/countTokens.ts

The core token accounting logic resides in core/llm/countTokens.ts, which provides synchronous and asynchronous methods for calculating token costs across different content types.

Synchronous and Async Token Counting

The countTokens(content, modelName) function synchronously calculates tokens for MessageContent (strings or arrays of parts). It selects the appropriate encoder—js-tiktoken for OpenAI models or a Llama-based tokenizer for others—and passes text through encoding.encode(). After obtaining the base count, it applies model-specific scaling via getAdjustedTokenCountFromModel:

return getAdjustedTokenCountFromModel(baseTokens, modelName);

For web-worker environments, countTokensAsync(content, modelName) returns a Promise using the LlamaAsyncEncoder:

return (await encoding.encode(content ?? "")).length;

Tool and Message Token Accounting

When calculating costs for function-calling capabilities, countToolsTokens(tools, modelName) computes the token overhead for tool names, descriptions, and parameter schemas, adding a fixed overhead of 12 tokens per tool:

return numTokens + 12;

For complete chat messages, countChatMessageTokens(modelName, chatMessage) aggregates tokens for content, role tags, tool-call markers, and optional fields like thinking or signature. It starts with a BASE_TOKENS constant, then adds the encoded content length:

let tokens = BASE_TOKENS;
// ...
tokens += countTokens(chatMessage.content, modelName);

Safety Buffer Calculation

To prevent edge cases where trimming cuts too close to the limit, getTokenCountingBufferSafety(contextLength) reserves either 1,000 tokens or 2% of the context length, whichever is smaller:

return Math.min(MAX_TOKEN_SAFETY_BUFFER, contextLength * TOKEN_SAFETY_PROPORTION);

Context Window Pruning Strategies

When a prompt exceeds the model's context length, Continue employs a hierarchy of pruning helpers in core/llm/countTokens.ts to remove content strategically without breaking code structure or losing critical context.

String-Based Pruning

pruneStringFromTop and pruneStringFromBottom remove tokens from the start or end of raw strings by encoding the full text, slicing the token array to maxTokens, then decoding the result. These functions handle unstructured text where line boundaries don't matter.

Line-Based Pruning

pruneLinesFromTop and pruneLinesFromBottom preserve code integrity by pruning whole lines. They pre-compute token counts per line using countTokens(line, modelName), then drop lines until the sum plus newline tokens is ≤ maxTokens. The VS Code extension uses these in extensions/vscode/src/apply/ApplyManager.ts and extensions/vscode/src/diff/vertical/manager.ts to fit diff payloads into the model window without splitting code blocks.

High-Level Pruning Orchestration

pruneRawPromptFromTop orchestrates the full calculation by determining the max usable tokens (contextLength - tokensForCompletion - safetyBuffer), then invoking pruneStringFromTop:

return pruneStringFromTop(modelName, maxTokens, prompt);

The autocomplete system leverages this in core/autocomplete/util/HelperVars.ts and core/autocomplete/templating/formatOpenedFilesContext.ts via pruneStringFromBottom to retain only the most recent parts of large files.

End-to-End Context Management Flow

When Continue builds an LLM request, it executes a deterministic pipeline to maximize context utilization:

1. Collect data: Aggregate system prompts, tool definitions, recent chat messages, and file snippets.

2. Count tokens: Calculate costs for each component using countTokens and countChatMessageTokens.

3. Calculate usable space:

   const usableTokens = contextLength
                       - tokensForCompletion
                       - getTokenCountingBufferSafety(contextLength);

4. Prune content: Remove oldest parts (typically previous assistant messages) using pruneRawPromptFromTop until the total ≤ usableTokens.

5. Log and send: The LLM wrapper in core/llm/index.ts records promptTokens and generatedTokens for telemetry before transmitting the trimmed payload.

The Next-Edit provider in core/nextEdit/providers/BaseNextEditProvider.ts additionally uses countTokens on individual lines to determine if file edits fit within remaining context.

Practical Implementation Examples

Counting Tokens for a Chat Message

import { countChatMessageTokens } from "core/llm/countTokens.js";

const msg = {
  role: "assistant",
  content: "Here is the refactored code you asked for.",
  toolCalls: [{ name: "search", arguments: { query: "token counting" } }],
};

const tokenCount = countChatMessageTokens("gpt-4", msg);
console.log(`Message uses ${tokenCount} tokens`);

This example calls the function defined at lines 84-100 in core/llm/countTokens.ts, which accounts for role tags, content, and tool call overhead.

Pruning a Large Code Snippet for an 8K Window

import { pruneStringFromBottom, getTokenCountingBufferSafety } from "core/llm/countTokens.js";

const fullCode = await readFile("src/largeFile.ts");
const contextLength = 8192;
const reservedForCompletion = 512;

const maxPromptTokens =
  contextLength - reservedForCompletion - getTokenCountingBufferSafety(contextLength);

const trimmed = pruneStringFromBottom("gpt-4", maxPromptTokens, fullCode);
console.log(`Trimmed prompt length: ${trimmed.length}`);

This approach ensures the model has guaranteed space for its response while keeping the most recent code context intact.

Using High-Level Pruning in a VS Code Extension

import { pruneRawPromptFromTop } from "core/llm/countTokens.js";

const prompt = await buildFullPrompt(); // Contains system msg, tools, file diff, etc.
const contextLength = 16384;            // gpt-4-turbo limit
const completionTokens = 1024;

const safePrompt = pruneRawPromptFromTop("gpt-4-turbo", contextLength, prompt, completionTokens);
sendToLLM(safePrompt);

The pruneRawPromptFromTop helper (lines 78-90) automatically calculates the safety buffer and trims from the beginning of the prompt, preserving the most recent conversational turns at the end.

Summary

• Centralized token counting in core/llm/countTokens.ts supports both OpenAI (js-tiktoken) and Llama-based tokenizers with sync and async variants.
• Model-specific adjustments apply via getAdjustedTokenCountFromModel, while getTokenCountingBufferSafety reserves 2% or 1,000 tokens (whichever is smaller) to prevent boundary violations.
• Strategic pruning operates at string levels (unstructured text) and line levels (code preservation), with pruneRawPromptFromTop handling the final orchestration.
• Integration across features extends from autocomplete (HelperVars.ts) to VS Code diff management (ApplyManager.ts) and Next-Edit providers, ensuring consistent context window compliance.
• Telemetry verification occurs in core/llm/index.ts, logging actual promptTokens and generatedTokens to confirm pruning succeeded.

Frequently Asked Questions

How does Continue count tokens for different models?

Continue selects the appropriate encoder based on the model name. OpenAI models use js-tiktoken, while other models use a Llama-based tokenizer. The countTokens function in core/llm/countTokens.ts automatically dispatches to the correct encoder and applies model-specific adjustments via getAdjustedTokenCountFromModel to account for different tokenization schemes.

What is the token safety buffer and why is it needed?

The safety buffer reserves either 1,000 tokens or 2% of the total context length (whichever is smaller) through getTokenCountingBufferSafety. This prevents Continue from cutting too close to the model's absolute limit, accounting for potential tokenization discrepancies between the client's counter and the model's actual tokenizer, and ensuring room for metadata or formatting overhead.

How does Continue decide what to prune from the context?

Continue prioritizes preserving system prompts, tool definitions, and the most recent user-assistant exchange. When pruning is necessary, it removes from the oldest parts first using pruneRawPromptFromTop or trims large file contents from the bottom (oldest lines) using pruneStringFromBottom. This strategy maintains conversational coherence while maximizing relevant recent context.

Can I configure the context window size manually?

While Continue automatically detects context lengths from model configurations, you can influence behavior by selecting specific model presets or configuring contextLength in your Continue configuration file. The system will respect these limits when calculating usableTokens and apply the standard safety buffer formula to your specified window size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}