How Open-Notebook Handles Token Estimation and Large Context Model Selection

Open-Notebook uses tiktoken's o200k_base encoder to precisely count tokens and automatically routes prompts exceeding 105,000 tokens to a large-context model family, ensuring reliable processing of very long documents.

The lfnovo/open-notebook repository implements a robust strategy for token estimation and large context model selection that prevents context window overflows while maintaining flexibility for different deployment environments. By combining accurate token counting with intelligent model routing, the system can safely process documents that far exceed standard LLM context limits.

Precise Token Counting with tiktoken

The foundation of Open-Notebook's token estimation strategy relies on the tiktoken library to deliver accurate token counts before sending requests to language models.

The token_count Utility

In open_notebook/utils/token_utils.py, the token_count function uses the o200k_base encoding—the same 200,000-token-capacity tokenizer employed by many modern providers. This encoder returns the exact number of tokens for any input string, allowing the system to make precise capacity decisions.

from open_notebook.utils.token_utils import token_count

text = "Open‑Notebook can process very long documents."
num_tokens = token_count(text)
print(f"The text contains {num_tokens} tokens")

Fallback for Offline Environments

When tiktoken cannot be imported—such as in offline or restricted environments—the utility gracefully falls back to a simple word-count heuristic. While this fallback is less precise, it ensures the application remains functional even without access to the full tokenizer library.

Automatic Large Context Model Selection

Once token counts are established, Open-Notebook applies a clear decision hierarchy to select an appropriate model capable of handling the payload.

The 105,000 Token Threshold

The system defines a hard threshold at 105,000 tokens to determine when a large-context model is required. This value sits safely below the 200,000-token ceiling of the o200k_base tokenizer while providing sufficient headroom above standard 8,000–32,000 token limits. Prompts exceeding this threshold automatically trigger selection from the "large_context" model family.

Model Resolution Logic in provision_langchain_model

The provision_langchain_model function in open_notebook/ai/provision.py implements the selection logic:

if tokens > 105_000:
    # Choose a model registered under the “large_context” family

elif model_id is supplied:
    # Use the explicitly requested model

else:
    # Use the default model for the requested type (e.g. “chat”, “completion”)

When the token count exceeds the threshold, the function calls model_manager.get_default_model("large_context") to retrieve a model with a 128,000 or 200,000-token window. If no model ID is specified and the content falls below the threshold, the system defaults to standard models configured for the requested operation type.

from open_notebook.ai.provision import provision_langchain_model

async def get_model_for_content(content: str):
    # No explicit model_id, default type is "chat"

    model = await provision_langchain_model(
        content=content,
        model_id=None,
        default_type="chat",
    )
    return model

You can manually specify a model ID, but the large-context override takes precedence when content exceeds 105,000 tokens:

model = await provision_langchain_model(
    content=very_long_text,
    model_id="gpt-4-32k",      # optional explicit ID

    default_type="chat",
)

# If `very_long_text` exceeds 105_000 tokens, the large-context default

# model will be selected regardless of the explicit ID.

Integration Across the Codebase

The token estimation and model selection strategy integrates with multiple components to enforce limits throughout the application.

Context Budgeting in context_builder.py

The context_builder.py module in open_notebook/utils/ utilizes token_count to enforce maximum token budgets when constructing request payloads. This prevents the system from accidentally building prompts that exceed model capacity before the provisioning stage even begins.

Model Family Registration

The ModelManager class in open_notebook/ai/models.py maintains the mapping between model families like "large_context" and concrete LLM provider implementations. When provision_langchain_model requests a large-context default, this manager resolves the configuration to an actual model instance capable of processing the extended input.

If no suitable model can be resolved—such as when no default large-context model is configured—the system raises a ConfigurationError with a clear message directing users to configure a default in the Settings → Models UI.

Summary

• Open-Notebook uses tiktoken with the o200k_base encoder in open_notebook/utils/token_utils.py for precise token estimation, with a word-count fallback for offline environments.
• The 105,000 token threshold in open_notebook/ai/provision.py triggers automatic selection of the "large_context" model family to handle very long documents.
• The provision_langchain_model function applies a hierarchical decision tree: large-context models for oversized content, explicit model IDs when provided, or defaults for standard sizes.
• ConfigurationError handling ensures users receive clear guidance when no suitable model is configured for the required context window.

Frequently Asked Questions

What tokenizer does Open-Notebook use for token estimation?

According to the source code in open_notebook/utils/token_utils.py, Open-Notebook uses the o200k_base encoding from the tiktoken library. This encoding supports up to 200,000 tokens and matches the tokenizer used by many modern providers, ensuring accurate counts across different model families.

Why is the threshold set at 105,000 tokens?

The 105,000-token threshold provides a safety margin below the 200,000-token capacity of the o200k_base tokenizer while remaining well above standard model limits of 8,000–32,000 tokens. This ensures that content exceeding typical limits gets routed to models explicitly registered with 128,000 or 200,000-token windows, preventing context overflow errors.

What happens if no large-context model is configured?

If provision_langchain_model cannot resolve a model—whether because no large-context default is set or the specified model ID is unavailable—the system raises a ConfigurationError. The error message directs users to the Settings → Models UI to configure a default model for the required family.

Can I force a specific model even for very long contexts?

You can supply an explicit model_id parameter to provision_langchain_model, but the large-context override logic takes precedence when content exceeds 105,000 tokens. If you need to force a specific model for very long content, you must ensure it is registered as the default "large_context" model in the ModelManager, or ensure your content stays below the 105,000-token threshold.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}