How ModelManager Selects AI Models Based on Token Context Size in Open Notebook

Open Notebook automatically routes prompts exceeding 105,000 tokens to a dedicated large-context model by counting tokens with tiktoken and querying ModelManager for the appropriate default based on size thresholds.

The lfnovo/open-notebook repository implements an intelligent model selection system that transparently handles token context limits. When processing AI requests, the application measures prompt size and automatically falls back to models configured for large contexts when thresholds are exceeded. This prevents truncation errors and ensures seamless handling of extensive documents.

Token Counting with tiktoken

The system begins by calculating the exact token count of input content using the token_count utility function.

Located in open_notebook/utils/token_utils.py, this helper leverages the tiktoken library with the o200k_base encoder to compute precise token counts:

from open_notebook.utils.token_utils import token_count

tokens = token_count(content)  # Returns integer token count

The o200k_base encoder provides accurate tokenization aligned with modern OpenAI model specifications, ensuring consistent measurements across different input types.

The 105K Token Threshold Logic

The core selection logic resides in open_notebook/ai/provision.py within the async function provision_langchain_model. This function evaluates three parameters: the raw content, an optional model_id, and a default_type for fallback scenarios.

When token_count(content) exceeds 105,000 tokens, the system treats the request as a large-context operation. Instead of using the explicitly requested model or standard default, it queries ModelManager for the large_context model type (lines 23-29):

# From open_notebook/ai/provision.py

if token_count(content) > 105_000:
    # Automatically switch to large-context model

    model_id = await ModelManager.get_default_model("large_context")
else:
    # Use explicit model_id or standard default

    model_id = model_id or await ModelManager.get_default_model(default_type)

This hardcoded threshold of 105_000 tokens serves as the primary decision point for model routing.

ModelManager Default Resolution

The ModelManager.get_default_model method, implemented in open_notebook/ai/models.py, retrieves the appropriate model identifier from SurrealDB. When the model_type parameter equals "large_context", the function extracts the ID stored in the large_context_model field of the DefaultModels record (lines 46-48):

# From open_notebook/ai/models.py

elif model_type == "large_context":
    model_id = defaults.large_context_model

The system supports multiple default model categories including "chat", "large_context", "embedding", and "language", each configurable through the Settings → Models interface.

Complete Execution Flow

The model selection process follows this deterministic sequence:

1. Tokenization: Input content passes through token_count() using the o200k_base encoder.
2. Threshold Check: Compare token count against 105,000 token limit.
3. Model Resolution:
   • Exceeds threshold: Query ModelManager.get_default_model("large_context").
   • Below threshold: Use explicit model_id or ModelManager.get_default_model(default_type).
4. Instantiation: ModelManager.get_model() constructs the LangChain-compatible model object.

Practical Implementation Example

To leverage automatic model selection in your own implementations, use the provision_langchain_model helper:

from open_notebook.ai.provision import provision_langchain_model

async def process_document(text: str):
    # Automatically selects large-context model if text > 105k tokens

    model = await provision_langchain_model(
        content=text,
        model_id=None,        # No explicit model override

        default_type="chat",  # Standard chat model for normal requests

    )
    
    # Model is ready for LangChain operations

    response = await model.ainvoke({"messages": [{"role": "user", "content": text}]})
    return response

If the input text exceeds 105,000 tokens, the function returns the configured large-context model (e.g., Claude-2-100k or Llama-2-70B-Chat-Extended). Otherwise, it returns the standard chat model configured in the system defaults.

Summary

• Token Measurement: The token_count function in open_notebook/utils/token_utils.py uses tiktoken with the o200k_base encoder to calculate exact token counts.
• 105K Threshold: Prompts exceeding 105,000 tokens automatically trigger large-context model selection in open_notebook/ai/provision.py.
• Dynamic Routing: ModelManager.get_default_model in open_notebook/ai/models.py retrieves the appropriate model ID from SurrealDB based on the context size classification.
• Transparent Fallback: The system requires no manual intervention; oversized prompts transparently route to capable models while standard requests use default configurations.

Frequently Asked Questions

What is the exact token threshold for large-context model selection?

Open Notebook uses a hardcoded threshold of 105,000 tokens (represented as 105_000 in the source code). Any prompt exceeding this count in open_notebook/ai/provision.py automatically triggers a fallback to the large-context model configured in the system settings.

Which tokenizer does Open Notebook use for counting tokens?

The system uses the o200k_base encoder from the tiktoken library, as implemented in open_notebook/utils/token_utils.py. This encoder aligns with modern OpenAI model specifications and provides accurate token counts for routing decisions.

Where are the large-context model defaults stored?

Default model configurations reside in SurrealDB within the DefaultModels record. The large_context_model field specifically stores the model identifier for high-capacity contexts, accessible via ModelManager.get_default_model("large_context") in open_notebook/ai/models.py.

How does ModelManager handle requests under the token threshold?

For requests below 105,000 tokens, provision_langchain_model first checks for an explicitly provided model_id. If none is supplied, it calls ModelManager.get_default_model(default_type) using the requested type parameter (typically "chat"), returning the standard model configured for that category.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}