How ModelManager Selects Models Based on Context Size: The 105,000 Token Threshold Explained

The ModelManager in Open Notebook automatically selects a large-context model when content exceeds 105,000 tokens, using the provision_langchain_model function in open_notebook/ai/provision.py to check token counts and route requests accordingly.

The Open Notebook project (lfnovo/open-notebook) handles large document processing by dynamically switching between standard and large-context language models based on content length. Understanding how the ModelManager selects models based on context size is essential for optimizing AI operations when working with extensive texts. This mechanism relies on a specific 105,000 token threshold defined in the provisioning layer to determine when to escalate to models with larger context windows.

The Two-Step Selection Process

The Open Notebook backend decides which language model to use through a coordinated two-step process involving token counting and conditional model provisioning.

Step 1: Token Counting with tiktoken

First, the system calculates the token count of incoming content using the open_notebook.utils.token_count helper. This utility, located in open_notebook/utils/token_utils.py, employs the tiktoken library with the "o200k_base" encoder to return an integer representing the total tokens in the supplied content.

Step 2: The 105,000 Token Threshold

Next, the provision_langchain_model function in open_notebook/ai/provision.py evaluates the token count against the threshold:

tokens = token_count(content)

if tokens > 105_000:                     # <-- 105k-token threshold

    # use the "large-context" model configured in the defaults

    model = await model_manager.get_default_model("large_context", **kwargs)
elif model_id:
    # an explicit model was requested in the UI/config

    model = await model_manager.get_model(model_id, **kwargs)
else:
    # fall back to the default for the requested type (chat, embedding, …)

    model = await model_manager.get_default_model(default_type, **kwargs)

When the token count exceeds 105,000, the function logs the decision (large_context (content has {tokens} tokens)) and requests the large-context default model from the ModelManager. If the count falls below this threshold, the function either respects an explicit model_id or returns the default model for the requested default_type.

Inside the ModelManager Facade

The ModelManager class, defined in open_notebook/ai/models.py, serves as a thin façade over the model retrieval system. According to the source code, it retrieves model records from SurrealDB, resolves linked credentials, and constructs the appropriate Esperanto factory call using methods like AIFactory.create_language or create_embedding.

Importantly, the large-context decision logic resides entirely within the provision_langchain_model function, not inside the ModelManager itself. The ModelManager simply executes the retrieval when asked for the large_context default via get_default_model("large_context", **kwargs).

Implementation Examples

Automatic Large-Context Selection

When processing documents exceeding the threshold, the system automatically selects the appropriate model:

# Example: provisioning a chat model for a 120k-token document

from open_notebook.ai.provision import provision_langchain_model

content = "…" * 150_000  # a very long string

model = await provision_langchain_model(
    content=content,
    model_id=None,            # no explicit model chosen by the user

    default_type="chat",      # we want a chat-type model

)

# → the function returns the large-context language model

Bypassing the Threshold with Explicit Model IDs

You can force a specific model regardless of content size by providing a model_id:

# Example: forcing a specific model, bypassing the token-threshold logic

model = await provision_langchain_model(
    content="short text",
    model_id="open_notebook:model:anthropic Claude-3.5-sonnet",  # explicit ID

    default_type="chat",
)

# → the explicit model is used even if the content were > 105k tokens

Direct ModelManager Usage

For advanced use cases, interact with the ModelManager directly:

# Direct use of ModelManager (rarely needed outside provisioning)

from open_notebook.ai.models import ModelManager

mgr = ModelManager()
large_ctx = await mgr.get_default_model("large_context")

# large_ctx is the LanguageModel instance configured as the large-context default

Summary

• Token counting occurs in open_notebook/utils/token_utils.py using the tiktoken "o200k_base" encoder.
• The 105,000 token threshold is hardcoded in open_notebook/ai/provision.py within the provision_langchain_model function.
• When content exceeds 105,000 tokens, the system automatically requests the large_context default model from the ModelManager.
• Explicit model IDs bypass the threshold logic entirely, allowing forced model selection regardless of content size.
• The ModelManager in open_notebook/ai/models.py handles model instantiation but does not contain the threshold logic—it receives instructions from the provisioning layer.

Frequently Asked Questions

What happens if my content is exactly 105,000 tokens?

Content with exactly 105,000 tokens uses the standard default model, not the large-context variant. The threshold check uses a strict greater-than comparison (tokens > 105_000), so only content exceeding 105,000 tokens triggers the large-context selection.

Can I force a specific model even if my content exceeds 105,000 tokens?

Yes. Providing an explicit model_id parameter to provision_langchain_model bypasses the token threshold logic entirely. The function checks for explicit IDs before evaluating the token count, ensuring your specified model handles the content regardless of size.

Where is the 105,000 token threshold configured?

The threshold is hardcoded as the integer literal 105_000 in the provision_langchain_model function within open_notebook/ai/provision.py. It is not defined as a configuration variable or environment setting in the current implementation.

Which tokenizer does Open Notebook use for counting tokens?

Open Notebook uses the tiktoken library with the "o200k_base" encoder, implemented in open_notebook/utils/token_utils.py. This provides accurate token counts compatible with OpenAI's model standards, ensuring reliable threshold calculations across different content types.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}