How Open Notebook Automatically Switches to Large Context AI Models

Open Notebook automatically routes content exceeding 105,000 tokens to a dedicated large-context model by comparing token counts in the provisioning layer and requesting the large_context default type from the model manager.

Open Notebook is an open-source note-taking application that intelligently handles extensive documents by automatically switching to large context AI models when inputs exceed specific thresholds. This mechanism ensures that processing of very long texts remains seamless without requiring manual model selection by the user. The implementation spans token utilities, model provisioning logic, and default model configuration across the lfnovo/open-notebook repository.

Token Counting and Threshold Detection

The automatic switching mechanism begins with token counting in open_notebook/utils/token_utils.py. The token_count() function processes raw content using the o200k_base tiktoken encoding to calculate the exact number of tokens. If tiktoken is unavailable, it falls back to a word-count estimate.

This token count is then compared against a hard-coded threshold of 105,000 tokens. When the input exceeds this limit, the system recognizes that standard context windows may be insufficient and triggers the large-context model selection pathway.

The Model Provision Logic

The core switching logic resides in open_notebook/ai/provision.py inside the provision_langchain_model() function. This async function evaluates the token count before initializing any model:

• If tokens > 105_000: The function logs the decision and calls model_manager.get_default_model("large_context") to retrieve the appropriate high-capacity model.
• Otherwise: The function respects an explicitly requested model_id or falls back to the default model for the requested type (chat, embedding, etc.).

This branching ensures that large inputs automatically receive a model capable of processing them, while normal inputs use standard, cost-effective models.

Default Model Resolution

When a large-context model is requested, ModelManager.get_default_model() in open_notebook/ai/models.py queries the DefaultModels record stored in the database. Specifically, it reads the large_context_model field, which users populate through the Settings → Models UI (labeled as "Large Context Model").

If a valid model ID exists in this field, the system fetches the corresponding Model record and returns a fully-initialized LanguageModel instance via Esperanto. The model is then converted to a LangChain-compatible object using model.to_langchain() and handed to the downstream LangGraph workflow for processing.

Practical Implementation Examples

Automatic Model Selection in LangGraph Nodes

When building custom nodes, you can rely on the automatic switching by calling provision_langchain_model():

from open_notebook.ai.provision import provision_langchain_model

async def chat_node(content: str, model_id: str | None = None):
    # Automatically picks a large-context model if content exceeds 105k tokens

    llm = await provision_langchain_model(
        content=content,
        model_id=model_id,
        default_type="chat",
        temperature=0.7,
    )
    response = await llm.ainvoke({"messages": [{"role": "user", "content": content}]})
    return response

Manual Token Count Verification

To check whether content will trigger the large-context switch:

from open_notebook.utils.token_utils import token_count

text = "..."  # Your large document

count = token_count(text)
print(f"Token count: {count}")

# If count > 105000, the provision logic will switch to the large-context model

Configuring the Large Context Model via API

You can set the default large-context model programmatically:

curl -X PATCH http://localhost:5055/api/models/defaults \
  -H "Content-Type: application/json" \
  -d '{"large_context_model": "gemini-1.5-flash"}'

Summary

• Token counting uses o200k_base encoding in open_notebook/utils/token_utils.py to calculate input size.
• Threshold detection occurs at 105,000 tokens within provision_langchain_model() in open_notebook/ai/provision.py.
• Automatic switching requests the large_context model type from ModelManager.get_default_model() when thresholds are exceeded.
• Configuration happens via the large_context_model field in the DefaultModels database record, set through the Settings UI.
• Transparency ensures users never manually select models; the system handles extensive documents seamlessly.

Frequently Asked Questions

What is the exact token threshold for switching to a large context model?

Open Notebook uses a hard-coded threshold of 105,000 tokens. Any input exceeding this count automatically triggers the large-context model selection in provision_langchain_model().

How does Open Notebook count tokens when processing content?

The system uses the token_count() function from open_notebook/utils/token_utils.py, which utilizes the o200k_base tiktoken encoding for accurate OpenAI-compatible token counting. If tiktoken is unavailable, it falls back to a word-count estimation method.

Can users configure which model is used for large context processing?

Yes. Users configure the large-context model through the Settings → Models UI by setting the "Large Context Model" field, which corresponds to the large_context_model column in the DefaultModels database record. This can also be updated via the REST API at /api/models/defaults.

Is the large context model switch transparent to the end user?

Yes. The switch is completely automatic and transparent. Users only need to ensure a suitable large-context model (such as Gemini 1.5 Flash or Claude 3 Opus) is configured in the Settings page. The provisioning layer handles all detection and model initialization without requiring manual intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}