How ModelManager Selects the Best AI Model Based on Context Size in Open Notebook

Open Notebook's ModelManager automatically routes inputs exceeding 105,000 tokens to a specialized large-context AI model while respecting explicit model overrides and standard defaults for smaller workloads.

The lfnovo/open-notebook repository implements an intelligent model selection pipeline that dynamically chooses the appropriate AI model based on input token count. The system relies on a hard-coded threshold and hierarchical decision logic to prevent context window overflow and ensure optimal performance across varying content sizes.

Token Counting and Threshold Detection

The selection process begins with the token_count helper function in open_notebook/utils.py, which calculates the exact number of tokens in the input content. This count drives the decision logic located in open_notebook/ai/provision.py (lines 10-35).

The system uses a hard-coded threshold of 105,000 tokens. When content exceeds this limit, the system categorizes the request as requiring "large_context" handling. This threshold represents the boundary where standard models risk context window exhaustion, automatically triggering the promotion to specialized high-capacity models configured for massive inputs.

The Selection Logic in provision_langchain_model

The provision_langchain_model function implements a three-tier decision hierarchy that determines which model configuration to instantiate:

1. Large Context Override: If tokens > 105_000, the function logs "large_context" as the selection_reason and calls await model_manager.get_default_model("large_context", **kwargs)
2. Explicit Model ID: If the caller provides a specific model_id, the system bypasses automatic selection entirely and uses await model_manager.get_model(model_id, **kwargs)
3. Standard Default: For normal token counts without explicit overrides, the system uses await model_manager.get_default_model(default_type, **kwargs) where default_type typically equals "chat" or "embedding"

The selection_reason variable captures the rationale for debugging purposes, while the actual model instantiation occurs through the ModelManager's async methods.

Resolving Large Context Models via ModelManager

When handling large context requests, ModelManager.get_default_model (defined in open_notebook/ai/models.py, lines 221-247) queries the DefaultModels record to retrieve the configured large-context model ID.

The system checks the large_context_model field defined in the DefaultModels dataclass (lines 62-68). If configured, the manager fetches the corresponding Model record, constructs the Esperanto configuration, and returns a LanguageModel instance ready for LangChain integration via the to_langchain() method.

If no large_context_model is configured in Settings, get_default_model returns None, causing provision_langchain_model to raise a ConfigurationError directing users to configure the large context model in the Settings UI.

Practical Implementation Examples

The following examples demonstrate how to leverage the automatic selection system in application code:

from open_notebook.ai.provision import provision_langchain_model

async def process_document(text: str):
    # Automatically selects large_context model if text > 105k tokens

    model = await provision_langchain_model(
        content=text,
        model_id=None,
        default_type="chat",
        temperature=0.7,
    )
    # Returns a LangChain BaseChatModel ready for .invoke()

    return model

For direct access to the large-context model without token counting:

from open_notebook.ai.models import model_manager

async def get_high_capacity_model():
    model = await model_manager.get_default_model("large_context")
    if model is None:
        raise RuntimeError("Configure large_context_model in Settings")
    return model.to_langchain()

To bypass automatic selection and force a specific model regardless of token count:

# Explicit model_id overrides the 105k threshold check

model = await provision_langchain_model(
    content=large_text,
    model_id="open_notebook:model:1234",
    default_type="chat"
)

Summary

• Token Threshold: Inputs exceeding 105,000 tokens automatically trigger the large-context branch in open_notebook/ai/provision.py
• Hierarchical Selection: The system prioritizes explicit model IDs, then large-context detection, then standard defaults
• Configuration Dependency: Large-context handling requires setting large_context_model in the DefaultModels record via the Settings UI
• LangChain Integration: All paths return LangChain-compatible models through the to_langchain() method
• Debug Visibility: The selection_reason variable logs the rationale for model selection to aid troubleshooting

Frequently Asked Questions

What happens if no large-context model is configured?

If DefaultModels.large_context_model is unset when content exceeds 105,000 tokens, ModelManager.get_default_model returns None, causing provision_langchain_model to raise a ConfigurationError with instructions to configure the model in Settings.

Can I force a specific model even for large inputs?

Yes. Providing an explicit model_id parameter to provision_langchain_model bypasses the automatic token-count check entirely, allowing any configured model to handle the content regardless of size.

Where is the 105,000 token threshold defined?

The threshold is hard-coded in open_notebook/ai/provision.py within the provision_langchain_model function (lines 10-35) as the constant 105_000.

How does ModelManager instantiate the selected model?

ModelManager.get_default_model retrieves the model ID from DefaultModels, fetches the corresponding database record, builds an Esperanto configuration object, and wraps it as a LangChain-compatible model through the to_langchain() method.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}