How Context Window Size Affects Model Selection in Open Notebook

Open Notebook automatically routes requests to a dedicated large-context model when input exceeds 105,000 tokens, while standard requests use the default chat or transformation model configured in the system.

When building AI applications that process varying document lengths, managing context window limitations is critical for both performance and cost. In the lfnovo/open-notebook repository, the framework implements an intelligent provisioning system that dynamically selects the appropriate language model based on the size of the incoming content. This mechanism ensures that normal conversations use cheaper, standard models while automatically upgrading to expensive, high-capacity models only when processing very long documents.

The 105,000 Token Threshold

The decision point for context-window sizing is hard-coded at 105,000 tokens in open_notebook/ai/provision.py. This threshold acts as a gatekeeper between standard and large-context processing paths.

When the provision_langchain_model function receives content, it immediately calculates the token count:

tokens = token_count(content)  # line 19

If this value exceeds 105,000, the system logs the selection reason and triggers the large-context branch:

if tokens > 105_000:  # lines 23-28

    selection_reason = f"large_context (content has {tokens} tokens)"
    logger.debug(...)
    model = await model_manager.get_default_model("large_context", **kwargs)

Requests below this threshold bypass the large-context logic and proceed to standard model selection paths.

Token Counting Mechanism

Before model selection occurs, the framework measures the payload using the token_count helper imported from open_notebook/utils/__init__.py. This utility analyzes the raw content string and returns an integer representing the estimated token count.

The provisioning function stores this count and uses it as the primary determinant for the subsequent routing logic. According to the source analysis, this counting happens at line 19 of provision.py, making it the first operation in the model selection pipeline.

Model Selection Logic in provision.py

The provision_langchain_model function implements a three-tiered decision tree in open_notebook/ai/provision.py (lines 10-38):

1. Large-context branch: Activated when tokens > 105_000, fetching the model configured for "large_context" type
2. Explicit override branch: Activated when a model_id parameter is provided, bypassing token checks entirely
3. Default type branch: Fallback for standard requests, using the default_type parameter (e.g., "chat" or "transformation")

This structure ensures that token size is evaluated before any other selection criteria, preventing context window overflow errors that would occur if a standard model attempted to process 105,000+ tokens.

Configuring the Large-Context Default

The identity of the large-context model is stored in the DefaultModels dataclass defined in open_notebook/ai/models.py. Specifically, line 66 defines the optional field:

large_context_model: Optional[str] = None

When the token threshold is exceeded, ModelManager.get_default_model (lines 46-48) reads this field and retrieves the corresponding model configuration. If large_context_model is not set in the database, the manager returns None, which subsequently triggers a ConfigurationError during the provisioning flow.

Administrators can configure this default through the API layer exposed in api/routers/models.py or programmatically via the model_manager interface.

Explicit Model Overrides

Users and developers can bypass the automatic context-window logic entirely by supplying a model_id parameter. When provided, the provisioning function skips the token count check and loads the specified model directly:

elif model_id:  # lines 29-32

    selection_reason = f"explicit model_id={model_id}"
    model = await model_manager.get_model(model_id, **kwargs)

This override mechanism is useful for testing specific models or forcing smaller, cheaper models for cost control even when processing long documents.

Code Examples

Automatic Selection Based on Content Size

The following example demonstrates how Open Notebook automatically handles varying document lengths without manual intervention:

from open_notebook.ai.provision import provision_langchain_model

async def process_document(content: str):
    # Let Open Notebook decide based on token count

    llm = await provision_langchain_model(
        content=content,
        model_id=None,
        default_type="chat",
        temperature=0.7,
    )
    # If content > 105,000 tokens, uses large_context_model

    # Otherwise uses the default chat model

    response = await llm.ainvoke({"messages": [{"role": "user", "content": content}]})
    return response

Forcing a Specific Model

To override the automatic selection and force a specific model regardless of document length:

llm = await provision_langchain_model(
    content=very_long_text,
    model_id="open_notebook:model:anthropic:claude-2",
    default_type="chat",
)

Configuring the Large-Context Default

Administrators can set the large-context model programmatically:

from open_notebook.ai.models import model_manager, DefaultModels

async def configure_large_context():
    defaults = await model_manager.get_defaults()
    defaults.large_context_model = "open_notebook:model:openai:gpt-4-32k"
    await defaults.save()

After execution, any request exceeding 105,000 tokens will automatically route to gpt-4-32k or the configured equivalent.

Summary

• Open Notebook uses a hard-coded threshold of 105,000 tokens in open_notebook/ai/provision.py to determine when to switch to large-context models.
• The token_count function from open_notebook/utils/__init__.py measures input size before model selection occurs.
• When the threshold is exceeded, the system retrieves the model ID stored in DefaultModels.large_context_model via ModelManager.get_default_model.
• Providing an explicit model_id parameter bypasses the token check and uses the specified model regardless of content size.
• If no large-context model is configured, the system emits a warning and raises a ConfigurationError for oversized requests.

Frequently Asked Questions

What happens if no large-context model is configured?

If the token count exceeds 105,000 but DefaultModels.large_context_model is None, ModelManager.get_default_model returns None, causing provision_langchain_model to raise a ConfigurationError. Administrators must configure the large-context default in open_notebook/ai/models.py before processing very long documents.

Can I force a specific model for a request regardless of token count?

Yes. By passing the model_id parameter to provision_langchain_model, you bypass the token threshold check entirely. The function will use ModelManager.get_model to load the specified model directly, skipping the 105,000 token evaluation logic.

Where is the 105,000 token threshold defined?

The threshold is hard-coded at line 23 of open_notebook/ai/provision.py as the integer literal 105_000. This value is not configurable at runtime and requires modifying the source code to change.

How does Open Notebook count tokens?

The framework uses the token_count utility imported from open_notebook/utils/__init__.py. This helper analyzes the content string and returns the token count, which provision_langchain_model then compares against the 105,000 token limit to determine the appropriate model selection path.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}