# How ModelManager Selects AI Models Based on Token Context Size in Open Notebook

> Learn how Open Notebook's ModelManager selects AI models by analyzing token context size with tiktoken. Discover automatic routing for large prompts.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-14

---

**Open Notebook automatically routes prompts exceeding 105,000 tokens to a dedicated large-context model by counting tokens with tiktoken and querying ModelManager for the appropriate default based on size thresholds.**

The `lfnovo/open-notebook` repository implements an intelligent model selection system that transparently handles token context limits. When processing AI requests, the application measures prompt size and automatically falls back to models configured for large contexts when thresholds are exceeded. This prevents truncation errors and ensures seamless handling of extensive documents.

## Token Counting with tiktoken

The system begins by calculating the exact token count of input content using the `token_count` utility function.

Located in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py), this helper leverages the `tiktoken` library with the **o200k_base** encoder to compute precise token counts:

```python
from open_notebook.utils.token_utils import token_count

tokens = token_count(content)  # Returns integer token count

```

The `o200k_base` encoder provides accurate tokenization aligned with modern OpenAI model specifications, ensuring consistent measurements across different input types.

## The 105K Token Threshold Logic

The core selection logic resides in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) within the async function `provision_langchain_model`. This function evaluates three parameters: the raw `content`, an optional `model_id`, and a `default_type` for fallback scenarios.

When `token_count(content)` exceeds **105,000 tokens**, the system treats the request as a large-context operation. Instead of using the explicitly requested model or standard default, it queries ModelManager for the `large_context` model type (lines 23-29):

```python

# From open_notebook/ai/provision.py

if token_count(content) > 105_000:
    # Automatically switch to large-context model

    model_id = await ModelManager.get_default_model("large_context")
else:
    # Use explicit model_id or standard default

    model_id = model_id or await ModelManager.get_default_model(default_type)

```

This hardcoded threshold of `105_000` tokens serves as the primary decision point for model routing.

## ModelManager Default Resolution

The `ModelManager.get_default_model` method, implemented in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py), retrieves the appropriate model identifier from SurrealDB. When the `model_type` parameter equals `"large_context"`, the function extracts the ID stored in the `large_context_model` field of the DefaultModels record (lines 46-48):

```python

# From open_notebook/ai/models.py

elif model_type == "large_context":
    model_id = defaults.large_context_model

```

The system supports multiple default model categories including `"chat"`, `"large_context"`, `"embedding"`, and `"language"`, each configurable through the Settings → Models interface.

## Complete Execution Flow

The model selection process follows this deterministic sequence:

1. **Tokenization**: Input content passes through `token_count()` using the `o200k_base` encoder.
2. **Threshold Check**: Compare token count against 105,000 token limit.
3. **Model Resolution**: 
   - **Exceeds threshold**: Query `ModelManager.get_default_model("large_context")`.
   - **Below threshold**: Use explicit `model_id` or `ModelManager.get_default_model(default_type)`.
4. **Instantiation**: `ModelManager.get_model()` constructs the LangChain-compatible model object.

## Practical Implementation Example

To leverage automatic model selection in your own implementations, use the `provision_langchain_model` helper:

```python
from open_notebook.ai.provision import provision_langchain_model

async def process_document(text: str):
    # Automatically selects large-context model if text > 105k tokens

    model = await provision_langchain_model(
        content=text,
        model_id=None,        # No explicit model override

        default_type="chat",  # Standard chat model for normal requests

    )
    
    # Model is ready for LangChain operations

    response = await model.ainvoke({"messages": [{"role": "user", "content": text}]})
    return response

```

If the input text exceeds 105,000 tokens, the function returns the configured large-context model (e.g., Claude-2-100k or Llama-2-70B-Chat-Extended). Otherwise, it returns the standard chat model configured in the system defaults.

## Summary

- **Token Measurement**: The `token_count` function in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) uses `tiktoken` with the `o200k_base` encoder to calculate exact token counts.
- **105K Threshold**: Prompts exceeding 105,000 tokens automatically trigger large-context model selection in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py).
- **Dynamic Routing**: `ModelManager.get_default_model` in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py) retrieves the appropriate model ID from SurrealDB based on the context size classification.
- **Transparent Fallback**: The system requires no manual intervention; oversized prompts transparently route to capable models while standard requests use default configurations.

## Frequently Asked Questions

### What is the exact token threshold for large-context model selection?

Open Notebook uses a hardcoded threshold of **105,000 tokens** (represented as `105_000` in the source code). Any prompt exceeding this count in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) automatically triggers a fallback to the large-context model configured in the system settings.

### Which tokenizer does Open Notebook use for counting tokens?

The system uses the **o200k_base** encoder from the `tiktoken` library, as implemented in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py). This encoder aligns with modern OpenAI model specifications and provides accurate token counts for routing decisions.

### Where are the large-context model defaults stored?

Default model configurations reside in **SurrealDB** within the `DefaultModels` record. The `large_context_model` field specifically stores the model identifier for high-capacity contexts, accessible via `ModelManager.get_default_model("large_context")` in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py).

### How does ModelManager handle requests under the token threshold?

For requests below 105,000 tokens, `provision_langchain_model` first checks for an explicitly provided `model_id`. If none is supplied, it calls `ModelManager.get_default_model(default_type)` using the requested type parameter (typically `"chat"`), returning the standard model configured for that category.