# How Open-Notebook Handles Token Estimation and Large Context Model Selection

> Discover Open Notebooks token estimation using tiktoken and its automatic large context model selection for prompts over 105k tokens. Process long documents reliably.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-19

---

**Open-Notebook uses tiktoken's o200k_base encoder to precisely count tokens and automatically routes prompts exceeding 105,000 tokens to a large-context model family, ensuring reliable processing of very long documents.**

The **lfnovo/open-notebook** repository implements a robust strategy for token estimation and large context model selection that prevents context window overflows while maintaining flexibility for different deployment environments. By combining accurate token counting with intelligent model routing, the system can safely process documents that far exceed standard LLM context limits.

## Precise Token Counting with tiktoken

The foundation of Open-Notebook's token estimation strategy relies on the **`tiktoken`** library to deliver accurate token counts before sending requests to language models.

### The token_count Utility

In [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py), the **`token_count`** function uses the **`o200k_base`** encoding—the same 200,000-token-capacity tokenizer employed by many modern providers. This encoder returns the exact number of tokens for any input string, allowing the system to make precise capacity decisions.

```python
from open_notebook.utils.token_utils import token_count

text = "Open‑Notebook can process very long documents."
num_tokens = token_count(text)
print(f"The text contains {num_tokens} tokens")

```

### Fallback for Offline Environments

When `tiktoken` cannot be imported—such as in offline or restricted environments—the utility gracefully falls back to a simple word-count heuristic. While this fallback is less precise, it ensures the application remains functional even without access to the full tokenizer library.

## Automatic Large Context Model Selection

Once token counts are established, Open-Notebook applies a clear decision hierarchy to select an appropriate model capable of handling the payload.

### The 105,000 Token Threshold

The system defines a hard threshold at **105,000 tokens** to determine when a large-context model is required. This value sits safely below the 200,000-token ceiling of the `o200k_base` tokenizer while providing sufficient headroom above standard 8,000–32,000 token limits. Prompts exceeding this threshold automatically trigger selection from the **"large_context"** model family.

### Model Resolution Logic in provision_langchain_model

The **`provision_langchain_model`** function in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) implements the selection logic:

```text
if tokens > 105_000:
    # Choose a model registered under the “large_context” family

elif model_id is supplied:
    # Use the explicitly requested model

else:
    # Use the default model for the requested type (e.g. “chat”, “completion”)

```

When the token count exceeds the threshold, the function calls `model_manager.get_default_model("large_context")` to retrieve a model with a 128,000 or 200,000-token window. If no model ID is specified and the content falls below the threshold, the system defaults to standard models configured for the requested operation type.

```python
from open_notebook.ai.provision import provision_langchain_model

async def get_model_for_content(content: str):
    # No explicit model_id, default type is "chat"

    model = await provision_langchain_model(
        content=content,
        model_id=None,
        default_type="chat",
    )
    return model

```

You can manually specify a model ID, but the large-context override takes precedence when content exceeds 105,000 tokens:

```python
model = await provision_langchain_model(
    content=very_long_text,
    model_id="gpt-4-32k",      # optional explicit ID

    default_type="chat",
)

# If `very_long_text` exceeds 105_000 tokens, the large-context default

# model will be selected regardless of the explicit ID.

```

## Integration Across the Codebase

The token estimation and model selection strategy integrates with multiple components to enforce limits throughout the application.

### Context Budgeting in context_builder.py

The **[`context_builder.py`](https://github.com/lfnovo/open-notebook/blob/main/context_builder.py)** module in `open_notebook/utils/` utilizes `token_count` to enforce maximum token budgets when constructing request payloads. This prevents the system from accidentally building prompts that exceed model capacity before the provisioning stage even begins.

### Model Family Registration

The **`ModelManager`** class in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py) maintains the mapping between model families like **"large_context"** and concrete LLM provider implementations. When `provision_langchain_model` requests a large-context default, this manager resolves the configuration to an actual model instance capable of processing the extended input.

If no suitable model can be resolved—such as when no default large-context model is configured—the system raises a **`ConfigurationError`** with a clear message directing users to configure a default in the Settings → Models UI.

## Summary

- **Open-Notebook** uses `tiktoken` with the `o200k_base` encoder in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) for precise token estimation, with a word-count fallback for offline environments.
- The **105,000 token threshold** in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) triggers automatic selection of the **"large_context"** model family to handle very long documents.
- The **`provision_langchain_model`** function applies a hierarchical decision tree: large-context models for oversized content, explicit model IDs when provided, or defaults for standard sizes.
- **ConfigurationError** handling ensures users receive clear guidance when no suitable model is configured for the required context window.

## Frequently Asked Questions

### What tokenizer does Open-Notebook use for token estimation?

According to the source code in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py), Open-Notebook uses the **`o200k_base`** encoding from the `tiktoken` library. This encoding supports up to 200,000 tokens and matches the tokenizer used by many modern providers, ensuring accurate counts across different model families.

### Why is the threshold set at 105,000 tokens?

The 105,000-token threshold provides a safety margin below the 200,000-token capacity of the `o200k_base` tokenizer while remaining well above standard model limits of 8,000–32,000 tokens. This ensures that content exceeding typical limits gets routed to models explicitly registered with 128,000 or 200,000-token windows, preventing context overflow errors.

### What happens if no large-context model is configured?

If `provision_langchain_model` cannot resolve a model—whether because no large-context default is set or the specified model ID is unavailable—the system raises a **`ConfigurationError`**. The error message directs users to the Settings → Models UI to configure a default model for the required family.

### Can I force a specific model even for very long contexts?

You can supply an explicit `model_id` parameter to `provision_langchain_model`, but the large-context override logic takes precedence when content exceeds 105,000 tokens. If you need to force a specific model for very long content, you must ensure it is registered as the default **"large_context"** model in the ModelManager, or ensure your content stays below the 105,000-token threshold.