# How Open Notebook's ModelManager Implements Large Context Fallback

> Discover how Open Notebook's ModelManager implements large context fallback, automatically routing requests over 105k tokens to a dedicated model for efficient handling.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-07-06

---

**Open Notebook's ModelManager automatically routes requests exceeding 105,000 tokens to a dedicated large-context model by detecting token counts in `provision_langchain_model` and retrieving the configured `large_context_model` from `DefaultModels`.**

The `lfnovo/open-notebook` repository implements an intelligent fallback system for handling oversized content through its **ModelManager** class. When processing content that exceeds standard token limits, the system transparently switches to a specialized large-context model without requiring manual intervention. This mechanism ensures that long documents are processed by models capable of handling extended contexts while maintaining a seamless developer experience.

## How Token Detection Triggers the Fallback

### The 105,000 Token Threshold

In [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py), the `provision_langchain_model` function performs the initial detection. Lines 19-25 tokenize the input content and compare it against the hard-coded limit:

```python
tokens = token_count(content)
if tokens > 105_000:
    # Trigger large context fallback

```

When this threshold is exceeded, the function ignores the standard `default_type` parameter and instead requests the large-context variant.

## The Three-Step Fallback Mechanism

### Step 1: Detecting Oversized Content

The detection occurs within `provision_langchain_model` before any model instantiation. The function calculates the token count using `token_count(content)` and validates it against the `105_000` limit. If the content exceeds this boundary, the flow immediately diverts from the standard chat model path.

### Step 2: Requesting the Large Context Default

Upon detecting large content, the provisioner calls `model_manager.get_default_model` with the specific `"large_context"` type parameter (line 28 of [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py)):

```python
await model_manager.get_default_model("large_context", **kwargs)

```

This call signals the ModelManager to retrieve the dedicated large-context configuration rather than the standard default.

### Step 3: Resolving the Model Configuration

In [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py), the `ModelManager.get_default_model` method (lines 241-250) queries the `DefaultModels` singleton record to extract the `large_context_model` field. If configured, it passes this model ID to `ModelManager.get_model` (lines 102-156), which:

1. Loads the `Model` record from the database
2. Validates that `type == "language"`
3. Resolves credentials or falls back to environment variables via `provision_provider_keys`
4. Creates the Esperanto language model using `AIFactory.create_language`

## Implementation Code Examples

### Automatic Fallback During Provisioning

To leverage the automatic fallback, use the `provision_langchain_model` function:

```python
from open_notebook.ai.provision import provision_langchain_model

async def get_chat_model(text: str):
    # Automatically switches to large_context model if text > 105,000 tokens

    chat_model = await provision_langchain_model(
        content=text,
        model_id=None,          # no explicit override

        default_type="chat",   # normal chat flow

        temperature=0.7,
    )
    return chat_model

```

### Direct Large Context Model Access

For explicit large-context model retrieval:

```python
from open_notebook.ai.models import model_manager

async def fetch_large_context_model():
    # Returns an Esperanto LanguageModel or None if not configured

    return await model_manager.get_default_model("large_context")

```

### Credential Fallback Chain

When the large-context model lacks configured credentials, the system falls back to environment variables (from [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py), lines 120-136):

```python
if model.credential:
    credential = await model.get_credential_obj()
    if credential:
        config = credential.to_esperanto_config()
    else:
        # Credential cannot be loaded → env-var fallback

        await provision_provider_keys(model.provider)
else:
    # No credential linked at all → env-var fallback

    await provision_provider_keys(model.provider)

```

## Handling Missing Configurations

If the `large_context_model` field in `DefaultModels` is empty, `ModelManager.get_default_model` logs a warning (lines 50-53) and returns `None`. This triggers a clear error path that prompts users to configure the large-context model through the Settings → Models UI, preventing silent failures when processing oversized content.

## Summary

- **Automatic Detection**: The system detects content exceeding 105,000 tokens in `provision_langchain_model` before model instantiation.
- **Transparent Routing**: Oversized requests are automatically routed to the `large_context_model` configured in `DefaultModels`.
- **Credential Resilience**: Missing credentials trigger a fallback to environment variables via `provision_provider_keys`, ensuring connectivity even without stored credentials.
- **Clear Error Handling**: Unconfigured large-context models generate explicit warnings rather than silent defaults, directing users to the configuration interface.

## Frequently Asked Questions

### What happens if no large-context model is configured?

If the `large_context_model` field in `DefaultModels` is unset, `ModelManager.get_default_model` logs a warning and returns `None`. The application will raise an error indicating that the large-context model must be configured in Settings → Models before processing oversized content.

### Why is the token limit set to 105,000?

The 105,000 token threshold represents a hard-coded safety boundary in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) that distinguishes standard context windows from extended capabilities. This limit ensures that content approaching typical model constraints (usually 128k tokens) is handled by models specifically designated for large context processing.

### Can I override the large-context model selection manually?

Yes. By passing a specific `model_id` parameter to `provision_langchain_model`, you bypass the automatic fallback logic entirely. When `model_id` is provided, the function skips token counting and default model resolution, using your specified model regardless of content size.

### How does the system handle API keys for the fallback model?

The ModelManager first attempts to load credentials from the database record associated with the large-context model. If credentials are missing or fail to load, it automatically calls `provision_provider_keys` from [`open_notebook/ai/key_provider.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/key_provider.py) to retrieve API keys from environment variables, ensuring the fallback model can authenticate even without explicit credential storage.