# How Model Overrides Work on a Per-Request Basis Using RunnableConfig in Open Notebook

> Discover how per-request model overrides work in Open Notebook using RunnableConfig. Effortlessly control your language models for individual requests.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-24

---

**Open Notebook’s LangGraph workflows accept a `model_id` inside the `configurable` dictionary of a `RunnableConfig` to override the default language model for individual requests, falling back to thread state or system defaults only when omitted.**

This mechanism allows callers to dynamically select which AI model processes a specific interaction without altering global configuration. When an API request arrives, the system extracts any per-request model preference and injects it into the **RunnableConfig** before invoking the graph, ensuring the selected model is provisioned for that invocation only.

## The Three-Tier Resolution Hierarchy

Open Notebook resolves which language model to use through a strict priority chain. This design ensures explicit per-request overrides take precedence over stored preferences while maintaining sensible defaults.

### 1. Explicit RunnableConfig Override

The highest priority source is the `configurable` dictionary passed within the **RunnableConfig** object at invocation time. Graph nodes check `config.get("configurable", {}).get("model_id")` first. If present, this value is sent directly to the provisioning layer.

### 2. Thread State Fallback

If the **RunnableConfig** lacks a `model_id`, the system checks `state.get("model_override")`. This allows a model selection to persist across turns in a conversation when set earlier in the thread lifecycle.

### 3. Default Model Provisioning

When neither the config nor the state specifies a model, the `provision_langchain_model` helper selects the system default based on the requested model type (e.g., large-context or standard chat).

## Implementation in the Source Code

### Chat Graph Node

In [`open_notebook/graphs/chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/chat.py), the `call_model_with_messages` node implements the resolution logic:

```python
model_id = config.get("configurable", {}).get("model_id") or state.get("model_override")

```

This line (lines 30-36 in the source) demonstrates the short-circuit evaluation that prioritizes the **RunnableConfig** over thread state. The resolved `model_id` is then passed to the provisioning helper.

### Source-Chat Graph Node

The source-chat workflow in [`open_notebook/graphs/source_chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source_chat.py) follows an identical pattern. Within `_call_model_with_source_context_inner`, the same configuration check occurs (lines 41-45), ensuring consistency across different graph types.

### Model Provisioning Helper

The `provision_langchain_model` function in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) receives the resolved identifier. When `model_id` is explicitly provided (lines 28-34), it invokes `model_manager.get_model(model_id, **kwargs)` to return a LangChain-compatible model instance. If the ID is invalid, it raises a configuration error before the node attempts generation.

### API Router Integration

The streaming endpoints in [`api/routers/source_chat.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/source_chat.py) construct the **RunnableConfig** explicitly. For example, when handling a request, the router builds:

```python
RunnableConfig(configurable={"thread_id": session_id, "model_id": model_override})

```

This pattern (lines 44-48) bridges the HTTP API layer with the graph, translating the JSON payload's `model_override` field into the configuration structure the nodes expect.

## Practical Code Examples

### Calling the Chat Endpoint with an Override

Send a per-request model selection via the REST API:

```python
import httpx

payload = {
    "message": "Explain the difference between GPT-4 and Claude-3.",
    "model_override": "gpt-4o-mini"  # Per-request model ID

}

resp = httpx.post(
    "http://localhost:5055/api/chat/sessions/12345/messages",
    json=payload,
)
print(resp.json())

```

The router converts the `model_override` field into a `RunnableConfig` with `{"model_id": "gpt-4o-mini"}`, which the chat graph node reads during execution.

### Direct Graph Invocation

Programmatically override the model when invoking the graph directly:

```python
from langchain_core.runnables import RunnableConfig
from open_notebook.graphs.chat import graph, ThreadState

state: ThreadState = {"messages": [], "model_override": None}
config = RunnableConfig(
    configurable={"model_id": "anthropic/claude-3-opus-20240229"}
)

result = graph.invoke(input=state, config=config)
print(result["messages"])

```

Here, the **RunnableConfig** explicitly sets the model, bypassing any value stored in `state["model_override"]`.

### Streaming Source-Chat with Override

For streaming responses, the override flows through the async generator:

```python
import asyncio
from api.routers.source_chat import stream_source_chat_response

async def demo():
    async for chunk in stream_source_chat_response(
        session_id="chat_session:abc",
        source_id="source:xyz",
        message="Summarize the key points.",
        model_override="gpt-4o-mini",  # Per-request override

    ):
        print(chunk)

asyncio.run(demo())

```

The `stream_source_chat_response` function forwards the override into the **RunnableConfig**, which the source-chat graph consumes exactly as the standard chat graph does.

## Summary

- **RunnableConfig** is the primary vehicle for per-request model overrides in Open Notebook, using the `configurable["model_id"]` key.
- Resolution follows a strict hierarchy: explicit config override → thread state fallback → system default.
- Both [`chat.py`](https://github.com/lfnovo/open-notebook/blob/main/chat.py) and [`source_chat.py`](https://github.com/lfnovo/open-notebook/blob/main/source_chat.py) graph nodes implement identical logic to check `config.get("configurable", {}).get("model_id")` before falling back to state.
- The `provision_langchain_model` helper in [`ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/ai/provision.py) instantiates the specific model instance based on the resolved identifier.
- API routers bridge HTTP requests to graph invocations by constructing **RunnableConfig** objects that carry the `model_override` from the request payload.

## Frequently Asked Questions

### What happens if I provide both a RunnableConfig model_id and a state model_override?

The **RunnableConfig** value takes precedence. In [`open_notebook/graphs/chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/chat.py), the node uses `config.get("configurable", {}).get("model_id") or state.get("model_override")`, meaning the explicit config is evaluated first and the state is only checked if the config value is absent or evaluates to False.

### Can I use any model ID available in the model manager?

Yes, provided the ID exists in the model manager's registry. The `provision_langchain_model` function calls `model_manager.get_model(model_id, **kwargs)`, which will raise a configuration error if the ID is unrecognized. Valid IDs depend on your Open Notebook installation's configured providers (e.g., OpenAI, Anthropic, or local models).

### Does the model override affect the entire thread or just the current request?

The override affects only the current request invocation. However, if your application logic saves the `model_id` back into `state["model_override"]` after processing, subsequent turns in the same thread will inherit that selection via the state fallback mechanism until a new **RunnableConfig** override is provided or the state is cleared.

### Is the RunnableConfig approach specific to certain graph types?

No, this is a consistent pattern across Open Notebook's LangGraph implementations. Both the standard chat graph ([`open_notebook/graphs/chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/chat.py)) and the source-chat graph ([`open_notebook/graphs/source_chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source_chat.py)) use identical configuration checking logic, as do their respective API routers, ensuring uniform behavior across the application.