How Model Overrides Work on a Per-Request Basis Using RunnableConfig in Open Notebook

Open Notebook’s LangGraph workflows accept a model_id inside the configurable dictionary of a RunnableConfig to override the default language model for individual requests, falling back to thread state or system defaults only when omitted.

This mechanism allows callers to dynamically select which AI model processes a specific interaction without altering global configuration. When an API request arrives, the system extracts any per-request model preference and injects it into the RunnableConfig before invoking the graph, ensuring the selected model is provisioned for that invocation only.

The Three-Tier Resolution Hierarchy

Open Notebook resolves which language model to use through a strict priority chain. This design ensures explicit per-request overrides take precedence over stored preferences while maintaining sensible defaults.

1. Explicit RunnableConfig Override

The highest priority source is the configurable dictionary passed within the RunnableConfig object at invocation time. Graph nodes check config.get("configurable", {}).get("model_id") first. If present, this value is sent directly to the provisioning layer.

2. Thread State Fallback

If the RunnableConfig lacks a model_id, the system checks state.get("model_override"). This allows a model selection to persist across turns in a conversation when set earlier in the thread lifecycle.

3. Default Model Provisioning

When neither the config nor the state specifies a model, the provision_langchain_model helper selects the system default based on the requested model type (e.g., large-context or standard chat).

Implementation in the Source Code

Chat Graph Node

In open_notebook/graphs/chat.py, the call_model_with_messages node implements the resolution logic:

model_id = config.get("configurable", {}).get("model_id") or state.get("model_override")

This line (lines 30-36 in the source) demonstrates the short-circuit evaluation that prioritizes the RunnableConfig over thread state. The resolved model_id is then passed to the provisioning helper.

Source-Chat Graph Node

The source-chat workflow in open_notebook/graphs/source_chat.py follows an identical pattern. Within _call_model_with_source_context_inner, the same configuration check occurs (lines 41-45), ensuring consistency across different graph types.

Model Provisioning Helper

The provision_langchain_model function in open_notebook/ai/provision.py receives the resolved identifier. When model_id is explicitly provided (lines 28-34), it invokes model_manager.get_model(model_id, **kwargs) to return a LangChain-compatible model instance. If the ID is invalid, it raises a configuration error before the node attempts generation.

API Router Integration

The streaming endpoints in api/routers/source_chat.py construct the RunnableConfig explicitly. For example, when handling a request, the router builds:

RunnableConfig(configurable={"thread_id": session_id, "model_id": model_override})

This pattern (lines 44-48) bridges the HTTP API layer with the graph, translating the JSON payload's model_override field into the configuration structure the nodes expect.

Practical Code Examples

Calling the Chat Endpoint with an Override

Send a per-request model selection via the REST API:

import httpx

payload = {
    "message": "Explain the difference between GPT-4 and Claude-3.",
    "model_override": "gpt-4o-mini"  # Per-request model ID

}

resp = httpx.post(
    "http://localhost:5055/api/chat/sessions/12345/messages",
    json=payload,
)
print(resp.json())

The router converts the model_override field into a RunnableConfig with {"model_id": "gpt-4o-mini"}, which the chat graph node reads during execution.

Direct Graph Invocation

Programmatically override the model when invoking the graph directly:

from langchain_core.runnables import RunnableConfig
from open_notebook.graphs.chat import graph, ThreadState

state: ThreadState = {"messages": [], "model_override": None}
config = RunnableConfig(
    configurable={"model_id": "anthropic/claude-3-opus-20240229"}
)

result = graph.invoke(input=state, config=config)
print(result["messages"])

Here, the RunnableConfig explicitly sets the model, bypassing any value stored in state["model_override"].

Streaming Source-Chat with Override

For streaming responses, the override flows through the async generator:

import asyncio
from api.routers.source_chat import stream_source_chat_response

async def demo():
    async for chunk in stream_source_chat_response(
        session_id="chat_session:abc",
        source_id="source:xyz",
        message="Summarize the key points.",
        model_override="gpt-4o-mini",  # Per-request override

    ):
        print(chunk)

asyncio.run(demo())

The stream_source_chat_response function forwards the override into the RunnableConfig, which the source-chat graph consumes exactly as the standard chat graph does.

Summary

• RunnableConfig is the primary vehicle for per-request model overrides in Open Notebook, using the configurable["model_id"] key.
• Resolution follows a strict hierarchy: explicit config override → thread state fallback → system default.
• Both chat.py and source_chat.py graph nodes implement identical logic to check config.get("configurable", {}).get("model_id") before falling back to state.
• The provision_langchain_model helper in ai/provision.py instantiates the specific model instance based on the resolved identifier.
• API routers bridge HTTP requests to graph invocations by constructing RunnableConfig objects that carry the model_override from the request payload.

Frequently Asked Questions

What happens if I provide both a RunnableConfig model_id and a state model_override?

The RunnableConfig value takes precedence. In open_notebook/graphs/chat.py, the node uses config.get("configurable", {}).get("model_id") or state.get("model_override"), meaning the explicit config is evaluated first and the state is only checked if the config value is absent or evaluates to False.

Can I use any model ID available in the model manager?

Yes, provided the ID exists in the model manager's registry. The provision_langchain_model function calls model_manager.get_model(model_id, **kwargs), which will raise a configuration error if the ID is unrecognized. Valid IDs depend on your Open Notebook installation's configured providers (e.g., OpenAI, Anthropic, or local models).

Does the model override affect the entire thread or just the current request?

The override affects only the current request invocation. However, if your application logic saves the model_id back into state["model_override"] after processing, subsequent turns in the same thread will inherit that selection via the state fallback mechanism until a new RunnableConfig override is provided or the state is cleared.

Is the RunnableConfig approach specific to certain graph types?

No, this is a consistent pattern across Open Notebook's LangGraph implementations. Both the standard chat graph (open_notebook/graphs/chat.py) and the source-chat graph (open_notebook/graphs/source_chat.py) use identical configuration checking logic, as do their respective API routers, ensuring uniform behavior across the application.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}