How to Use Model Override in LangGraph Configuration: Dynamic LLM Selection in Open Notebook

You can override the LLM model in LangGraph workflows by setting model_override in the session state, which the graph checks when configurable.model_id is not provided in the RunnableConfig.

The open-notebook repository implements a flexible model selection system for its LangGraph workflows, allowing developers to specify which LLM powers individual chat sessions. This article explains how to leverage model_override in LangGraph configuration to dynamically select models at the session level or per-invocation level.

Understanding the Model Override Hierarchy

Open Notebook’s LangGraph pipelines resolve the target LLM through a two-tier fallback mechanism defined in open_notebook/graphs/chat.py and open_notebook/graphs/source_chat.py. The system checks two sources in order of priority:

1. The configurable field of the LangGraph RunnableConfig – passed directly during graph invocation
2. The model_override attribute in session state – persisted in the ChatSession domain object

The resolution logic follows this pattern:

# open_notebook/graphs/chat.py (lines 34-36)

model_id = config.get("configurable", {}).get("model_id") \
           or state.get("model_override")

# open_notebook/graphs/source_chat.py (lines 42-44)

config.get("configurable", {}).get("model_id") or state.get("model_override")

The first non-null value among these sources determines which model provision_langchain_model() instantiates for the workflow.

Implementing Model Override in Open Notebook

Session-Scoped Configuration via API

The most common approach sets a default model for an entire chat session. When you create or update a session via the FastAPI router, the system stores your preference in SurrealDB.

Creating a new session with override:

curl -X POST http://localhost:5055/chat/sessions \
  -H "Content-Type: application/json" \
  -d '{
        "title": "Research on AI safety",
        "model_override": "gpt-4o-mini"
      }'

The router in api/routers/chat.py (lines 104-107) assigns this value to session.model_override, which persists in the database schema defined in open_notebook/domain/notebook.py (lines 79-84).

Updating an existing session:

curl -X PATCH http://localhost:5055/chat/sessions/12345 \
  -H "Content-Type: application/json" \
  -d '{"model_override": "claude-3-5-sonnet"}'

This modifies the stored override (see api/routers/chat.py, lines 269-270), affecting all future graph invocations for that session.

Runtime Configuration via RunnableConfig

For one-off model switches without modifying session state, pass the model_id directly in the config parameter during graph invocation:

from open_notebook.graphs.chat import graph

# Override the session's default for this specific call only

result = await graph.ainvoke(
    state, 
    config={"configurable": {"model_id": "gemini-1.5-pro"}}
)

Because configurable.model_id takes precedence over state["model_override"], this approach wins for the current execution while leaving the session default unchanged.

Step-by-Step Implementation Examples

Creating a Session with Model Override

When initiating a new conversation, explicitly set the model to control costs or capability levels:

curl -X POST http://localhost:5055/chat/sessions \
  -H "Content-Type: application/json" \
  -d '{
        "title": "Code Review Session",
        "model_override": "gpt-4-turbo"
      }'

Behind the scenes, the ChatSession object stores this value, making it available to call_model_with_messages during graph execution.

Invoking the Graph Without Explicit Configuration

When you invoke the graph without specifying a model_id in the config, it automatically falls back to the session’s stored override:

from open_notebook.graphs.chat import graph

state = {
    "messages": [],
    "notebook": None,
    "context": None,
    "model_override": None,  # Retrieved from database session

}

# Empty config triggers fallback to state["model_override"]

result = await graph.ainvoke(state, config={})

The graph reads the stored value from SurrealDB and provisions the corresponding LangChain model instance.

Forcing a Different Model for a Single Call

To temporarily bypass the session default without updating the database record:

result = await graph.ainvoke(
    state, 
    config={"configurable": {"model_id": "claude-3-opus-20240229"}}
)

This pattern is useful for A/B testing different models or handling specific message types that require particular capabilities.

Key Source Files

Understanding the following files helps you trace how model override in LangGraph configuration propagates through the system:

• open_notebook/graphs/chat.py – Core chat workflow containing the fallback logic for model selection (lines 30-36)
• open_notebook/graphs/source_chat.py – Source-specific chat implementation with identical override handling (lines 42-44)
• open_notebook/domain/notebook.py – Defines the ChatSession model with the model_override field (lines 79-84)
• api/routers/chat.py – REST endpoints for session creation and updates that propagate overrides to the database
• api/routers/source_chat.py – Corresponding endpoints for source-chat sessions

Summary

• Model override in LangGraph configuration uses a two-tier priority system: configurable.model_id overrides state["model_override"]
• Session-scoped overrides persist in the ChatSession domain object and apply when no explicit config is provided
• Per-call overrides pass through the RunnableConfig without modifying stored session data
• The fallback chain ends with the default model defined in open_notebook/config.py if neither source specifies a model
• All chat and source-chat graphs in the open-notebook repository implement this consistent override pattern

Frequently Asked Questions

What happens if both model_id and model_override are null?

If neither configurable.model_id nor state["model_override"] contains a value, the system falls back to the default model specified in the Esperanto configuration within open_notebook/config.py. This ensures the graph always has a valid LLM instance to execute.

Can I change the model mid-conversation?

Yes. Send a PATCH request to /chat/sessions/{id} with a new model_override value, or update the field directly in SurrealDB. All subsequent graph invocations for that session will use the new model automatically, while previous messages retain their original processing context.

Does model_override work

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}