How to Override AI Models per Request Using RunnableConfig in LangGraph

You can override AI models for individual requests in LangGraph by passing a model_id through the configurable dictionary inside RunnableConfig, which flows from the API layer through graph nodes to the model provisioning helper without affecting global settings.

Open Notebook uses LangGraph to orchestrate AI-driven workflows, and you can dynamically select which language model processes each request by leveraging the RunnableConfig object. This approach allows you to specify a model_id for a single invocation while keeping the rest of your application configuration intact.

How RunnableConfig Enables Per-Request Model Overrides

LangGraph's RunnableConfig object provides a configurable dictionary that travels with every graph invocation. In lfnovo/open-notebook, this dictionary carries two critical pieces of metadata: the thread_id for state isolation and the model_id for model selection. Because RunnableConfig is thread-local and request-scoped, you can override the default model for a single execution without side effects on concurrent requests or global settings.

The Three-Layer Override Flow

The model override travels through three distinct layers, from the HTTP API down to the model instantiation logic.

API Layer: Capturing the Model Override

In api/routers/chat.py, the FastAPI router accepts an optional model_override field in the request body. The ExecuteChatRequest model defines this field, and the route handler packages it into a RunnableConfig:

class ExecuteChatRequest(BaseModel):
    session_id: str
    message: str
    context: Dict[str, Any]
    model_override: Optional[str] = None   # Per-request model selection

@router.post("/chat/execute", response_model=ExecuteChatResponse)
async def execute_chat(req: ExecuteChatRequest):
    cfg = RunnableConfig(
        configurable={
            "thread_id": f"chat_session:{req.session_id}",
            "model_id": req.model_override,  # Passed downstream to nodes

        }
    )
    result = await chat_graph.ainvoke(
        {"messages": [...], "context": req.context},
        config=cfg,
    )

Graph Layer: Extracting Configuration in Nodes

Each node in open_notebook/graphs/chat.py receives the RunnableConfig as a second argument. The call_model_with_messages function extracts the model_id from the configurable dictionary and passes it to the provisioning helper:

def call_model_with_messages(state: ThreadState, config: RunnableConfig) -> dict:
    # Pull the optional override supplied by the API

    model_id = config.get("configurable", {}).get("model_id") \
                or state.get("model_override")
    
    # Provision the concrete LangChain model

    model = await provision_langchain_model(
        str(payload), model_id, "chat", max_tokens=8192
    )
    ai_message = model.invoke(payload)
    return {"messages": [ai_message]}

Provisioning Layer: Resolving the Model Instance

The provision_langchain_model function in open_notebook/ai/provision.py makes the final decision. If a model_id is present in the configuration, it loads that specific model; otherwise, it falls back to the default model for the requested type:

async def provision_langchain_model(content, model_id, default_type, **kwargs):
    if model_id:
        # Explicit request – load that model

        model = await model_manager.get_model(model_id, **kwargs)
    else:
        # No override – use the default for the type (chat, embed, etc.)

        model = await model_manager.get_default_model(default_type, **kwargs)
    
    return model.to_langchain()

Complete Implementation Example

Here is the end-to-end flow showing how to trigger a model override from a client request:

curl -X POST https://api.example.com/chat/execute \
  -H "Content-Type: application/json" \
  -d '{
        "session_id": "12345",
        "message": "Explain quantum tunneling",
        "context": { "sources": [] },
        "model_override": "gpt-4o-mini"
      }'

When this request hits the graph, the call_model_with_messages node will use gpt-4o-mini instead of the default chat model. You can apply this same pattern to other graphs like open_notebook/graphs/ask.py or open_notebook/graphs/source_chat.py by reading the same configurable key in their node functions.

Practical Use Cases for Model Overrides

Testing new LLMs: Send model_override: "gpt-4o-mini" in the request JSON to evaluate a new model's behavior without changing the application-wide configuration.

Handling large contexts: Force a specific long-context model (such as "anthropic/claude-3-5-sonnet") only for requests that exceed a certain token threshold, while keeping standard queries on cheaper models.

Provider-specific features: Pass provider-specific model IDs to access unique capabilities (like extended thinking modes or specific tool-calling formats) for experimental features.

Summary

• RunnableConfig carries a configurable dictionary that includes thread_id for state isolation and model_id for model selection.
• The API layer in api/routers/chat.py accepts an optional model_override field and packages it into the configuration.
• Graph nodes in open_notebook/graphs/chat.py extract the model_id from config.get("configurable", {}) and pass it to the provisioning helper.
• The provisioning layer in open_notebook/ai/provision.py resolves the explicit model_id or falls back to defaults, ensuring no global side effects.
• This pattern works across all LangGraph workflows in the repository, including source_chat.py and transformation.py.

Frequently Asked Questions

What is RunnableConfig in LangGraph?

RunnableConfig is a configuration object that LangGraph passes to every node during graph execution. It contains a configurable dictionary that can hold arbitrary key-value pairs, allowing you to inject request-specific data like thread_id for state management or model_id for model selection without modifying global state.

How does Open Notebook isolate state between chat sessions?

Open Notebook uses the thread_id key inside the configurable dictionary to isolate LangGraph state. Each chat session receives a unique thread_id (formatted as chat_session:{session_id}), ensuring that conversation history and context remain separate across different users and sessions.

Can I use model overrides in other graphs besides chat?

Yes. The same pattern applies to open_notebook/graphs/ask.py, open_notebook/graphs/source_chat.py, and open_notebook/graphs/transformation.py. Any node that receives RunnableConfig can extract model_id from config.get("configurable", {}) and pass it to provision_langchain_model to override the default model for that specific execution.

What happens if the specified model_id is not found?

If the model_id provided in the override does not exist in the model manager, the provision_langchain_model function in open_notebook/ai/provision.py will attempt to load it via model_manager.get_model(model_id). If that fails, the system will raise an error rather than silently falling back to the default, ensuring explicit behavior and preventing unexpected model switches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}