How to Set Up LLM Fallbacks for Resilient Agent Networks in Neuro‑SAN

Configure the fallbacks array inside the llm_config block of your agent-network HOCON file to enable sequential failover between multiple LLM providers when rate limits or outages occur.

Neuro‑SAN lets you build production-grade agent networks that automatically survive LLM provider outages by falling back to secondary models. Setting up LLM fallbacks ensures your agents remain operational even when primary services hit rate limits or experience downtime. This resilience is configured declaratively in HOCON files and handled automatically by the runtime engine according to the cognizant-ai-lab/neuro-san-studio source code.

Understanding the Fallback Architecture

The fallback mechanism relies on three core components working together. The registries/llm_config.hocon file provides global defaults for model classes and base configurations. Your specific agent-network HOCON (such as registries/basic/music_nerd_llm_fallbacks.hocon) declares the ordered fallbacks array that overrides single-model configurations. Finally, the coded_tools/agent_network_designer/hocon_agent_network_assembler.py merges these configurations at runtime to produce the final executable network definition.

During execution, the runtime engine iterates through the fallbacks list sequentially. If a call to the first model raises an exception—such as a RateLimitError or network timeout—the engine immediately attempts the next entry until a valid response is returned or the list is exhausted.

Configuring LLM Fallbacks in HOCON

Define your fallback strategy in the llm_config block of your network configuration file. The fallbacks array contains one or more model specifications that the runtime attempts in order.

{
    "metadata": {
        "description": "Music‑nerd agent with LLM fallbacks for resilience"
    },

    "llm_config": {
        "fallbacks": [
            {   // Primary: OpenAI GPT‑4o
                "model_name": "gpt-4o"
            },
            {   // Secondary: Anthropic Claude
                "model_name": "claude-3-7-sonnet"
            }
        ]
    },

    "tools": [
        {
            "name": "MusicNerd",
            "function": {
                "description": "Answer music‑related questions."
            },
            "instructions": "You are Music Nerd …"
        }
    ]
}

Store this configuration in your registry directory (e.g., registries/basic/music_nerd_llm_fallbacks.hocon). The assembler automatically merges this with the shared defaults from registries/llm_config.hocon, inheriting common parameters like class and temperature while preserving your specific fallback ordering.

Runtime Execution Flow

When an agent receives a request, the runtime executes the following fallback sequence:

1. Configuration Assembly – The hocon_agent_network_assembler.py merges your network HOCON with global defaults to create the final llm_config object.
2. Primary Invocation – The system instantiates a LangChain ChatModel for the first entry (gpt-4o) and sends the prompt.
3. Error Detection – If the invocation throws an HTTP 429, credential error, or service outage exception, the runtime catches the failure and logs the specific error.
4. Secondary Attempt – The engine immediately initializes the next LLM in the fallbacks array (claude-3-7-sonnet) and retries the identical prompt.
5. Success Propagation – Upon receiving a valid response, the result flows back through the agent network immediately, skipping any remaining fallback entries.

This ordered approach lets you prioritize cost-effective models first while keeping premium models as safety nets.

Deploying a Fallback-Enabled Network

Launch your resilient agent network by setting the required API keys for all providers in your fallback chain, then starting the server with your HOCON file.

# Export API keys for all configured providers

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# Launch the network with fallback support

python -m run \
    --network registries/basic/music_nerd_llm_fallbacks.hocon

The run.py entry point handles configuration assembly automatically. No additional flags are required to enable fallback behavior; the runtime detects the fallbacks array in llm_config and initializes the resilient execution path.

Manual Fallback Implementation

For custom tooling outside the declarative HOCON system, you can implement identical sequential fallback logic using LangChain directly:

from langchain.chat_models import ChatOpenAI, ChatAnthropic
from langchain.schema import HumanMessage

fallbacks = [
    {"provider": "openai", "model_name": "gpt-4o"},
    {"provider": "anthropic", "model_name": "claude-3-7-sonnet"},
]

def get_llm(entry):
    if entry["provider"] == "openai":
        return ChatOpenAI(model_name=entry["model_name"])
    elif entry["provider"] == "anthropic":
        return ChatAnthropic(model_name=entry["model_name"])
    raise ValueError("Unsupported provider")

def ask(prompt: str) -> str:
    for entry in fallbacks:
        llm = get_llm(entry)
        try:
            resp = llm.invoke([HumanMessage(content=prompt)])
            return resp.content
        except Exception as exc:
            print(f"{entry['model_name']} failed: {exc}")
    raise RuntimeError("All LLM fallbacks failed")

This pattern mirrors the internal implementation used by Neuro‑SAN's runtime engine when processing the fallbacks configuration.

Summary

• Define fallbacks by adding a fallbacks array to the llm_config block in your agent-network HOCON file.
• Order matters – List faster or cheaper models first, with premium models as backups.
• Automatic merging – The hocon_agent_network_assembler.py combines your network config with registries/llm_config.hocon defaults.
• Zero-config runtime – Start the network with python -m run --network <file>; fallback handling activates automatically when the array is present.
• Resilient execution – The runtime catches RateLimitError and service exceptions, seamlessly switching to the next available model.

Frequently Asked Questions

What types of errors trigger a fallback in Neuro‑SAN?

The runtime catches any exception during LLM invocation, including HTTP 429 rate limit errors, authentication failures, network timeouts, and service outages. When any error occurs, the engine immediately logs the failure and attempts the next model in the fallbacks array.

Can I mix different LLM providers in the same fallback chain?

Yes. The fallbacks array supports heterogeneous providers. You can sequence OpenAI models, Anthropic Claude, or any other LangChain-compatible model in a single chain. Ensure you export the corresponding API keys for each provider before starting the server.

How does the assembler merge global defaults with network-specific fallbacks?

The coded_tools/agent_network_designer/hocon_agent_network_assembler.py loads the shared registries/llm_config.hocon first, then overlays your network-specific HOCON. If your network defines a fallbacks array, it completely overrides any single model_name defined in the global defaults while inheriting other parameters like temperature or max_tokens.

Is there a performance penalty for configuring multiple fallbacks?

No runtime overhead occurs during normal operation. The system only initializes the primary LLM initially. Secondary models are instantiated only if the primary fails, meaning you pay the initialization cost only during failover scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}