# How to Set Up LLM Fallbacks for Resilient Agent Networks in Neuro‑SAN

> Learn how to set up LLM fallbacks for resilient agent networks in Neuro-SAN. Configure failover between LLM providers to ensure network stability and uptime.

- Repository: [Cognizant AI Lab/neuro-san-studio](https://github.com/cognizant-ai-lab/neuro-san-studio)
- Tags: how-to-guide
- Published: 2026-02-27

---

**Configure the `fallbacks` array inside the `llm_config` block of your agent-network HOCON file to enable sequential failover between multiple LLM providers when rate limits or outages occur.**

Neuro‑SAN lets you build production-grade **agent networks** that automatically survive LLM provider outages by falling back to secondary models. Setting up LLM fallbacks ensures your agents remain operational even when primary services hit rate limits or experience downtime. This resilience is configured declaratively in HOCON files and handled automatically by the runtime engine according to the cognizant-ai-lab/neuro-san-studio source code.

## Understanding the Fallback Architecture

The fallback mechanism relies on three core components working together. The `registries/llm_config.hocon` file provides global defaults for model classes and base configurations. Your specific agent-network HOCON (such as `registries/basic/music_nerd_llm_fallbacks.hocon`) declares the ordered `fallbacks` array that overrides single-model configurations. Finally, the [`coded_tools/agent_network_designer/hocon_agent_network_assembler.py`](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/coded_tools/agent_network_designer/hocon_agent_network_assembler.py) merges these configurations at runtime to produce the final executable network definition.

During execution, the runtime engine iterates through the `fallbacks` list sequentially. If a call to the first model raises an exception—such as a `RateLimitError` or network timeout—the engine immediately attempts the next entry until a valid response is returned or the list is exhausted.

## Configuring LLM Fallbacks in HOCON

Define your fallback strategy in the `llm_config` block of your network configuration file. The `fallbacks` array contains one or more model specifications that the runtime attempts in order.

```hocon
{
    "metadata": {
        "description": "Music‑nerd agent with LLM fallbacks for resilience"
    },

    "llm_config": {
        "fallbacks": [
            {   // Primary: OpenAI GPT‑4o
                "model_name": "gpt-4o"
            },
            {   // Secondary: Anthropic Claude
                "model_name": "claude-3-7-sonnet"
            }
        ]
    },

    "tools": [
        {
            "name": "MusicNerd",
            "function": {
                "description": "Answer music‑related questions."
            },
            "instructions": "You are Music Nerd …"
        }
    ]
}

```

Store this configuration in your registry directory (e.g., `registries/basic/music_nerd_llm_fallbacks.hocon`). The assembler automatically merges this with the shared defaults from `registries/llm_config.hocon`, inheriting common parameters like `class` and `temperature` while preserving your specific fallback ordering.

## Runtime Execution Flow

When an agent receives a request, the runtime executes the following fallback sequence:

1. **Configuration Assembly** – The [`hocon_agent_network_assembler.py`](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/hocon_agent_network_assembler.py) merges your network HOCON with global defaults to create the final `llm_config` object.
2. **Primary Invocation** – The system instantiates a LangChain `ChatModel` for the first entry (`gpt-4o`) and sends the prompt.
3. **Error Detection** – If the invocation throws an HTTP 429, credential error, or service outage exception, the runtime catches the failure and logs the specific error.
4. **Secondary Attempt** – The engine immediately initializes the next LLM in the `fallbacks` array (`claude-3-7-sonnet`) and retries the identical prompt.
5. **Success Propagation** – Upon receiving a valid response, the result flows back through the agent network immediately, skipping any remaining fallback entries.

This ordered approach lets you prioritize cost-effective models first while keeping premium models as safety nets.

## Deploying a Fallback-Enabled Network

Launch your resilient agent network by setting the required API keys for all providers in your fallback chain, then starting the server with your HOCON file.

```bash

# Export API keys for all configured providers

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# Launch the network with fallback support

python -m run \
    --network registries/basic/music_nerd_llm_fallbacks.hocon

```

The [`run.py`](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/run.py) entry point handles configuration assembly automatically. No additional flags are required to enable fallback behavior; the runtime detects the `fallbacks` array in `llm_config` and initializes the resilient execution path.

## Manual Fallback Implementation

For custom tooling outside the declarative HOCON system, you can implement identical sequential fallback logic using LangChain directly:

```python
from langchain.chat_models import ChatOpenAI, ChatAnthropic
from langchain.schema import HumanMessage

fallbacks = [
    {"provider": "openai", "model_name": "gpt-4o"},
    {"provider": "anthropic", "model_name": "claude-3-7-sonnet"},
]

def get_llm(entry):
    if entry["provider"] == "openai":
        return ChatOpenAI(model_name=entry["model_name"])
    elif entry["provider"] == "anthropic":
        return ChatAnthropic(model_name=entry["model_name"])
    raise ValueError("Unsupported provider")

def ask(prompt: str) -> str:
    for entry in fallbacks:
        llm = get_llm(entry)
        try:
            resp = llm.invoke([HumanMessage(content=prompt)])
            return resp.content
        except Exception as exc:
            print(f"{entry['model_name']} failed: {exc}")
    raise RuntimeError("All LLM fallbacks failed")

```

This pattern mirrors the internal implementation used by Neuro‑SAN's runtime engine when processing the `fallbacks` configuration.

## Summary

- **Define fallbacks** by adding a `fallbacks` array to the `llm_config` block in your agent-network HOCON file.
- **Order matters** – List faster or cheaper models first, with premium models as backups.
- **Automatic merging** – The [`hocon_agent_network_assembler.py`](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/hocon_agent_network_assembler.py) combines your network config with `registries/llm_config.hocon` defaults.
- **Zero-config runtime** – Start the network with `python -m run --network <file>`; fallback handling activates automatically when the array is present.
- **Resilient execution** – The runtime catches `RateLimitError` and service exceptions, seamlessly switching to the next available model.

## Frequently Asked Questions

### What types of errors trigger a fallback in Neuro‑SAN?

The runtime catches any exception during LLM invocation, including HTTP 429 rate limit errors, authentication failures, network timeouts, and service outages. When any error occurs, the engine immediately logs the failure and attempts the next model in the `fallbacks` array.

### Can I mix different LLM providers in the same fallback chain?

Yes. The `fallbacks` array supports heterogeneous providers. You can sequence OpenAI models, Anthropic Claude, or any other LangChain-compatible model in a single chain. Ensure you export the corresponding API keys for each provider before starting the server.

### How does the assembler merge global defaults with network-specific fallbacks?

The [`coded_tools/agent_network_designer/hocon_agent_network_assembler.py`](https://github.com/cognizant-ai-lab/neuro-san-studio/blob/main/coded_tools/agent_network_designer/hocon_agent_network_assembler.py) loads the shared `registries/llm_config.hocon` first, then overlays your network-specific HOCON. If your network defines a `fallbacks` array, it completely overrides any single `model_name` defined in the global defaults while inheriting other parameters like `temperature` or `max_tokens`.

### Is there a performance penalty for configuring multiple fallbacks?

No runtime overhead occurs during normal operation. The system only initializes the primary LLM initially. Secondary models are instantiated only if the primary fails, meaning you pay the initialization cost only during failover scenarios.