# Advanced Nanobot Provider Features: Multi-LLM Architecture and Fallback Strategies

> Explore advanced nanobot provider features including multi-LLM architecture and fallback strategies. Integrate dynamic provider registration OAuth and custom endpoints seamlessly.

- Repository: [✨Data Intelligence Lab@HKU✨/nanobot](https://github.com/HKUDS/nanobot)
- Tags: deep-dive
- Published: 2026-07-22

---

**Nanobot's provider system abstracts LLM backend communication through a three-layer architecture that supports dynamic provider registration, custom OpenAI-compatible endpoints, intelligent fallback chains, and OAuth authentication without requiring changes to agent logic.**

The **HKUDS/nanobot** repository implements a sophisticated provider abstraction that enables seamless switching between hosted services like OpenAI and Anthropic, local servers such as Ollama and vLLM, and custom enterprise gateways. This article examines the advanced nanobot provider features that power resilient, multi-backend AI agent deployments.

## Provider Architecture: Three-Layer Design

Nanobot's provider system consists of three tightly-coupled layers that separate transport concerns from business logic.

### Provider Registry

The **Provider Registry** maintains a mapping from provider names (e.g., `openrouter`, `anthropic`, `custom`) to concrete Python classes capable of building HTTP requests, signing them, and parsing responses. Located in [`nanobot/providers/registry.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/registry.py), this registry populates at import time and handles the `resolve_fallback` helper that merges active presets with fallback lists.

### Provider Base Class

All providers inherit from `ProviderBase` in [`nanobot/providers/base.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/base.py), which defines the common API including `chat_completion`, `embed`, and `image_generation` methods. This base class supplies utilities for pagination, streaming, and error translation, ensuring consistent behavior across diverse backends.

### Concrete Implementations

Individual provider files under `nanobot/providers/` translate generic request shapes into provider-specific payloads. For example:
- [`openai_compat_provider.py`](https://github.com/HKUDS/nanobot/blob/main/openai_compat_provider.py) handles OpenAI-compatible endpoints
- [`anthropic_provider.py`](https://github.com/HKUDS/nanobot/blob/main/anthropic_provider.py) manages Anthropic's native API
- [`bedrock_provider.py`](https://github.com/HKUDS/nanobot/blob/main/bedrock_provider.py) interfaces with AWS Bedrock

These implementations attach authentication headers (`apiKey`, `apiBase`, OAuth tokens) and return unified response models.

## Provider Selection and Model Presets

Nanobot resolves the active model through a three-step hierarchy defined in the configuration system:

1. **Explicit preset** – `agents.defaults.modelPreset` references a named entry under `modelPresets` containing `provider`, `model`, and generation parameters
2. **Implicit default** – Falls back to `agents.defaults.provider` and `agents.defaults.model` when no preset is specified
3. **Auto-resolution** – When `provider: "auto"`, the engine inspects model ID prefixes (`anthropic/…`, `openai/…`) to match configured providers

This preset system prevents accidental cross-provider calls by pinning the gateway (e.g., OpenRouter) independently of the model family.

## Custom and Named Providers

For internal OpenAI-compatible endpoints, Nanobot offers two configuration pathways via [`nanobot/providers/factory.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/factory.py):

**Custom provider** uses the reserved `providers.custom` key in [`config.json`](https://github.com/HKUDS/nanobot/blob/main/config.json):
- Requires `apiBase` (mandatory)
- Accepts `apiKey` (optional)

**Named custom providers** allow arbitrary keys under `providers.{my_name}`, where the key becomes the provider identifier used in presets. Both types are automatically treated as OpenAI-compatible.

```json
{
  "providers": {
    "companyProxy": {
      "apiKey": "${COMPANY_PROXY_API_KEY}",
      "apiBase": "https://llm-proxy.example.com/v1"
    }
  },
  "modelPresets": {
    "enterprise": {
      "provider": "companyProxy",
      "model": "gpt-4o-mini",
      "maxTokens": 8192,
      "contextWindowTokens": 65536
    }
  },
  "agents": {
    "defaults": {
      "modelPreset": "enterprise"
    }
  }
}

```

## Fallback Chains for Resilience

When requests fail due to rate limits, 5xx errors, or missing models, Nanobot walks a fallback chain defined under `agents.defaults.fallbackModels`. Each fallback specifies its own provider, model, and generation limits, ensuring retries target compatible backends.

```json
{
  "modelPresets": {
    "fast": {
      "provider": "openrouter",
      "model": "anthropic/claude-sonnet-4.5",
      "maxTokens": 4096
    },
    "localSmall": {
      "provider": "ollama",
      "model": "llama3.2",
      "maxTokens": 4096,
      "contextWindowTokens": 32768
    }
  },
  "agents": {
    "defaults": {
      "modelPreset": "fast",
      "fallbackModels": ["localSmall"]
    }
  }
}

```

The fallback logic resides in [`nanobot/providers/registry.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/registry.py), where the `resolve_fallback` helper orchestrates the transition between presets.

## Proxy Support and Network Configuration

Proxy handling is implemented in `nanobot/providers/base._http_client` and injected into the request pipeline. However, support varies by provider type:

- **Supported**: OpenAI-compatible providers (`openai`, `custom`, `ollama`, `vllm`) respect the per-provider `proxy` field
- **Unsupported**: Native backends including Anthropic, Bedrock, and Azure OpenAI reject proxy configurations and require endpoint-specific network setup

## OAuth Authentication Flows

Providers like OpenAI Codex and GitHub Copilot implement interactive OAuth flows through the CLI:

```bash
nanobot provider login openai-codex
nanobot provider login github-copilot

```

Credentials store in the user's config directory and reference by provider name without exposing tokens in configuration files. Implementation details live in [`nanobot/providers/openai_codex_provider.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/openai_codex_provider.py) and [`nanobot/providers/github_copilot_provider.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/github_copilot_provider.py).

## Extending the Provider System

Adding a new provider requires three steps:

1. **Subclass `ProviderBase`** in [`nanobot/providers/base.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/base.py) and implement abstract methods (`chat_completion`, etc.)
2. **Register the class** in [`nanobot/providers/registry.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/providers/registry.py) (auto-discovered via `pkgutil`)
3. **Add config schema** (optional) in [`nanobot/config/schema.py`](https://github.com/HKUDS/nanobot/blob/main/nanobot/config/schema.py) for Pydantic validation

Because agent loops, tool calls, and the WebUI rely on the unified interface, new backends immediately become available throughout the system without modifying consumer code.

## Summary

- **Three-layer architecture**: Registry ([`registry.py`](https://github.com/HKUDS/nanobot/blob/main/registry.py)), Base class ([`base.py`](https://github.com/HKUDS/nanobot/blob/main/base.py)), and concrete implementations enable clean separation of transport logic
- **Flexible configuration**: Support for custom OpenAI-compatible endpoints via named providers in [`config.json`](https://github.com/HKUDS/nanobot/blob/main/config.json)
- **Intelligent resolution**: Three-step model selection with auto-detection and explicit preset pinning
- **Production resilience**: Fallback chains defined in `fallbackModels` automatically retry failed requests against alternative backends
- **Network limitations**: Proxy support limited to OpenAI-compatible providers; native backends require direct connectivity
- **OAuth integration**: CLI-based authentication for Codex and Copilot stores credentials securely outside configuration files

## Frequently Asked Questions

### How do I configure Nanobot to use a local vLLM server?

Create a named provider under `providers` in your [`config.json`](https://github.com/HKUDS/nanobot/blob/main/config.json) with the `apiBase` pointing to your vLLM endpoint. Since vLLM is OpenAI-compatible, Nanobot automatically routes requests through [`openai_compat_provider.py`](https://github.com/HKUDS/nanobot/blob/main/openai_compat_provider.py) without additional plugins.

### Why does my Anthropic provider ignore the proxy settings?

Native providers like Anthropic, Bedrock, and Azure OpenAI do not support the `proxy` field in Nanobot's configuration. Only OpenAI-compatible providers (including `custom`, `ollama`, and `vllm`) respect proxy settings implemented in `nanobot/providers/base._http_client`.

### How does the fallback chain handle different context window sizes?

Each fallback preset in `fallbackModels` defines its own `contextWindowTokens` and `maxTokens`. When Nanobot switches to a fallback provider via the `resolve_fallback` logic in [`registry.py`](https://github.com/HKUDS/nanobot/blob/main/registry.py), it uses the generation parameters defined in that specific preset, ensuring the new backend's limits are respected.

### Can I switch providers mid-conversation?

Yes. Use the CLI command `nanobot agent -m "/model {preset_name}"` to change the active model preset on-the-fly. Subsequent turns automatically use the new provider specified in the preset, whether it's a hosted service like OpenRouter or a local Ollama instance.