Advanced Nanobot Provider Features: Multi-LLM Architecture and Fallback Strategies

Nanobot's provider system abstracts LLM backend communication through a three-layer architecture that supports dynamic provider registration, custom OpenAI-compatible endpoints, intelligent fallback chains, and OAuth authentication without requiring changes to agent logic.

The HKUDS/nanobot repository implements a sophisticated provider abstraction that enables seamless switching between hosted services like OpenAI and Anthropic, local servers such as Ollama and vLLM, and custom enterprise gateways. This article examines the advanced nanobot provider features that power resilient, multi-backend AI agent deployments.

Provider Architecture: Three-Layer Design

Nanobot's provider system consists of three tightly-coupled layers that separate transport concerns from business logic.

Provider Registry

The Provider Registry maintains a mapping from provider names (e.g., openrouter, anthropic, custom) to concrete Python classes capable of building HTTP requests, signing them, and parsing responses. Located in nanobot/providers/registry.py, this registry populates at import time and handles the resolve_fallback helper that merges active presets with fallback lists.

Provider Base Class

All providers inherit from ProviderBase in nanobot/providers/base.py, which defines the common API including chat_completion, embed, and image_generation methods. This base class supplies utilities for pagination, streaming, and error translation, ensuring consistent behavior across diverse backends.

Concrete Implementations

Individual provider files under nanobot/providers/ translate generic request shapes into provider-specific payloads. For example:

• openai_compat_provider.py handles OpenAI-compatible endpoints
• anthropic_provider.py manages Anthropic's native API
• bedrock_provider.py interfaces with AWS Bedrock

These implementations attach authentication headers (apiKey, apiBase, OAuth tokens) and return unified response models.

Provider Selection and Model Presets

Nanobot resolves the active model through a three-step hierarchy defined in the configuration system:

1. Explicit preset – agents.defaults.modelPreset references a named entry under modelPresets containing provider, model, and generation parameters
2. Implicit default – Falls back to agents.defaults.provider and agents.defaults.model when no preset is specified
3. Auto-resolution – When provider: "auto", the engine inspects model ID prefixes (anthropic/…, openai/…) to match configured providers

This preset system prevents accidental cross-provider calls by pinning the gateway (e.g., OpenRouter) independently of the model family.

Custom and Named Providers

For internal OpenAI-compatible endpoints, Nanobot offers two configuration pathways via nanobot/providers/factory.py:

Custom provider uses the reserved providers.custom key in config.json:

• Requires apiBase (mandatory)
• Accepts apiKey (optional)

Named custom providers allow arbitrary keys under providers.{my_name}, where the key becomes the provider identifier used in presets. Both types are automatically treated as OpenAI-compatible.

{
  "providers": {
    "companyProxy": {
      "apiKey": "${COMPANY_PROXY_API_KEY}",
      "apiBase": "https://llm-proxy.example.com/v1"
    }
  },
  "modelPresets": {
    "enterprise": {
      "provider": "companyProxy",
      "model": "gpt-4o-mini",
      "maxTokens": 8192,
      "contextWindowTokens": 65536
    }
  },
  "agents": {
    "defaults": {
      "modelPreset": "enterprise"
    }
  }
}

Fallback Chains for Resilience

When requests fail due to rate limits, 5xx errors, or missing models, Nanobot walks a fallback chain defined under agents.defaults.fallbackModels. Each fallback specifies its own provider, model, and generation limits, ensuring retries target compatible backends.

{
  "modelPresets": {
    "fast": {
      "provider": "openrouter",
      "model": "anthropic/claude-sonnet-4.5",
      "maxTokens": 4096
    },
    "localSmall": {
      "provider": "ollama",
      "model": "llama3.2",
      "maxTokens": 4096,
      "contextWindowTokens": 32768
    }
  },
  "agents": {
    "defaults": {
      "modelPreset": "fast",
      "fallbackModels": ["localSmall"]
    }
  }
}

The fallback logic resides in nanobot/providers/registry.py, where the resolve_fallback helper orchestrates the transition between presets.

Proxy Support and Network Configuration

Proxy handling is implemented in nanobot/providers/base._http_client and injected into the request pipeline. However, support varies by provider type:

• Supported: OpenAI-compatible providers (openai, custom, ollama, vllm) respect the per-provider proxy field
• Unsupported: Native backends including Anthropic, Bedrock, and Azure OpenAI reject proxy configurations and require endpoint-specific network setup

OAuth Authentication Flows

Providers like OpenAI Codex and GitHub Copilot implement interactive OAuth flows through the CLI:

nanobot provider login openai-codex
nanobot provider login github-copilot

Credentials store in the user's config directory and reference by provider name without exposing tokens in configuration files. Implementation details live in nanobot/providers/openai_codex_provider.py and nanobot/providers/github_copilot_provider.py.

Extending the Provider System

Adding a new provider requires three steps:

1. Subclass ProviderBase in nanobot/providers/base.py and implement abstract methods (chat_completion, etc.)
2. Register the class in nanobot/providers/registry.py (auto-discovered via pkgutil)
3. Add config schema (optional) in nanobot/config/schema.py for Pydantic validation

Because agent loops, tool calls, and the WebUI rely on the unified interface, new backends immediately become available throughout the system without modifying consumer code.

Summary

• Three-layer architecture: Registry (registry.py), Base class (base.py), and concrete implementations enable clean separation of transport logic
• Flexible configuration: Support for custom OpenAI-compatible endpoints via named providers in config.json
• Intelligent resolution: Three-step model selection with auto-detection and explicit preset pinning
• Production resilience: Fallback chains defined in fallbackModels automatically retry failed requests against alternative backends
• Network limitations: Proxy support limited to OpenAI-compatible providers; native backends require direct connectivity
• OAuth integration: CLI-based authentication for Codex and Copilot stores credentials securely outside configuration files

Frequently Asked Questions

How do I configure Nanobot to use a local vLLM server?

Create a named provider under providers in your config.json with the apiBase pointing to your vLLM endpoint. Since vLLM is OpenAI-compatible, Nanobot automatically routes requests through openai_compat_provider.py without additional plugins.

Why does my Anthropic provider ignore the proxy settings?

Native providers like Anthropic, Bedrock, and Azure OpenAI do not support the proxy field in Nanobot's configuration. Only OpenAI-compatible providers (including custom, ollama, and vllm) respect proxy settings implemented in nanobot/providers/base._http_client.

How does the fallback chain handle different context window sizes?

Each fallback preset in fallbackModels defines its own contextWindowTokens and maxTokens. When Nanobot switches to a fallback provider via the resolve_fallback logic in registry.py, it uses the generation parameters defined in that specific preset, ensuring the new backend's limits are respected.

Can I switch providers mid-conversation?

Yes. Use the CLI command nanobot agent -m "/model {preset_name}" to change the active model preset on-the-fly. Subsequent turns automatically use the new provider specified in the preset, whether it's a hosted service like OpenRouter or a local Ollama instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}