Configuring Multiple Model Providers with Provider-Specific Settings in Flue

Flue lets you register unlimited LLM endpoints and patch their transport settings at runtime using registerProvider and configureProvider, enabling you to route traffic to local Ollama instances, custom gateways, or Cloudflare Workers AI without rebuilding your agents.

The withastro/flue SDK decouples your agent code from specific LLM backends through a flexible provider registry. When you need to switch between OpenAI, Anthropic, and local models—or route traffic through corporate proxies—you can configure multiple model providers with provider‑specific settings using a URL‑prefix based registration system that persists at runtime.

Understanding Flue's Provider Registry

Flue maintains two critical data structures in packages/sdk/src/runtime/providers.ts: the userModels registry that maps URL prefixes to provider implementations, and the providerOverrides map that stores transport‑level patches. According to the source code, when you call registerProvider(name, registration), the system stores your configuration in userModels using a last‑write‑wins strategy (lines 89‑94).

The configureProvider function then writes to providerOverrides (lines 138‑146), allowing you to modify settings like baseUrl or apiKey without touching the original registration. When resolving models, Flue merges these overrides via getProviderConfiguration (lines 71‑75) before passing the final configuration to the runtime harness.

Registration Object Structure

For HTTP‑based providers, registrations require the api field (a pi‑ai API slug like "openai-completions"), a baseUrl string, and optional apiKey, headers, and provider override fields.

{
  api: "openai-completions",
  baseUrl: "https://api.anthropic.com/v1",
  apiKey?: string,
  headers?: Record<string, string>,
  provider?: string
}

Cloudflare Workers AI deployments use a separate CloudflareAIBindingRegistration interface that accepts a Workers AI binding instead of HTTP credentials.

Registering Custom Model Providers

To add a new provider, import registerProvider from @flue/sdk/app and specify the transport details. The registration creates a URL prefix that agents reference using the "prefix/modelId" syntax.

The following example from examples/hello-world/.flue/app.ts registers two local OpenAI‑compatible servers:

import { registerProvider } from '@flue/sdk/app';

// Register Ollama endpoint
registerProvider('ollama', {
  api: 'openai-completions',
  baseUrl: 'http://localhost:11434/v1',
});

// Register LM Studio endpoint
registerProvider('lmstudio', {
  api: 'openai-completions',
  baseUrl: 'http://localhost:1234/v1',
});

Each registration creates a unique namespace. When an agent requests ollama/llama3.2, Flue resolves the prefix to the Ollama registration, constructs the appropriate pi‑ai Model object, and routes the request to http://localhost:11434/v1.

Patching Provider Settings with configureProvider

The configureProvider(slug, settings) function enables runtime modifications to any registered provider, including built‑ins like Anthropic. The system uses the provider slug—the resolved Model.provider value—to look up overrides and merge them into the final configuration.

import { configureProvider } from '@flue/sdk/app';

// Route Anthropic traffic through a corporate gateway
if (process.env.ANTHROPIC_GATEWAY_URL) {
  configureProvider('anthropic', {
    baseUrl: process.env.ANTHROPIC_GATEWAY_URL,
    apiKey: process.env.ANTHROPIC_API_KEY,
    headers: {
      'X-Custom-Auth': process.env.INTERNAL_AUTH_TOKEN
    }
  });
}

This patching occurs at runtime without requiring a rebuild. The overrides are stored separately from the base registration, allowing you to modify transport details while preserving the catalog metadata that pi‑ai supplies.

Referencing Models in Agent Code

Once configured, models become immediately available using the prefix syntax. The runtime resolves these references through resolveRegisteredModel → buildModelFromRegistration, applying any configured overrides before executing the API call.

// Uses the patched Anthropic gateway
await ctx.session.prompt({
  model: 'anthropic/claude-3-5-sonnet-20241022',
});

// Routes to local Ollama instance
await ctx.session.prompt({
  model: 'ollama/llama3.2',
});

// Routes to LM Studio
await ctx.session.prompt({
  model: 'lmstudio/mistral',
});

Implementing Custom API Handlers

For providers not supported by pi‑ai natively, use registerApiProvider to supply your own streaming implementations, then alias them with registerProvider. The API registration must implement stream and streamSimple methods that conform to the pi‑ai transport interface.

import { registerApiProvider, registerProvider } from '@flue/sdk/app';
import { myCustomStream, myCustomStreamSimple } from './my-api-implementation';

// 1. Register the low-level API implementation
registerApiProvider({
  api: 'my-custom-api',
  stream: myCustomStream,
  streamSimple: myCustomStreamSimple,
});

// 2. Create a friendly URL prefix
registerProvider('custom', {
  api: 'my-custom-api',
  baseUrl: 'https://api.custom-service.com/v1',
  apiKey: process.env.CUSTOM_API_KEY,
});

Now agents can request custom/my-model and Flue will route the call through your custom stream handlers instead of the default HTTP transport.

Summary

• registerProvider creates URL‑prefix mappings in userModels that link short names to HTTP endpoints or Cloudflare Workers AI bindings.
• configureProvider patches transport settings (baseUrl, apiKey, headers) at runtime via the providerOverrides map without losing catalog metadata.
• Provider slugs are the resolved Model.provider values used for override lookups, distinct from the URL prefixes used for registration.
• Custom APIs require registerApiProvider to define stream implementations before aliasing them with registerProvider.
• All configuration happens in packages/sdk/src/runtime/providers.ts and is exposed through packages/sdk/src/app.ts for application code.

Frequently Asked Questions

What is the difference between registerProvider and configureProvider in Flue?

registerProvider establishes the initial URL‑prefix mapping and specifies the pi‑ai API type (e.g., "openai-completions") and endpoint location, storing data in the userModels registry. configureProvider modifies existing registrations (including built‑ins) by writing to providerOverrides, allowing you to change transport details like authentication headers or base URLs at runtime without redefining the provider's core metadata.

How do I route a built-in provider like Anthropic through a corporate proxy?

Use configureProvider('anthropic', { baseUrl: 'https://gateway.company.com', ... }) after importing the function from @flue/sdk/app. The system looks up overrides via getProviderConfiguration before each request, ensuring your gateway receives the traffic while the agent code continues using the standard anthropic/model-id syntax.

Can I register multiple local OpenAI-compatible servers simultaneously?

Yes. Register each local endpoint with a unique prefix in your app.ts file. For example, registerProvider('ollama', { baseUrl: 'http://localhost:11434/v1', ... }) and registerProvider('lmstudio', { baseUrl: 'http://localhost:1234/v1', ... }) allow agents to reference ollama/llama3.2 and lmstudio/mistral concurrently, with each request routed to the appropriate localhost port.

Where does Flue store provider configurations at runtime?

Flue maintains provider data in two maps defined in packages/sdk/src/runtime/providers.ts: userModels stores the base registrations from registerProvider, while providerOverrides stores patches from configureProvider. When resolving a model, the runtime merges these configurations via buildModelFromRegistration before executing the API call.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}