How OpenMAIC Handles LLM, TTS, and ASR Provider Integrations: A Technical Deep Dive

OpenMAIC uses a unified, data-driven architecture with environment-variable mapping, static registries, and server-side provider implementations to securely integrate LLM, TTS, ASR, and other external services without exposing credentials to the client.

The THU-MAIC/OpenMAIC repository implements a modular provider system that abstracts away the complexity of connecting to various AI services. This architecture ensures that API keys and sensitive configuration remain server-side while giving operators granular control over which providers are enabled fleet-wide.

The Three-Layer Provider Architecture

OpenMAIC organizes its provider integrations into distinct layers that separate configuration, metadata, and runtime execution.

Environment-to-ID Mapping

The system begins with environment variable scanning in lib/server/provider-config.ts. This file defines mapping dictionaries—LLM_ENV_MAP, TTS_ENV_MAP, and ASR_ENV_MAP—that translate environment variable prefixes into canonical provider IDs. For example, variables like OPENAI_API_KEY or TTS_LEMONADE_API_KEY are mapped to internal identifiers such as openai or lemonade.

Static Provider Registries

Centralized metadata lives in lib/audio/constants.ts, which exports TTS_PROVIDERS and ASR_PROVIDERS objects. These registries contain static information including display names, required API keys, default base URLs, available models or voices, and supported audio formats. Both client and server code import these constants, ensuring a single source of truth for provider capabilities without leaking sensitive configuration.

Server-Side Configuration Loading

The loadEnvSection function in lib/server/provider-config.ts merges YAML configuration files with environment variables and processes force-disable switches. The DISABLE_ENV_MAPS structure extends the regular environment maps to include "keyless" providers like browser-native or ComfyUI integrations, allowing operators to disable capabilities globally using flags such as ASR_OPENAI_ENABLED=false.

How Provider Integration Works at Runtime

The architecture operates through five coordinated stages that move from configuration detection to API execution.

Stage 1: Environment Variable Resolution

During server startup, the provider-config module scans process.env for provider-specific variables. The prefix-to-ID maps translate these into canonical identifiers. This approach allows operators to configure multiple providers for the same capability (e.g., both OpenAI and Azure for TTS) without code changes.

Stage 2: Registry-Based UI Population

Client-side components import TTS_PROVIDERS and ASR_PROVIDERS from lib/audio/constants.ts to render provider selection dropdowns. Because the registry includes only public metadata—never API keys—the UI can validate selections and assemble request payloads safely in the browser.

Stage 3: Force-Disable Enforcement

Operators can globally disable specific providers using <CAP>_<PREFIX>_ENABLED=false patterns. When ASR_OPENAI_ENABLED is set to false, the configuration loader marks openai-whisper as unavailable. Client code in store/settings.ts automatically filters disabled providers and triggers fallback logic to select viable alternatives.

Stage 4: Server-Side API Execution

Actual external API calls happen in dedicated implementation files. lib/audio/tts-providers.ts and lib/audio/asr-providers.ts receive compact configuration objects containing providerId, apiKey, baseUrl, and modelId. These modules construct the appropriate HTTP requests—such as POST requests to OpenAI's /audio/transcriptions endpoint for Whisper or /audio/speech for TTS—and return normalized responses.

Stage 5: Client State Synchronization

The global store in lib/store/settings.ts tracks selected provider IDs (ttsProviderId, asrProviderId, llmProviderId) and validates them against registries using hasProviderId. When a provider becomes unavailable due to configuration changes, the sync logic automatically migrates to an enabled alternative, as verified in tests/settings-server-sync.test.ts.

Implementation Examples

Selecting a TTS Provider on the Client

Client components use the registry and global store to build provider requests without accessing credentials:

import { TTS_PROVIDERS } from '@/lib/audio/constants';
import { useStore } from '@/lib/store/settings';

// Get the current provider ID from the global store
const { ttsProviderId } = useStore(state => ({
  ttsProviderId: state.ttsProviderId,
}));

// Resolve the full provider config (used to build the request payload)
const provider = TTS_PROVIDERS[ttsProviderId as keyof typeof TTS_PROVIDERS];

// Example: send a request to the server-side TTS route
await fetch('/api/generate/tts', {
  method: 'POST',
  body: JSON.stringify({
    text: 'Hello world',
    providerId: provider.id,
    modelId: provider.defaultModelId,
  }),
});

Processing ASR Requests on the Server

Server API routes in app/api/transcription/route.ts resolve credentials and dispatch to provider implementations:

import { ASR_PROVIDERS } from '@/lib/audio/constants';
import { resolveASRBaseUrl, resolveASRApiKey } from '@/lib/server/provider-config';

// Assume the client sent `asrProviderId = 'openai-whisper'`
const providerId = 'openai-whisper';
const config = ASR_PROVIDERS[providerId as keyof typeof ASR_PROVIDERS];

// Resolve credentials and endpoint from env/YAML
const apiKey = resolveASRApiKey(providerId);
const baseUrl = resolveASRBaseUrl(providerId, config.defaultBaseUrl);

// Build a multipart/form-data request for OpenAI Whisper
const form = new FormData();
form.append('file', audioBlob);
form.append('model', config.defaultModelId);

await fetch(`${baseUrl}/audio/transcriptions`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
  body: form,
});

Disabling Providers via Environment Variables

Operators can force-disable specific backends without deploying code changes:

# In the deployment environment

export ASR_OPENAI_ENABLED=false   # disables the OpenAI Whisper backend fleet-wide

export TTS_AZURE_ENABLED=false    # disables Azure TTS

The provider-config loader processes these flags through DISABLE_ENV_MAPS, marking affected providers as unavailable across the entire deployment.

Key Files in the Provider System

• lib/server/provider-config.ts: Core configuration loader defining environment maps and disable logic
• lib/audio/constants.ts: Static TTS_PROVIDERS and ASR_PROVIDERS registries
• lib/audio/tts-providers.ts: Server-side TTS implementation for OpenAI, Azure, Qwen, and others
• lib/audio/asr-providers.ts: Server-side ASR implementation for Whisper, Qwen, and FunASR
• app/api/generate/tts/route.ts: Public API endpoint routing TTS requests to provider implementations
• app/api/transcription/route.ts: Public API endpoint handling speech-to-text requests
• lib/store/settings.ts: Global state management for selected provider IDs and fallback logic
• lib/hooks/use-audio-recorder.ts: Client-side hook for ASR provider selection and request building

Summary

• OpenMAIC implements a unified provider architecture that separates environment configuration from runtime execution
• Environment-variable mapping in lib/server/provider-config.ts translates deployment settings into canonical provider IDs
• Static registries in lib/audio/constants.ts provide both client and server with provider metadata while keeping secrets server-side
• Force-disable logic allows operators to turn off specific providers fleet-wide using *_ENABLED=false flags
• Server-side implementations handle all external API calls, ensuring API keys never reach the client
• Automatic fallback in the settings store ensures continuous operation when providers become unavailable

Frequently Asked Questions

How does OpenMAIC prevent API keys from leaking to the client?

All credential resolution happens server-side in lib/server/provider-config.ts through functions like resolveASRApiKey and resolveASRBaseUrl. The client only receives public metadata from TTS_PROVIDERS and ASR_PROVIDERS in lib/audio/constants.ts, which contain display names and supported models but never API keys or base URLs.

Can I use multiple providers for the same capability simultaneously?

Yes. The environment mapping system supports multiple providers per capability. You can configure both OPENAI_API_KEY and AZURE_TTS_API_KEY simultaneously. The client can then select between them, and the server will route requests to the appropriate implementation based on the providerId sent in the request payload.

What happens if a configured provider becomes unavailable?

The global store in lib/store/settings.ts implements validation logic using hasProviderId to check provider availability against the current registry. If a selected provider is disabled via environment variables or removed from configuration, the system automatically falls back to an available alternative, as tested in settings-server-sync.test.ts.

How do I add a new TTS or ASR provider to OpenMAIC?

To add a new provider, you must update three components: add the environment variable mapping to the appropriate *_ENV_MAP in lib/server/provider-config.ts, add the provider metadata to TTS_PROVIDERS or ASR_PROVIDERS in lib/audio/constants.ts, and implement the API-specific logic in lib/audio/tts-providers.ts or lib/audio/asr-providers.ts. Finally, expose any new capabilities through the API routes in app/api/generate/tts/route.ts or app/api/transcription/route.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}