How to Integrate Custom TTS or ASR Providers with OpenMAIC

OpenMAIC supports custom Text-to-Speech (TTS) and Automatic Speech Recognition (ASR) providers through a prefix-based identification system using custom-tts-* and custom-asr-* namespaces, with configuration management centralized in lib/store/settings.ts and runtime resolution handled by provider-specific modules in lib/audio/.

OpenMAIC is an open-source multimodal AI framework designed for extensible audio processing workflows. Whether connecting to proprietary cloud APIs or self-hosted speech models, the architecture provides standardized extension points for integrating custom TTS and ASR providers without modifying core framework logic. This guide examines the source code implementation in the THU-MAIC/OpenMAIC repository to demonstrate the precise steps for registering and configuring new speech services.

Understanding the Provider Identifier Convention

OpenMAIC distinguishes custom providers from built-in ones through strict naming conventions defined in lib/audio/types.ts. Custom TTS providers must use the prefix custom-tts- (for example, custom-tts-myhost or custom-tts-siliconflow), while custom ASR providers use the custom-asr- prefix (such as custom-asr-siliconflow).

The type system exports identifier types TTSProviderId and ASRProviderId, which include these custom prefixes along with built-in options like browser-native providers. Utility functions isKnownTTSProviderId() and isKnownASRProviderId() validate provider strings against these type definitions, ensuring any ID starting with the appropriate custom prefix is recognized as valid throughout the application. These validation guards are extensively tested in tests/audio/tts-provider-guard.test.ts and tests/audio/custom-asr.test.ts.

Configuration Storage and Validation

Provider configurations reside in the global settings store located at lib/store/settings.ts. The store maintains two primary configuration objects: ttsProvidersConfig and asrProvidersConfig, which map provider IDs to their respective settings structures.

Each custom provider configuration typically includes:

• apiKey: Authentication credentials for the external service
• baseUrl: Endpoint URL for API requests
• enabled: Boolean flag to activate or deactivate the provider
• customVoices (TTS only): Array of available voice options with id and name properties

The helper functions isTTSProviderConfigured() and isASRProviderConfigured() verify that a custom provider has valid required fields before allowing runtime usage. These guards check both the presence of the provider ID in the configuration store and the existence of essential parameters like API keys, preventing requests to improperly initialized endpoints.

Server-Side Provider Resolution

When OpenMAIC operates in server-side environments, the framework automatically resolves provider configurations from environment variables. The resolution logic in lib/audio/ reads variables formatted as TTS_<PROVIDER>_API_KEY and ASR_<PROVIDER>_API_KEY, mapping them to the corresponding custom provider slots.

Functions resolveServerTTSProviderId() and resolveServerASRProviderId() handle this initialization, respecting the custom-provider prefixes while maintaining fallback logic to browser-native providers when server configurations are missing or disabled. This automatic hydration ensures the system degrades gracefully if a custom provider loses its server-side credentials, as verified in tests/settings/settings-server-sync.test.ts.

Implementing a Custom Provider Module

To integrate a new speech service, you must implement a provider module that conforms to the existing interface contracts. The framework expects specific request handlers for TTS and ASR operations located in lib/audio/tts-providers.ts and lib/audio/asr-providers.ts.

Step 1: Configure the Provider Programmatically

First, register your custom provider configuration using the settings store methods. The following example demonstrates setting up a custom TTS provider:

import { useSettingsStore } from '@/lib/store/settings';

useSettingsStore.getState().setTTSProviderConfig('custom-tts-myhost', {
  apiKey: 'YOUR_API_KEY',
  baseUrl: 'https://my-tts-service.example.com/v1',
  enabled: true,
  customVoices: [{ id: 'voice-1', name: 'Demo Voice' }],
});

For ASR providers, use the corresponding configuration method:

import { useSettingsStore } from '@/lib/store/settings';

useSettingsStore.getState().setASRProviderConfig('custom-asr-siliconflow', {
  apiKey: 'YOUR_ASR_KEY',
  baseUrl: 'https://siliconflow-asr.example.com/api',
  enabled: true,
});

Step 2: Implement the Request Handler

Create provider logic within lib/audio/ that exports functions matching the framework's provider interface. Your implementation should read the stored configuration and perform HTTP calls to your external service.

Step 3: Register in Provider Registries

Add your provider to the central registries TTS_PROVIDERS or ASR_PROVIDERS exported from lib/audio/tts-providers.ts and lib/audio/asr-providers.ts respectively. This registration ensures the runtime dispatcher can locate your implementation when the corresponding provider ID is selected in the application settings.

Runtime Integration and Fallback Behavior

During execution, OpenMAIC routes audio requests through provider-specific dispatchers. The function applyGenerateTtsResultToScenes() (utilized in workbench synchronization workflows as tested in tests/workbench/tts-stage-sync.test.ts) retrieves the active ttsProviderId from settings and forwards synthesis requests to the appropriate implementation module.

For speech recognition, the transcribeAudio() function validates the active asrProviderId against configuration stores before invoking the custom provider's transcription logic:

import { transcribeAudio } from '@/lib/audio/asr-providers';

const audioBlob = await fetchAudioFile(); // Obtain WAV audio data
const result = await transcribeAudio(
  { providerId: 'custom-asr-siliconflow', language: 'en-US' },
  audioBlob,
);

console.log(result.transcript);

If a custom provider becomes unavailable or loses configuration (such as a removed API key), the system automatically falls back to browser-native TTS/ASR capabilities. This fallback mechanism is tested in tests/audio/provider-enablement.test.ts and tests/audio/disable-switches-selection.test.ts, ensuring the UI never references dead endpoints.

Summary

• OpenMAIC identifies custom providers using strict prefixes (custom-tts-* and custom-asr-*) defined in lib/audio/types.ts
• Configuration storage happens through lib/store/settings.ts with validation helpers isTTSProviderConfigured() and isASRProviderConfigured()
• Server-side initialization automatically reads environment variables formatted as TTS_<PROVIDER>_API_KEY and maps them to custom provider configs via resolveServerTTSProviderId()
• Implementation requires creating conformant modules under lib/audio/ and registering them in TTS_PROVIDERS or ASR_PROVIDERS registries
• Runtime dispatching uses applyGenerateTtsResultToScenes() for TTS and transcribeAudio() for ASR, with automatic fallback to native providers when custom services fail validation

Frequently Asked Questions

How does OpenMAIC validate that a custom provider is properly configured before use?

OpenMAIC uses guard functions isTTSProviderConfigured() and isASRProviderConfigured() to inspect the settings store before executing requests. These utilities verify that the provider ID exists in ttsProvidersConfig or asrProvidersConfig and that required fields like apiKey and baseUrl are non-empty strings. The validation logic prevents runtime errors by blocking requests to improperly initialized providers, as demonstrated in tests/audio/provider-enablement.test.ts.

What environment variables are required for server-side custom provider setup?

Server-side resolution expects environment variables following the pattern TTS_<PROVIDER>_API_KEY for text-to-speech and ASR_<PROVIDER>_API_KEY for speech recognition, where <PROVIDER> corresponds to your custom identifier suffix (the portion after custom-tts- or custom-asr-). The resolution functions in lib/audio/ automatically hydrate the settings store with these values on startup, eliminating the need for manual configuration in headless deployments.

Can I extend the provider configuration to include custom fields like authentication headers or model parameters?

Yes, you can extend the configuration schema in lib/audio/types.ts to include additional fields specific to your service. The settings store architecture supports arbitrary configuration objects within ttsProvidersConfig and asrProvidersConfig. When implementing your provider module in lib/audio/tts-providers.ts or lib/audio/asr-providers.ts, you can access these custom fields through the configuration object passed to your request handler functions.

What happens if my custom TTS provider API becomes unavailable during runtime?

OpenMAIC implements automatic fallback behavior that switches to browser-native TTS capabilities when a custom provider fails configuration validation or returns errors. The fallback is triggered by the provider guard logic tested in tests/audio/tts-provider-guard.test.ts and tests/audio/provider-enablement.test.ts, ensuring continuous functionality even when third-party services experience outages or credential expiration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}