OpenMAIC Audio Providers: Complete Guide to TTS and ASR Integrations in lib/audio

OpenMAIC integrates 10 TTS providers and 7 ASR providers through a factory pattern in lib/audio, supporting everything from OpenAI and Azure to specialized Chinese services like Qwen, Doubao, and FunASR.

The OpenMAIC framework provides a unified audio subsystem for conversational AI agents. Located at THU-MAIC/OpenMAIC, the lib/audio module abstracts Text-to-Speech and Automatic Speech Recognition behind provider-agnostic factories, allowing developers to swap services without changing application code.

TTS Providers in lib/audio

The TTS factory in lib/audio/tts-providers.ts supports 10 distinct providers, ranging from global APIs to browser-native synthesis.

Cloud TTS Services

OpenAI TTS – Uses the /audio/speech endpoint with support for multiple voice presets and speed adjustments.

Azure TTS – Sends SSML payloads to Azure Speech Service for fine-grained prosody control.

ElevenLabs TTS – Direct integration with ElevenLabs API for high-quality voice cloning and multilingual synthesis.

MiniMax TTS – Synchronous HTTP API to MiniMax's speech synthesis service.

Chinese-Language TTS Providers

GLM TTS – Calls the GLM API for Chinese-optimized speech generation.

Qwen TTS – DashScope Qwen3 TTS Flash with built-in voice-clone support.

Doubao TTS – Volcengine Seed-TTS 2.0, accepting either plan keys or appId:accessKey authentication.

VoxCPM TTS – Flexible backend supporting Nano-VLLM, Python-API, and VLLM-Omni deployments.

Specialized and Local Options

Lemonade TTS – OpenAI-compatible /audio/speech endpoint for self-hosted or alternative providers.

Browser Native TTS – Web Speech API for client-side synthesis without server roundtrips.

ASR Providers in lib/audio

The ASR factory in lib/audio/asr-providers.ts exposes 7 provider options for speech-to-text conversion.

Cloud ASR Services

OpenAI Whisper – Vercel AI SDK integration with OpenAI's Whisper API (whisper-1 model).

Azure ASR – Fast Transcription REST API for batch and streaming recognition.

Qwen ASR – DashScope Qwen3 ASR Flash optimized for Mandarin and mixed-language audio.

OpenAI-Compatible ASR Options

FunASR – WAV-only OpenAI-compatible endpoint from Alibaba's FunASR project.

Lemonade ASR – OpenAI-compatible endpoint for consistent API patterns across providers.

Custom OpenAI-Compatible ASR – Generic adapter for any provider exposing /audio/transcriptions, including SiliconFlow and LocalAI.

Browser-Based Alternative

Browser Native ASR – Web Speech API for real-time, client-side transcription.

Working with the TTS and ASR Factories

Both factories follow identical patterns: import the generator function, construct a configuration object, and await the result.

Generating Speech with generateTTS

import { generateTTS } from '@/lib/audio/tts-providers';
import type { TTSModelConfig } from '@/lib/audio/types';

const cfg: TTSModelConfig = {
  providerId: 'elevenlabs-tts',
  apiKey: 'YOUR_ELEVENLABS_API_KEY',
  voice: 'en_us_001',
  speed: 1.0,
  format: 'mp3',
};

const result = await generateTTS(cfg, 'Hello, OpenMAIC!');
// result.audio → Uint8Array of audio data
// result.format → 'mp3'

Transcribing Audio with transcribeAudio

import { transcribeAudio } from '@/lib/audio/asr-providers';
import type { ASRModelConfig } from '@/lib/audio/types';

// audioBlob: Blob or Buffer of recorded audio
const cfg: ASRModelConfig = {
  providerId: 'openai-whisper',
  apiKey: 'YOUR_OPENAI_API_KEY',
  modelId: 'whisper-1',
};

const { text } = await transcribeAudio(cfg, audioBlob);
console.log('Transcription:', text);

Runtime Provider Switching

import { getCurrentTTSConfig, getCurrentASRConfig } from '@/lib/audio';

const ttsConfig = await getCurrentTTSConfig();
const asrConfig = await getCurrentASRConfig();

const speech = await generateTTS(ttsConfig, 'Dynamic provider demo');
const { text } = await transcribeAudio(asrConfig, speech.audio);

Key Files in lib/audio

File	Purpose
lib/audio/tts-providers.ts	TTS factory implementation (generateTTS) and provider registry
lib/audio/asr-providers.ts	ASR factory implementation (transcribeAudio) and provider registry
lib/audio/constants.ts	Provider configuration objects (IDs, default URLs, required keys)
lib/audio/types.ts	TypeScript definitions: TTSModelConfig, ASRModelConfig, provider unions
lib/audio/use-tts-preview.ts	React hook for UI-based TTS preview
lib/audio/agent-voice.ts	Voice selection and registration for agent configurations

Summary

• 10 TTS providers are available in lib/audio/tts-providers.ts, covering cloud APIs, Chinese services, and browser-native synthesis
• 7 ASR providers are available in lib/audio/asr-providers.ts, including Whisper, Azure, Qwen, FunASR, and generic OpenAI-compatible endpoints
• Both subsystems use factory functions (generateTTS, transcribeAudio) with runtime provider selection via providerId
• Type-safe configurations are enforced through TTSModelConfig and ASRModelConfig interfaces
• Provider-specific logic is isolated; the rest of OpenMAIC remains provider-agnostic

Frequently Asked Questions

How do I add a new TTS or ASR provider to OpenMAIC?

Extend the provider union type in lib/audio/types.ts, add configuration constants in lib/audio/constants.ts, and implement the provider logic in the respective factory file (tts-providers.ts or asr-providers.ts). The factory pattern ensures your new provider integrates without changes to consuming code.

Can I use multiple providers simultaneously in one application?

Yes. Each call to generateTTS or transcribeAudio accepts an independent configuration object. Applications can route requests to different providers based on user preferences, language requirements, or fallback scenarios.

What is the difference between OpenAI-compatible and native providers?

OpenAI-compatible providers (Lemonade TTS/ASR, FunASR, custom ASR) use identical endpoint structures and request formats to OpenAI's API. Native providers use service-specific protocols—Azure's SSML for TTS, Azure's Fast Transcription for ASR, Qwen's DashScope endpoints, etc. OpenAI-compatible providers simplify migration and self-hosting.

Does OpenMAIC support real-time streaming for TTS or ASR?

The analysis of lib/audio source files shows synchronous HTTP-based APIs. Browser Native TTS and ASR via Web Speech API provide client-side streaming. For server-side streaming, check the specific provider implementation in tts-providers.ts or asr-providers.ts—some backends like Azure and MiniMax may support streaming through extended options not shown in the base factory interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}