# OpenMAIC Audio Providers: Complete Guide to TTS and ASR Integrations in lib/audio

> Explore OpenMAIC's lib/audio for TTS and ASR integrations. Discover 10 TTS and 7 ASR providers including OpenAI, Azure, Qwen, Doubao, and FunASR.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: complete-guide
- Published: 2026-09-02

---

**OpenMAIC integrates 10 TTS providers and 7 ASR providers through a factory pattern in `lib/audio`, supporting everything from OpenAI and Azure to specialized Chinese services like Qwen, Doubao, and FunASR.**

The OpenMAIC framework provides a unified audio subsystem for conversational AI agents. Located at `THU-MAIC/OpenMAIC`, the `lib/audio` module abstracts Text-to-Speech and Automatic Speech Recognition behind provider-agnostic factories, allowing developers to swap services without changing application code.

## TTS Providers in lib/audio

The TTS factory in [`lib/audio/tts-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/tts-providers.ts) supports **10 distinct providers**, ranging from global APIs to browser-native synthesis.

### Cloud TTS Services

**OpenAI TTS** – Uses the `/audio/speech` endpoint with support for multiple voice presets and speed adjustments.

**Azure TTS** – Sends SSML payloads to Azure Speech Service for fine-grained prosody control.

**ElevenLabs TTS** – Direct integration with ElevenLabs API for high-quality voice cloning and multilingual synthesis.

**MiniMax TTS** – Synchronous HTTP API to MiniMax's speech synthesis service.

### Chinese-Language TTS Providers

**GLM TTS** – Calls the GLM API for Chinese-optimized speech generation.

**Qwen TTS** – DashScope Qwen3 TTS Flash with built-in voice-clone support.

**Doubao TTS** – Volcengine Seed-TTS 2.0, accepting either plan keys or `appId:accessKey` authentication.

**VoxCPM TTS** – Flexible backend supporting Nano-VLLM, Python-API, and VLLM-Omni deployments.

### Specialized and Local Options

**Lemonade TTS** – OpenAI-compatible `/audio/speech` endpoint for self-hosted or alternative providers.

**Browser Native TTS** – Web Speech API for client-side synthesis without server roundtrips.

## ASR Providers in lib/audio

The ASR factory in [`lib/audio/asr-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/asr-providers.ts) exposes **7 provider options** for speech-to-text conversion.

### Cloud ASR Services

**OpenAI Whisper** – Vercel AI SDK integration with OpenAI's Whisper API (`whisper-1` model).

**Azure ASR** – Fast Transcription REST API for batch and streaming recognition.

**Qwen ASR** – DashScope Qwen3 ASR Flash optimized for Mandarin and mixed-language audio.

### OpenAI-Compatible ASR Options

**FunASR** – WAV-only OpenAI-compatible endpoint from Alibaba's FunASR project.

**Lemonade ASR** – OpenAI-compatible endpoint for consistent API patterns across providers.

**Custom OpenAI-Compatible ASR** – Generic adapter for any provider exposing `/audio/transcriptions`, including SiliconFlow and LocalAI.

### Browser-Based Alternative

**Browser Native ASR** – Web Speech API for real-time, client-side transcription.

## Working with the TTS and ASR Factories

Both factories follow identical patterns: import the generator function, construct a configuration object, and await the result.

### Generating Speech with generateTTS

```typescript
import { generateTTS } from '@/lib/audio/tts-providers';
import type { TTSModelConfig } from '@/lib/audio/types';

const cfg: TTSModelConfig = {
  providerId: 'elevenlabs-tts',
  apiKey: 'YOUR_ELEVENLABS_API_KEY',
  voice: 'en_us_001',
  speed: 1.0,
  format: 'mp3',
};

const result = await generateTTS(cfg, 'Hello, OpenMAIC!');
// result.audio → Uint8Array of audio data
// result.format → 'mp3'

```

### Transcribing Audio with transcribeAudio

```typescript
import { transcribeAudio } from '@/lib/audio/asr-providers';
import type { ASRModelConfig } from '@/lib/audio/types';

// audioBlob: Blob or Buffer of recorded audio
const cfg: ASRModelConfig = {
  providerId: 'openai-whisper',
  apiKey: 'YOUR_OPENAI_API_KEY',
  modelId: 'whisper-1',
};

const { text } = await transcribeAudio(cfg, audioBlob);
console.log('Transcription:', text);

```

### Runtime Provider Switching

```typescript
import { getCurrentTTSConfig, getCurrentASRConfig } from '@/lib/audio';

const ttsConfig = await getCurrentTTSConfig();
const asrConfig = await getCurrentASRConfig();

const speech = await generateTTS(ttsConfig, 'Dynamic provider demo');
const { text } = await transcribeAudio(asrConfig, speech.audio);

```

## Key Files in lib/audio

| File | Purpose |
|------|---------|
| [`lib/audio/tts-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/tts-providers.ts) | TTS factory implementation (`generateTTS`) and provider registry |
| [`lib/audio/asr-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/asr-providers.ts) | ASR factory implementation (`transcribeAudio`) and provider registry |
| [`lib/audio/constants.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/constants.ts) | Provider configuration objects (IDs, default URLs, required keys) |
| [`lib/audio/types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/types.ts) | TypeScript definitions: `TTSModelConfig`, `ASRModelConfig`, provider unions |
| [`lib/audio/use-tts-preview.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/use-tts-preview.ts) | React hook for UI-based TTS preview |
| [`lib/audio/agent-voice.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/agent-voice.ts) | Voice selection and registration for agent configurations |

## Summary

- **10 TTS providers** are available in [`lib/audio/tts-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/tts-providers.ts), covering cloud APIs, Chinese services, and browser-native synthesis
- **7 ASR providers** are available in [`lib/audio/asr-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/asr-providers.ts), including Whisper, Azure, Qwen, FunASR, and generic OpenAI-compatible endpoints
- Both subsystems use **factory functions** (`generateTTS`, `transcribeAudio`) with runtime provider selection via `providerId`
- **Type-safe configurations** are enforced through `TTSModelConfig` and `ASRModelConfig` interfaces
- Provider-specific logic is isolated; the rest of OpenMAIC remains **provider-agnostic**

## Frequently Asked Questions

### How do I add a new TTS or ASR provider to OpenMAIC?

Extend the provider union type in [`lib/audio/types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/types.ts), add configuration constants in [`lib/audio/constants.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/audio/constants.ts), and implement the provider logic in the respective factory file ([`tts-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tts-providers.ts) or [`asr-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/asr-providers.ts)). The factory pattern ensures your new provider integrates without changes to consuming code.

### Can I use multiple providers simultaneously in one application?

Yes. Each call to `generateTTS` or `transcribeAudio` accepts an independent configuration object. Applications can route requests to different providers based on user preferences, language requirements, or fallback scenarios.

### What is the difference between OpenAI-compatible and native providers?

OpenAI-compatible providers (Lemonade TTS/ASR, FunASR, custom ASR) use identical endpoint structures and request formats to OpenAI's API. Native providers use service-specific protocols—Azure's SSML for TTS, Azure's Fast Transcription for ASR, Qwen's DashScope endpoints, etc. OpenAI-compatible providers simplify migration and self-hosting.

### Does OpenMAIC support real-time streaming for TTS or ASR?

The analysis of `lib/audio` source files shows synchronous HTTP-based APIs. Browser Native TTS and ASR via Web Speech API provide client-side streaming. For server-side streaming, check the specific provider implementation in [`tts-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tts-providers.ts) or [`asr-providers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/asr-providers.ts)—some backends like Azure and MiniMax may support streaming through extended options not shown in the base factory interface.