How to Use the Audio Transcription API Across Different Providers in aisuite

Initialize the aisuite.Client with your provider configurations, then call client.audio.transcriptions.create() using a provider:model string to transcribe audio files through OpenAI, Deepgram, or Google with automatic parameter translation and consistent result formats.

The aisuite library from Andrew Ng provides a unified abstraction layer for audio transcription services, eliminating the need to learn disparate SDKs for each provider. By implementing a provider-agnostic interface in aisuite/client.py, the library routes requests to OpenAI Whisper, Deepgram Nova, or Google Speech-to-Text while normalizing parameters and results. This guide demonstrates how to use the audio transcription API across different providers in aisuite using the actual source code implementation and architectural patterns found in the repository.

The Unified Audio Transcription Interface

At the core of aisuite's audio capabilities is the Audio class exposed through Client.audio, which provides a Transcriptions subclass for speech-to-text operations. Rather than instantiating provider-specific clients, you interact with a single method signature regardless of the backend service.

The create method in aisuite/client.py accepts a model identifier string in the format provider:model_name, such as openai:whisper-1 or deepgram:nova-2. This string determines which provider implementation handles the request, while the method signature remains constant across all providers:

result = client.audio.transcriptions.create(
    model="openai:whisper-1",
    file="path/to/audio.wav",
    language="en",
    prompt="Custom vocabulary context"
)

All providers return a unified TranscriptionResult data structure defined in aisuite/framework/message.py, ensuring that downstream code processes results identically regardless of whether OpenAI, Deepgram, or Google generated the transcription.

Parameter Validation and Provider Mapping

Before invoking a provider, aisuite validates and transforms parameters using the ParamValidator class located in aisuite/framework/asr_params.py. This component handles three critical tasks:

• Mapping common parameters to provider-specific names (e.g., converting language to language_code for Google)
• Validating against provider whitelists defined in PROVIDER_PARAMS to prevent invalid API calls
• Applying extra parameter modes (strict, warn, or permissive) to handle unknown keys

The extra_param_mode configuration determines how the client handles parameters not recognized by the target provider. In strict mode, unknown parameters raise a ValueError immediately. In warn mode, the client logs a warning but proceeds. In permissive mode, all parameters pass through to the provider API unchecked.

The mapping logic resides in ParamValidator._transform_value, which consults provider-specific dictionaries to translate standard AISuite parameter names into the format required by each underlying API.

Provider-Specific Implementations

The ProviderFactory in aisuite/provider.py instantiates concrete provider classes (OpenaiProvider, DeepgramProvider, GoogleProvider) based on the model prefix. Each provider implements an Audio subclass with Transcriptions methods that handle API-specific payload construction.

OpenAI Provider

Located in aisuite/providers/openai_provider.py, the OpenAI implementation forwards requests directly to the native client.audio.transcriptions.create method. Parameter mapping remains largely one-to-one, with common parameters like language, prompt, and temperature passing through unchanged.

Key provider-specific parameters include response_format, timestamp_granularities, and stream for enabling real-time transcription.

Deepgram Provider

The Deepgram implementation in aisuite/providers/deepgram_provider.py prepares raw byte payloads for the Deepgram SDK v5. While the language parameter maps directly, Deepgram offers unique capabilities accessible through provider-specific parameters:

• punctuate for automatic punctuation
• diarize for speaker identification
• smart_format for automatic formatting
• interim_results for streaming partial transcripts

Google Provider

The Google provider in aisuite/providers/google_provider.py constructs a RecognitionConfig object for the Google Speech-to-Text API. This requires the most aggressive parameter mapping:

• language transforms to language_code using the GOOGLE_LANGUAGE_MAP to expand two-letter codes (e.g., en becomes en-US)
• prompt becomes speech_contexts wrapped as JSON
• Standard parameters map to encoding, sample_rate_hertz, and enable_automatic_punctuation

Batch and Streaming Transcription Modes

AISuite supports both synchronous batch processing and asynchronous streaming transcription across compatible providers.

Batch Transcription

The standard create method processes entire audio files and returns complete TranscriptionResult objects containing the full text, segments, and metadata. This mode works identically across all providers:

result = client.audio.transcriptions.create(
    model="google:gemini-1.5-flash",
    file="audio.flac",
    language="en",
    enable_word_time_offsets=True
)
print(result.text)
print(result.words)  # Available when requested

Streaming Implementation

For real-time transcription, providers implement create_stream_output, an async generator yielding StreamingTranscriptionChunk objects. When you pass stream=True, the client routes to the streaming implementation instead of the batch method.

OpenAI, Deepgram, and Google all support streaming through this unified interface:

async for chunk in client.audio.transcriptions.create_stream_output(
    model="openai:whisper-1",
    file="live_audio.wav",
    language="en",
    stream=True
):
    print(f"{'FINAL' if chunk.is_final else 'PART'}: {chunk.text}")

Complete Implementation Examples

Basic Batch Transcription Across Providers

Configure multiple providers in a single client instance and switch between them by changing the model string:

from aisuite import Client

client = Client(
    provider_configs={
        "openai": {"api_key": "sk-..."},
        "deepgram": {"api_key": "DG..."},
        "google": {
            "project_id": "my-gcp-proj",
            "region": "us-central1",
            "application_credentials": "/path/to/key.json",
        },
    }
)

# OpenAI Whisper

result = client.audio.transcriptions.create(
    model="openai:whisper-1",
    file="speech.mp3",
    language="en",
)
print(result.text)

# Deepgram Nova-2 with provider-specific features

result = client.audio.transcriptions.create(
    model="deepgram:nova-2",
    file="speech.wav",
    language="en",
    punctuate=True,
    diarize=True,
)
print(result.segments)

# Google with language code expansion

result = client.audio.transcriptions.create(
    model="google:gemini-1.5-flash",
    file="speech.flac",
    language="en",  # Auto-expands to "en-US"

    prompt="medical terminology",
)
print(result.words)

Streaming Transcription with Async Iteration

import asyncio
from aisuite import Client

async def stream_transcription():
    client = Client(provider_configs={"openai": {"api_key": "sk-..."}})
    async for chunk in client.audio.transcriptions.create_stream_output(
        model="openai:whisper-1",
        file="long_speech.wav",
        language="en",
        stream=True,
    ):
        print(f"{'FINAL' if chunk.is_final else 'PART'}: {chunk.text}")

asyncio.run(stream_transcription())

Controlling Parameter Validation Strictness

# Strict mode rejects unknown parameters

client_strict = Client(extra_param_mode="strict")
try:
    client_strict.audio.transcriptions.create(
        model="openai:whisper-1",
        file="speech.mp3",
        unknown_param=123,
    )
except ValueError as e:
    print(f"Strict mode caught: {e}")

# Permissive mode passes unknown parameters to the API

client_perm = Client(extra_param_mode="permissive")
result = client_perm.audio.transcriptions.create(
    model="openai:whisper-1",
    file="speech.mp3",
    unknown_param=123,  # Sent to OpenAI API without error

)

Error Handling and Result Normalization

Provider-specific errors are wrapped as ASRError exceptions (or LLMError for chat operations) to present a consistent exception interface. This allows you to catch transcription failures uniformly regardless of whether the underlying error originated from OpenAI's rate limits, Deepgram's authentication issues, or Google's quota restrictions.

The TranscriptionResult object in aisuite/framework/message.py normalizes response structures across providers. While OpenAI returns simple text objects and Deepgram returns complex segment arrays, aisuite exposes these through standardized attributes like result.text, result.segments, and result.words where supported.

Summary

• Unified Interface: Use client.audio.transcriptions.create() with a provider:model string to access OpenAI, Deepgram, or Google transcription services through identical method signatures.
• Automatic Parameter Mapping: The ParamValidator in aisuite/framework/asr_params.py translates common parameters like language to provider-specific names (e.g., language_code for Google) and validates against provider whitelists.
• Flexible Validation Modes: Configure extra_param_mode as strict, warn, or permissive to control how the client handles unknown parameters.
• Streaming Support: Use create_stream_output() with stream=True for real-time transcription, yielding StreamingTranscriptionChunk objects across all supported providers.
• Consistent Results: All providers return TranscriptionResult objects defined in aisuite/framework/message.py, normalizing text, segments, and timing information into a standard format.

Frequently Asked Questions

How do I switch between transcription providers without changing my code?

Change the model parameter string from provider:model_name to target a different backend. For example, changing openai:whisper-1 to deepgram:nova-2 routes the request through Deepgram instead of OpenAI, while keeping the same client.audio.transcriptions.create() call and result handling code. Ensure the target provider is configured in your Client initialization.

What is the difference between strict and permissive parameter modes?

Strict mode raises a ValueError immediately when you pass parameters not recognized by the provider's whitelist, preventing accidental API errors. Permissive mode passes all unknown parameters through to the underlying API, allowing access to beta features or provider-specific parameters not yet mapped in aisuite. Warn mode (the default) logs a warning but continues execution.

How does streaming transcription work across different providers?

Streaming uses the async create_stream_output() method, which returns an async generator yielding StreamingTranscriptionChunk objects. Each chunk contains partial text and an is_final boolean indicating whether the segment is complete. OpenAI, Deepgram, and Google all implement this interface, though the underlying streaming protocols differ significantly between providers.

Why does Google Speech-to-Text convert my two-letter language codes?

The Google provider in aisuite/providers/google_provider.py automatically expands standard two-letter codes (like en or es) to Google-specific regional codes (like en-US or es-ES) using the GOOGLE_LANGUAGE_MAP dictionary. This ensures compatibility with Google's API requirements while allowing you to use standard language codes consistently across all providers in your aisuite code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}