How to Implement Audio Transcription with the Audio Class in aisuite

Call client.audio.transcriptions.create() with a "provider:model" string and an audio file path to transcribe speech using any supported provider like OpenAI, Google, or Deepgram without changing your application code.

The aisuite library provides a unified, provider-agnostic interface for audio transcription tasks. By using the Audio class exposed through Client.audio, you can implement speech-to-text functionality across multiple providers including OpenAI Whisper, Google Cloud Speech-to-Text, and Deepgram. This article explains how to implement audio transcription with the Audio class in aisuite using actual source code patterns from the andrewyng/aisuite repository.

Understanding the Audio API Architecture

The Audio API consists of three distinct layers that abstract provider complexity while maintaining flexibility.

The Client Façade Layer

In aisuite/client.py (around lines 36-48), the Audio class acts as the entry point. The Client exposes an audio attribute that returns this Audio instance, which in turn exposes a transcriptions sub-interface. When you call client.audio.transcriptions.create(), the client handles model string parsing, parameter validation, and provider delegation automatically.

The Provider Abstraction

The abstract contract is defined in aisuite/provider.py (around lines 26-45). Here, the base Audio class defines the structure that all providers must implement, including the nested Transcription class that specifies the create and create_stream_output method signatures. This ensures every provider implements the same interface for batch and streaming transcription.

Provider-Specific Implementations

Concrete implementations reside in provider-specific files. For example, aisuite/providers/openai_provider.py contains the OpenAIAudio.Transcriptions class that implements the actual API calls to OpenAI's Whisper model. Similarly, aisuite/providers/deepgram_provider.py implements DeepgramAudio.Transcriptions for Deepgram's speech-to-text API.

Prerequisites and Setup

Install aisuite with the specific provider extras you intend to use:

pip install "aisuite[openai]"  # or [google], [deepgram], etc.

Configure credentials via environment variables (e.g., OPENAI_API_KEY) or pass them explicitly through the provider_configs dictionary when initializing the Client.

Implementing Batch Audio Transcription

For standard transcription tasks where you process an entire audio file and receive the complete text, use the batch method. The model parameter uses the format "provider:model" (e.g., "openai:whisper-1"), which the Client validates and routes via ProviderFactory.

from aisuite import Client

# Initialize client - API key can be loaded from env var OPENAI_API_KEY

client = Client(provider_configs={"openai": {}})

# Transcribe audio file

result = client.audio.transcriptions.create(
    model="openai:whisper-1",
    file="speech.mp3",          # Can be a file path or file-like object

    language="en",              # Common ASR parameter

    prompt="Transcribe the meeting notes clearly."
)

print(result.text)  # result is a TranscriptionResponse from aisuite/framework/message.py

The ParamValidator in aisuite/framework/asr_params.py normalizes common parameters like language, prompt, and temperature before forwarding the request to the provider's concrete implementation.

Implementing Streaming Audio Transcription

For real-time processing or large files where you want incremental results, pass stream=True to receive chunks as they become available:

client = Client(provider_configs={"openai": {}})

# Stream chunks of transcription as they arrive

for chunk in client.audio.transcriptions.create(
    model="openai:whisper-1",
    file="lecture.wav",
    stream=True,                # Request streaming output

    response_format="verbose_json"
):
    # chunk is a StreamingTranscriptionChunk

    print(chunk.text, end="")

When stream=True, the client recognizes the flag and invokes the provider's create_stream_output method instead of the standard create method. This allows providers like Deepgram or OpenAI to stream partial results back to your application.

Working with Alternative Providers

Switching providers requires only changing the model string and configuration. The Client automatically loads the appropriate provider class via ProviderFactory.create_provider based on the prefix before the colon.

client = Client(
    provider_configs={
        "deepgram": {"api_key": "YOUR_DEEPGRAM_KEY"}
    }
)

result = client.audio.transcriptions.create(
    model="deepgram:nova-2",
    file="interview.mp3",
    language="en",
    punctuate=True,   # Deepgram-specific parameter

    diarize=True      # Deepgram-specific parameter

)

print(result.text)

Provider-specific kwargs like punctuate and diarize are passed through untouched after the ParamValidator extracts and validates common ASR parameters. This happens in aisuite/framework/asr_params.py, ensuring unknown parameters don't break the request while still allowing provider-native features.

Parameter Validation and Error Handling

Control how the client handles unknown or extra parameters using the extra_param_mode setting:

client = Client(extra_param_mode="warn")  # Default behavior - logs warning

result = client.audio.transcriptions.create(
    model="openai:whisper-1",
    file="audio.mp3",
    foobar="ignored"  # Unknown parameter triggers warning but call proceeds

)

Set extra_param_mode="strict" to raise an exception on unknown keys, or "permissive" to allow them silently. The ParamValidator.validate_and_map method in aisuite/framework/asr_params.py implements this logic, checking parameters against known ASR fields before delegation.

Summary

• The Audio class in aisuite/client.py provides a unified interface for transcription via client.audio.transcriptions.create()
• Model strings use the format "provider:model" (e.g., "openai:whisper-1"), parsed by ProviderFactory to load the correct provider implementation
• Batch transcription returns a TranscriptionResponse object containing the full text; streaming with stream=True yields StreamingTranscriptionChunk objects
• Provider implementations reside in files like aisuite/providers/openai_provider.py and aisuite/providers/deepgram_provider.py, each implementing Audio.Transcription.create() and optionally create_stream_output()
• Parameters are validated by ParamValidator in aisuite/framework/asr_params.py with configurable extra_param_mode settings for handling unknown keys

Frequently Asked Questions

What audio file formats does aisuite support?

aisuite delegates format handling to the underlying provider. OpenAI Whisper supports mp3, mp4, mpeg, mpga, m4a, wav, and webm. Deepgram supports over 40 formats including mp3, wav, and flac. The library passes your file object directly to the provider's API without intermediate conversion, so supported formats depend on your chosen provider's specifications.

How do I switch between transcription providers?

Change the model string from "openai:whisper-1" to "deepgram:nova-2" or "google:gemini-2.0-flash-exp". Ensure you have installed the appropriate provider extras (e.g., pip install "aisuite[deepgram]") and configured the API credentials. The Client automatically instantiates the correct provider class via ProviderFactory.create_provider based on the prefix before the colon, requiring zero changes to your transcription logic.

Can I use aisuite for real-time streaming transcription?

Yes, pass stream=True to client.audio.transcriptions.create(). This triggers the create_stream_output method in the provider's Audio.Transcription implementation. Providers like Deepgram return chunks as they become available through their streaming APIs, while others may buffer and simulate streaming depending on native capabilities. Each chunk returned is a StreamingTranscriptionChunk object as defined in aisuite/framework/message.py.

Where are provider-specific parameters validated?

The ParamValidator class in aisuite/framework/asr_params.py normalizes common ASR parameters like language, prompt, and temperature across all providers. Unknown parameters are handled according to the extra_param_mode setting: "warn" logs a warning (default), "strict" raises an exception, and "permissive" allows them silently. Provider-specific parameters like Deepgram's punctuate or diarize are passed through untouched after common parameter extraction, ensuring you can access native provider features without wrapper limitations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}