How to Implement Voice Processing Capabilities Using transcribe_audio and speak_text in Heurist Agent Framework

To implement voice processing in the Heurist Agent Framework, use the transcribe_audio function in core/voice.py for OpenAI Whisper speech-to-text and the speak_text function for OpenAI TTS text-to-speech, ensuring you set the OPENAI_API_KEY environment variable.

The Heurist Agent Framework provides native voice processing capabilities that enable agents to convert speech to text and generate spoken responses. This functionality is implemented through two high-level utilities in the core/voice.py module, which wrap OpenAI's Whisper and Text-to-Speech APIs. By integrating these functions, developers can build conversational agents that handle voice messages in platforms like Telegram or custom voice interfaces.

Prerequisites: OpenAI API Configuration

Before using voice processing functions, you must configure your OpenAI API credentials. Both transcribe_audio and speak_text rely on a lazy-initialized OpenAI client that reads the OPENAI_API_KEY environment variable.

Create a .env file in your project root:

OPENAI_API_KEY=sk-your-api-key-here

If the environment variable is missing, the framework raises a clear runtime error during the first call to either function, preventing accidental execution without credentials.

Core Voice Processing Functions

The core/voice.py module contains the synchronous implementations for audio transcription and speech generation. These functions handle file I/O, API communication, and error logging.

Transcribing Audio with transcribe_audio

The transcribe_audio function sends audio files to OpenAI's Whisper model and returns the recognized text. Located at lines 31-38 in core/voice.py, this function accepts a file path string, opens the file in binary mode, and posts it to the whisper-1 model endpoint.

from core.voice import transcribe_audio

# Transcribe a local audio file (OGG, MP3, WAV supported)

transcribed_text = transcribe_audio("/path/to/voice_message.ogg")
print(f"User said: {transcribed_text}")

The function returns the transcription.text attribute from the OpenAI response. Errors during the API call are logged and re-raised to allow upstream handling.

Generating Speech with speak_text

The speak_text function converts text strings into spoken audio using OpenAI's TTS model. Implemented at lines 43-64 in core/voice.py, this function uses the tts-1 model with the alloy voice by default.

from core.voice import speak_text

# Generate speech from text

audio_file_path = speak_text("Your request has been processed successfully.")
print(f"Audio saved to: {audio_file_path}")

The function generates a unique filename using an 8-character SHA-1 hash of a random integer, saves the MP3 to the audio/ directory within the repository, and returns a Path object pointing to the saved file. The audio content is streamed directly from the API response to minimize memory usage.

Async Integration with MediaHandler

For asynchronous applications, the framework provides MediaHandler in core/components/media_handler.py. This component wraps the synchronous voice functions with async methods, enabling non-blocking voice processing in agent workflows.

Using MediaHandler for Asynchronous Processing

The MediaHandler class exposes transcribe_audio and text_to_speech methods that delegate to the underlying voice utilities while allowing integration with async LLM calls.

from core.components.media_handler import MediaHandler
from core.llm_provider import LLMProvider
import asyncio

async def process_voice_interaction(audio_file_path):
    # Initialize handler with any compatible LLM provider

    handler = MediaHandler(LLMProvider())
    
    # Step 1: Transcribe voice to text (async wrapper)

    user_text = await handler.transcribe_audio(audio_file_path)
    
    # Step 2: Generate response (example logic)

    response_text = f"Processing your request: {user_text}"
    
    # Step 3: Convert response to speech (async wrapper)

    speech_path = await handler.text_to_speech(response_text)
    
    return speech_path

# Run the async workflow

# asyncio.run(process_voice_interaction("audio/input.ogg"))

This pattern isolates I/O operations from core logic and makes voice capabilities reusable across different agent interfaces.

Real-World Implementation: Telegram Voice Bot

The TelegramAgent in interfaces/telegram.py demonstrates a complete voice processing pipeline. Located at lines 55-82, the handle_voice method shows how to integrate transcription and speech generation in a production chatbot.

Processing Voice Messages in TelegramAgent

The implementation downloads incoming voice notes, transcribes them using the MediaHandler integration, and routes the text through the standard message pipeline.

# Example pattern from interfaces/telegram.py

async def handle_voice(self, update, context):
    # Extract file ID from Telegram voice message

    file_id = update.message.voice.file_id
    file = await context.bot.get_file(file_id)
    
    # Setup local audio directory

    audio_dir = Path(__file__).parents[1] / "audio"
    audio_dir.mkdir(exist_ok=True)
    voice_path = audio_dir / f"{file_id}.ogg"
    
    # Download voice note locally

    await file.download_to_drive(voice_path)
    await update.message.reply_text("Voice note received. Processing…")
    
    # Transcribe using MediaHandler (delegates to transcribe_audio)

    user_message = await self.transcribe_audio(voice_path)
    
    # Process through standard message handler

    text_response, image_url, _ = await self.handle_message(user_message)
    
    # Send reply (text or image)

    if image_url:
        await update.message.reply_photo(photo=image_url)
    elif text_response:
        await update.message.reply_text(text_response)
        
    # Optional: Convert text_response to speech using speak_text

    # speech_file = speak_text(text_response)

    # await update.message.reply_voice(voice=open(speech_file, 'rb'))

This pattern demonstrates error handling, file management, and the seamless transition from voice input to text processing and back to voice output.

Summary

• Environment Setup: Export OPENAI_API_KEY before using voice functions; the framework validates this at runtime in core/voice.py.
• Synchronous Functions: Use transcribe_audio(file_path) in core/voice.py for Whisper transcription and speak_text(text) for TTS generation to MP3 files in the audio/ directory.
• Async Integration: Leverage MediaHandler in core/components/media_handler.py to wrap voice operations for non-blocking agent workflows.
• Production Example: Reference interfaces/telegram.py for a complete implementation handling voice message downloads, transcription, and response generation in a Telegram bot context.

Frequently Asked Questions

What audio file formats are supported by transcribe_audio?

The transcribe_audio function delegates to OpenAI's Whisper API, which supports common formats including OGG, MP3, WAV, MPEG, and MP4. The function opens files in binary mode and streams them directly to the API, so any format compatible with OpenAI's whisper-1 model endpoint will work without local conversion.

Do I need to initialize the OpenAI client manually before calling these functions?

No manual initialization is required. Both transcribe_audio and speak_text use an internal _get_client() helper that implements lazy initialization. On the first call, it creates a singleton OpenAI client using os.getenv("OPENAI_API_KEY"). Subsequent calls reuse the same client instance, ensuring efficient connection handling while requiring only the environment variable to be set.

How does the async MediaHandler differ from the direct voice functions?

The MediaHandler class in core/components/media_handler.py provides async wrappers around the synchronous transcribe_audio and speak_text functions. While the core functions in core/voice.py block during API calls, MediaHandler.transcribe_audio() and MediaHandler.text_to_speech() can be awaited, allowing them to integrate with async agent loops, Telegram handlers, or FastAPI endpoints without blocking the event loop.

Where are generated audio files stored when using speak_text?

The speak_text function saves generated MP3 files to the audio/ directory relative to the repository root. It generates unique filenames using an 8-character SHA-1 hash of a random integer (e.g., a3f7b2c1.mp3) to prevent collisions. The function returns a Path object pointing to the saved file, making it easy to reference the audio for playback or transmission to users.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}