# How to Implement Voice Processing Capabilities Using transcribe_audio and speak_text in Heurist Agent Framework

> Implement voice processing in Heurist Agent Framework with transcribe_audio for speech-to-text and speak_text for text-to-speech. Learn how to use these functions and set OPENAI_API_KEY.

- Repository: [Heurist/heurist-agent-framework](https://github.com/heurist-network/heurist-agent-framework)
- Tags: how-to-guide
- Published: 2026-03-03

---

**To implement voice processing in the Heurist Agent Framework, use the `transcribe_audio` function in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py) for OpenAI Whisper speech-to-text and the `speak_text` function for OpenAI TTS text-to-speech, ensuring you set the `OPENAI_API_KEY` environment variable.**

The Heurist Agent Framework provides native voice processing capabilities that enable agents to convert speech to text and generate spoken responses. This functionality is implemented through two high-level utilities in the [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py) module, which wrap OpenAI's Whisper and Text-to-Speech APIs. By integrating these functions, developers can build conversational agents that handle voice messages in platforms like Telegram or custom voice interfaces.

## Prerequisites: OpenAI API Configuration

Before using voice processing functions, you must configure your OpenAI API credentials. Both `transcribe_audio` and `speak_text` rely on a lazy-initialized OpenAI client that reads the `OPENAI_API_KEY` environment variable.

Create a `.env` file in your project root:

```bash
OPENAI_API_KEY=sk-your-api-key-here

```

If the environment variable is missing, the framework raises a clear runtime error during the first call to either function, preventing accidental execution without credentials.

## Core Voice Processing Functions

The [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py) module contains the synchronous implementations for audio transcription and speech generation. These functions handle file I/O, API communication, and error logging.

### Transcribing Audio with transcribe_audio

The `transcribe_audio` function sends audio files to OpenAI's Whisper model and returns the recognized text. Located at lines 31-38 in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py), this function accepts a file path string, opens the file in binary mode, and posts it to the `whisper-1` model endpoint.

```python
from core.voice import transcribe_audio

# Transcribe a local audio file (OGG, MP3, WAV supported)

transcribed_text = transcribe_audio("/path/to/voice_message.ogg")
print(f"User said: {transcribed_text}")

```

The function returns the `transcription.text` attribute from the OpenAI response. Errors during the API call are logged and re-raised to allow upstream handling.

### Generating Speech with speak_text

The `speak_text` function converts text strings into spoken audio using OpenAI's TTS model. Implemented at lines 43-64 in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py), this function uses the `tts-1` model with the `alloy` voice by default.

```python
from core.voice import speak_text

# Generate speech from text

audio_file_path = speak_text("Your request has been processed successfully.")
print(f"Audio saved to: {audio_file_path}")

```

The function generates a unique filename using an 8-character SHA-1 hash of a random integer, saves the MP3 to the `audio/` directory within the repository, and returns a `Path` object pointing to the saved file. The audio content is streamed directly from the API response to minimize memory usage.

## Async Integration with MediaHandler

For asynchronous applications, the framework provides `MediaHandler` in [`core/components/media_handler.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/components/media_handler.py). This component wraps the synchronous voice functions with async methods, enabling non-blocking voice processing in agent workflows.

### Using MediaHandler for Asynchronous Processing

The `MediaHandler` class exposes `transcribe_audio` and `text_to_speech` methods that delegate to the underlying voice utilities while allowing integration with async LLM calls.

```python
from core.components.media_handler import MediaHandler
from core.llm_provider import LLMProvider
import asyncio

async def process_voice_interaction(audio_file_path):
    # Initialize handler with any compatible LLM provider

    handler = MediaHandler(LLMProvider())
    
    # Step 1: Transcribe voice to text (async wrapper)

    user_text = await handler.transcribe_audio(audio_file_path)
    
    # Step 2: Generate response (example logic)

    response_text = f"Processing your request: {user_text}"
    
    # Step 3: Convert response to speech (async wrapper)

    speech_path = await handler.text_to_speech(response_text)
    
    return speech_path

# Run the async workflow

# asyncio.run(process_voice_interaction("audio/input.ogg"))

```

This pattern isolates I/O operations from core logic and makes voice capabilities reusable across different agent interfaces.

## Real-World Implementation: Telegram Voice Bot

The `TelegramAgent` in [`interfaces/telegram.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/interfaces/telegram.py) demonstrates a complete voice processing pipeline. Located at lines 55-82, the `handle_voice` method shows how to integrate transcription and speech generation in a production chatbot.

### Processing Voice Messages in TelegramAgent

The implementation downloads incoming voice notes, transcribes them using the `MediaHandler` integration, and routes the text through the standard message pipeline.

```python

# Example pattern from interfaces/telegram.py

async def handle_voice(self, update, context):
    # Extract file ID from Telegram voice message

    file_id = update.message.voice.file_id
    file = await context.bot.get_file(file_id)
    
    # Setup local audio directory

    audio_dir = Path(__file__).parents[1] / "audio"
    audio_dir.mkdir(exist_ok=True)
    voice_path = audio_dir / f"{file_id}.ogg"
    
    # Download voice note locally

    await file.download_to_drive(voice_path)
    await update.message.reply_text("Voice note received. Processing…")
    
    # Transcribe using MediaHandler (delegates to transcribe_audio)

    user_message = await self.transcribe_audio(voice_path)
    
    # Process through standard message handler

    text_response, image_url, _ = await self.handle_message(user_message)
    
    # Send reply (text or image)

    if image_url:
        await update.message.reply_photo(photo=image_url)
    elif text_response:
        await update.message.reply_text(text_response)
        
    # Optional: Convert text_response to speech using speak_text

    # speech_file = speak_text(text_response)

    # await update.message.reply_voice(voice=open(speech_file, 'rb'))

```

This pattern demonstrates error handling, file management, and the seamless transition from voice input to text processing and back to voice output.

## Summary

- **Environment Setup**: Export `OPENAI_API_KEY` before using voice functions; the framework validates this at runtime in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py).
- **Synchronous Functions**: Use `transcribe_audio(file_path)` in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py) for Whisper transcription and `speak_text(text)` for TTS generation to MP3 files in the `audio/` directory.
- **Async Integration**: Leverage `MediaHandler` in [`core/components/media_handler.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/components/media_handler.py) to wrap voice operations for non-blocking agent workflows.
- **Production Example**: Reference [`interfaces/telegram.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/interfaces/telegram.py) for a complete implementation handling voice message downloads, transcription, and response generation in a Telegram bot context.

## Frequently Asked Questions

### What audio file formats are supported by transcribe_audio?

The `transcribe_audio` function delegates to OpenAI's Whisper API, which supports common formats including **OGG**, **MP3**, **WAV**, **MPEG**, and **MP4**. The function opens files in binary mode and streams them directly to the API, so any format compatible with OpenAI's `whisper-1` model endpoint will work without local conversion.

### Do I need to initialize the OpenAI client manually before calling these functions?

No manual initialization is required. Both `transcribe_audio` and `speak_text` use an internal `_get_client()` helper that implements **lazy initialization**. On the first call, it creates a singleton `OpenAI` client using `os.getenv("OPENAI_API_KEY")`. Subsequent calls reuse the same client instance, ensuring efficient connection handling while requiring only the environment variable to be set.

### How does the async MediaHandler differ from the direct voice functions?

The `MediaHandler` class in [`core/components/media_handler.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/components/media_handler.py) provides **async wrappers** around the synchronous `transcribe_audio` and `speak_text` functions. While the core functions in [`core/voice.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/core/voice.py) block during API calls, `MediaHandler.transcribe_audio()` and `MediaHandler.text_to_speech()` can be awaited, allowing them to integrate with async agent loops, Telegram handlers, or FastAPI endpoints without blocking the event loop.

### Where are generated audio files stored when using speak_text?

The `speak_text` function saves generated MP3 files to the **`audio/`** directory relative to the repository root. It generates unique filenames using an 8-character SHA-1 hash of a random integer (e.g., `a3f7b2c1.mp3`) to prevent collisions. The function returns a `Path` object pointing to the saved file, making it easy to reference the audio for playback or transmission to users.