# How Free-Claude-Code Implements Voice Note Transcription Functionality with OpenAI Whisper

> Discover how Free-Claude-Code uses OpenAI Whisper in its transcribe audio function to convert audio files to text. Explore local and cloud transcription options.

- Repository: [Ali Khokhar/free-claude-code](https://github.com/Alishahryar1/free-claude-code)
- Tags: how-to-guide
- Published: 2026-04-24

---

**Free-Claude-Code provides a single `transcribe_audio` function in [`messaging/transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/transcription.py) that converts OGG, MP3, WAV, and other audio formats to text using either a local OpenAI Whisper model or the NVIDIA NIM cloud service.**

The voice note transcription functionality in free-claude-code enables automatic speech-to-text conversion for voice messages received across multiple messaging platforms. According to the Alishahryar1/free-claude-code repository source code, the implementation supports both local inference via Hugging Face Transformers and remote processing through NVIDIA's Riva ASR service, configurable through environment variables.

## Architecture Overview

The transcription system centers on the public function **`transcribe_audio`** defined in **[`messaging/transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/transcription.py)** (lines 29-67). This function accepts a file path, MIME type, and optional parameters for model selection and compute device, returning a plain-text transcription string.

The architecture follows a backend-selection pattern:

- **Local execution**: Uses the Hugging Face `transformers` pipeline with OpenAI Whisper checkpoints running on CPU or CUDA
- **Cloud execution**: Uses the NVIDIA NIM (Riva) gRPC API for high-throughput inference via `riva.client`

Both paths share common preprocessing steps including file validation and audio resampling.

## Configuration and Validation

### Global Settings

Configuration resides in **[`config/settings.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/config/settings.py)** (lines 169-181), where the application reads:

- **`voice_note_enabled`**: Master toggle for the feature
- **`whisper_device`**: Compute target (`"cpu"`, `"cuda"`, or `"nvidia_nim"`)
- **`whisper_model`**: Model shorthand (e.g., `"base"`, `"large-v3"`) or full Hugging Face identifier
- **`hf_token`**: Optional authentication token for model downloads

The settings validator ensures only supported device strings are accepted and enforces NVIDIA API key presence when using the NIM backend.

### Input Validation

Before processing, `transcribe_audio` performs strict validation at lines 29-37:

- **File existence**: Raises `FileNotFoundError` if the path is invalid
- **Size limits**: Enforces **`MAX_AUDIO_SIZE_BYTES`** (25 MiB), raising `ValueError` for oversized files
- **Device compatibility**: Validates `"cpu"` and `"cuda"` locally; delegates `"nvidia_nim"` to the remote handler

## Transcription Backends

### Local Whisper Pipeline (CPU/CUDA)

For local inference, the system implements a lazy-loaded pipeline pattern in **[`messaging/transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/transcription.py)** (lines 45-67):

1. **Model Resolution**: Short names like `"base"` or `"tiny"` map to full Hugging Face identifiers via **`_MODEL_MAP`**
2. **Pipeline Caching**: **`_get_pipeline`** constructs a `transformers` pipeline on first use and caches it in **`_pipeline_cache`** keyed by `(model_id, device)`
3. **Audio Loading**: **`_load_audio`** uses **librosa** to resample audio to 16 kHz, the required sampling rate for Whisper (lines 48-54)
4. **Inference**: The pipeline executes with `generate_kwargs={"language": "en", "task": "transcribe"}` and extracts the `"text"` field (lines 56-66)

If `whisper_device="cuda"` but CUDA is unavailable, the system gracefully falls back to CPU with a logged warning.

### NVIDIA NIM Cloud Service

When configured for `"nvidia_nim"`, the function routes to **`_transcribe_nim`** (lines 70-92 and 118-150):

1. **Model Lookup**: Maps model names to function IDs and language codes via **`_NIM_MODEL_MAP`**
2. **gRPC Client**: Creates a `riva.client.Auth` object with SSL and bearer token authentication
3. **Service Initialization**: Instantiates `ASRService` targeting `grpc.nvcf.nvidia.com:443`
4. **Recognition**: Sends raw audio bytes via `offline_recognize` with `RecognitionConfig` containing the appropriate language code
5. **Result Extraction**: Safely retrieves `response.results[0].alternatives[0].transcript` (lines 122-128)

If the backend returns no speech segments, the function returns the placeholder **`"(no speech detected)"`** (lines 64-67).

## Integration with Messaging Platforms

The transcription function integrates into platform-specific handlers through lazy imports:

- **Telegram**: **[`messaging/platforms/telegram.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/platforms/telegram.py)** (lines 612-618) calls `transcribe_audio` when processing voice note attachments
- **Discord**: **[`messaging/platforms/discord.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/platforms/discord.py)** (lines 213-218) triggers the same function for voice message events

This design keeps the transcription logic decoupled from platform-specific protocols while providing a unified speech-to-text interface across all supported integrations.

## Code Examples

### Basic Local Transcription

The default configuration uses CPU inference with the `"base"` model:

```python
from pathlib import Path
from messaging.transcription import transcribe_audio

audio_path = Path("voice_note.ogg")
text = transcribe_audio(audio_path, mime_type="audio/ogg")
print(text)  # → "Hello world"

```

### GPU-Accelerated Transcription

For faster inference on NVIDIA GPUs:

```python
text = transcribe_audio(
    Path("voice_note.wav"),
    mime_type="audio/wav",
    whisper_device="cuda",
    whisper_model="large-v3",
)

```

This requires `torch` with CUDA support installed. The function automatically caches the pipeline for subsequent calls.

### Cloud-Based Transcription with NVIDIA NIM

To offload processing to NVIDIA's cloud infrastructure:

```python
text = transcribe_audio(
    Path("voice_note.m4a"),
    mime_type="audio/m4a",
    whisper_device="nvidia_nim",
    whisper_model="nvidia/parakeet-ctc-1.1b-asr",
)

```

Prerequisites include setting `VOICE_NOTE_ENABLED=true`, `WHISPER_DEVICE=nvidia_nim`, and a valid `NVIDIA_NIM_API_KEY` in the environment configuration.

### Error Handling

The function exposes specific exception types for robust error management:

```python
try:
    txt = transcribe_audio(Path("big_file.ogg"), "audio/ogg")
except FileNotFoundError:
    print("Audio file not found")
except ValueError as e:
    print(f"Validation failed: {e}")
except ImportError as e:
    print(f"Missing dependency: {e}")

```

These error paths are exercised in the test suite at **[`tests/messaging/test_transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/tests/messaging/test_transcription.py)**.

## Summary

- **Single Entry Point**: The `transcribe_audio` function in [`messaging/transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/transcription.py) provides a unified interface for all voice note transcription
- **Dual Backend Support**: Choose between local Whisper (CPU/CUDA) via Hugging Face Transformers or cloud-based NVIDIA NIM via gRPC
- **Smart Caching**: Local pipelines cache between requests to avoid reloading models
- **Strict Validation**: 25 MiB file size limits and format checking prevent processing errors
- **Platform Agnostic**: Integrated with Telegram and Discord handlers through clean imports

## Frequently Asked Questions

### What audio formats does free-claude-code support for transcription?

The system supports OGG, MP3, MP4, WAV, and M4A files. The `_load_audio` function uses librosa to normalize all inputs to 16 kHz mono audio, ensuring compatibility with Whisper's requirements regardless of the original format or sample rate.

### How does the voice note transcription handle large audio files?

Files exceeding `MAX_AUDIO_SIZE_BYTES` (25 MiB) trigger a `ValueError` during the validation phase in [`messaging/transcription.py`](https://github.com/Alishahryar1/free-claude-code/blob/main/messaging/transcription.py) (lines 29-37). This limit prevents memory exhaustion on local deployments and ensures compliance with NVIDIA NIM API constraints when using the cloud backend.

### Can I use a custom Whisper model not included in the default model map?

Yes. While the `_MODEL_MAP` provides shortcuts like `"base"` and `"large-v3"`, you can pass any full Hugging Face model identifier (e.g., `"openai/whisper-medium"`) directly to the `whisper_model` parameter. The system will attempt to load the specified checkpoint via the Hugging Face Hub.

### What happens if the transcription detects no speech in the audio?

When Whisper or the NVIDIA NIM backend returns an empty result, the function returns the string `"(no speech detected)"` rather than an empty string. This ensures downstream handlers receive a valid string while clearly indicating that no transcribable content was found.