How Free-Claude-Code Implements Voice Note Transcription Functionality with OpenAI Whisper

Free-Claude-Code provides a single transcribe_audio function in messaging/transcription.py that converts OGG, MP3, WAV, and other audio formats to text using either a local OpenAI Whisper model or the NVIDIA NIM cloud service.

The voice note transcription functionality in free-claude-code enables automatic speech-to-text conversion for voice messages received across multiple messaging platforms. According to the Alishahryar1/free-claude-code repository source code, the implementation supports both local inference via Hugging Face Transformers and remote processing through NVIDIA's Riva ASR service, configurable through environment variables.

Architecture Overview

The transcription system centers on the public function transcribe_audio defined in messaging/transcription.py (lines 29-67). This function accepts a file path, MIME type, and optional parameters for model selection and compute device, returning a plain-text transcription string.

The architecture follows a backend-selection pattern:

• Local execution: Uses the Hugging Face transformers pipeline with OpenAI Whisper checkpoints running on CPU or CUDA
• Cloud execution: Uses the NVIDIA NIM (Riva) gRPC API for high-throughput inference via riva.client

Both paths share common preprocessing steps including file validation and audio resampling.

Configuration and Validation

Global Settings

Configuration resides in config/settings.py (lines 169-181), where the application reads:

• voice_note_enabled: Master toggle for the feature
• whisper_device: Compute target ("cpu", "cuda", or "nvidia_nim")
• whisper_model: Model shorthand (e.g., "base", "large-v3") or full Hugging Face identifier
• hf_token: Optional authentication token for model downloads

The settings validator ensures only supported device strings are accepted and enforces NVIDIA API key presence when using the NIM backend.

Input Validation

Before processing, transcribe_audio performs strict validation at lines 29-37:

• File existence: Raises FileNotFoundError if the path is invalid
• Size limits: Enforces MAX_AUDIO_SIZE_BYTES (25 MiB), raising ValueError for oversized files
• Device compatibility: Validates "cpu" and "cuda" locally; delegates "nvidia_nim" to the remote handler

Transcription Backends

Local Whisper Pipeline (CPU/CUDA)

For local inference, the system implements a lazy-loaded pipeline pattern in messaging/transcription.py (lines 45-67):

1. Model Resolution: Short names like "base" or "tiny" map to full Hugging Face identifiers via _MODEL_MAP
2. Pipeline Caching: _get_pipeline constructs a transformers pipeline on first use and caches it in _pipeline_cache keyed by (model_id, device)
3. Audio Loading: _load_audio uses librosa to resample audio to 16 kHz, the required sampling rate for Whisper (lines 48-54)
4. Inference: The pipeline executes with generate_kwargs={"language": "en", "task": "transcribe"} and extracts the "text" field (lines 56-66)

If whisper_device="cuda" but CUDA is unavailable, the system gracefully falls back to CPU with a logged warning.

NVIDIA NIM Cloud Service

When configured for "nvidia_nim", the function routes to _transcribe_nim (lines 70-92 and 118-150):

1. Model Lookup: Maps model names to function IDs and language codes via _NIM_MODEL_MAP
2. gRPC Client: Creates a riva.client.Auth object with SSL and bearer token authentication
3. Service Initialization: Instantiates ASRService targeting grpc.nvcf.nvidia.com:443
4. Recognition: Sends raw audio bytes via offline_recognize with RecognitionConfig containing the appropriate language code
5. Result Extraction: Safely retrieves response.results[0].alternatives[0].transcript (lines 122-128)

If the backend returns no speech segments, the function returns the placeholder "(no speech detected)" (lines 64-67).

Integration with Messaging Platforms

The transcription function integrates into platform-specific handlers through lazy imports:

• Telegram: messaging/platforms/telegram.py (lines 612-618) calls transcribe_audio when processing voice note attachments
• Discord: messaging/platforms/discord.py (lines 213-218) triggers the same function for voice message events

This design keeps the transcription logic decoupled from platform-specific protocols while providing a unified speech-to-text interface across all supported integrations.

Code Examples

Basic Local Transcription

The default configuration uses CPU inference with the "base" model:

from pathlib import Path
from messaging.transcription import transcribe_audio

audio_path = Path("voice_note.ogg")
text = transcribe_audio(audio_path, mime_type="audio/ogg")
print(text)  # → "Hello world"

GPU-Accelerated Transcription

For faster inference on NVIDIA GPUs:

text = transcribe_audio(
    Path("voice_note.wav"),
    mime_type="audio/wav",
    whisper_device="cuda",
    whisper_model="large-v3",
)

This requires torch with CUDA support installed. The function automatically caches the pipeline for subsequent calls.

Cloud-Based Transcription with NVIDIA NIM

To offload processing to NVIDIA's cloud infrastructure:

text = transcribe_audio(
    Path("voice_note.m4a"),
    mime_type="audio/m4a",
    whisper_device="nvidia_nim",
    whisper_model="nvidia/parakeet-ctc-1.1b-asr",
)

Prerequisites include setting VOICE_NOTE_ENABLED=true, WHISPER_DEVICE=nvidia_nim, and a valid NVIDIA_NIM_API_KEY in the environment configuration.

Error Handling

The function exposes specific exception types for robust error management:

try:
    txt = transcribe_audio(Path("big_file.ogg"), "audio/ogg")
except FileNotFoundError:
    print("Audio file not found")
except ValueError as e:
    print(f"Validation failed: {e}")
except ImportError as e:
    print(f"Missing dependency: {e}")

These error paths are exercised in the test suite at tests/messaging/test_transcription.py.

Summary

• Single Entry Point: The transcribe_audio function in messaging/transcription.py provides a unified interface for all voice note transcription
• Dual Backend Support: Choose between local Whisper (CPU/CUDA) via Hugging Face Transformers or cloud-based NVIDIA NIM via gRPC
• Smart Caching: Local pipelines cache between requests to avoid reloading models
• Strict Validation: 25 MiB file size limits and format checking prevent processing errors
• Platform Agnostic: Integrated with Telegram and Discord handlers through clean imports

Frequently Asked Questions

What audio formats does free-claude-code support for transcription?

The system supports OGG, MP3, MP4, WAV, and M4A files. The _load_audio function uses librosa to normalize all inputs to 16 kHz mono audio, ensuring compatibility with Whisper's requirements regardless of the original format or sample rate.

How does the voice note transcription handle large audio files?

Files exceeding MAX_AUDIO_SIZE_BYTES (25 MiB) trigger a ValueError during the validation phase in messaging/transcription.py (lines 29-37). This limit prevents memory exhaustion on local deployments and ensures compliance with NVIDIA NIM API constraints when using the cloud backend.

Can I use a custom Whisper model not included in the default model map?

Yes. While the _MODEL_MAP provides shortcuts like "base" and "large-v3", you can pass any full Hugging Face model identifier (e.g., "openai/whisper-medium") directly to the whisper_model parameter. The system will attempt to load the specified checkpoint via the Hugging Face Hub.

What happens if the transcription detects no speech in the audio?

When Whisper or the NVIDIA NIM backend returns an empty result, the function returns the string "(no speech detected)" rather than an empty string. This ensures downstream handlers receive a valid string while clearly indicating that no transcribable content was found.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}