How to Implement Voice Assistants Using Whisper and LLaVA: A Complete Multimodal Guide

Combine OpenAI Whisper for speech-to-text transcription with LLaVA's multimodal capabilities to build a voice assistant that understands both spoken commands and visual inputs.

Building a voice assistant capable of processing speech and images requires integrating specialized AI components. According to the aishwaryanr/awesome-generative-ai-guide repository, you can implement voice assistants using Whisper and LLaVA by connecting Whisper's transcription pipeline with LLaVA's vision-language reasoning. This guide walks through the complete implementation based on the reference project documented in resources/60_ai_projects.md.

Architecture Overview

The multimodal voice assistant follows a sequential data flow where audio input undergoes transcription before multimodal processing. As documented in the repository's project entry "AI Voice Assistant App using Multimodal LLM 'Llava' and Whisper" found at resources/60_ai_projects.md lines 68-85, the architecture consists of:

• Whisper: Converts raw audio into text transcripts using local speech recognition
• LLaVA: Processes the transcribed text alongside optional image inputs to generate contextual responses
• Optional TTS: Converts text responses back to speech for voice interaction

Both models operate independently, allowing you to swap Whisper for alternative STT solutions or replace LLaVA with newer multimodal architectures.

Prerequisites and Model Selection

Select model variants based on your hardware constraints and latency requirements. The reference implementation recommends:

• Whisper: Use tiny or base variants for real-time streaming, small for balanced accuracy, or large for maximum transcription quality
• LLaVA: The liuhaotian/llava-1.5-7b checkpoint provides robust multimodal reasoning while fitting within single GPU memory constraints
• Hardware: Load both models into GPU memory simultaneously to avoid allocation overhead during inference

Step-by-Step Implementation

Step 1: Capture Audio Input

Capture microphone input at 16kHz to match Whisper's expected sample rate:

import sounddevice as sd
import numpy as np

def record(duration=5, samplerate=16000):
    return sd.rec(int(duration * samplerate), samplerate=samplerate, channels=1, dtype='float32')

Step 2: Transcribe Speech with Whisper

Load the Whisper model once at startup and transcribe captured audio:

import whisper

model = whisper.load_model("small")  # Choose tiny/base/small/large based on latency budget

def transcribe(waveform, sr=16000):
    result = model.transcribe(np.squeeze(waveform), fp16=False)
    return result["text"]

Step 3: Process Visual Inputs (Optional)

For image-based queries, encode visuals as base64 strings to include in LLaVA's prompt structure:

from PIL import Image
import base64
import io

def encode_image(path):
    img = Image.open(path).convert("RGB")
    buffered = io.BytesIO()
    img.save(buffered, format="PNG")
    return base64.b64encode(buffered.getvalue()).decode()

Step 4: Generate Responses with LLaVA

Initialize LLaVA using Hugging Face transformers with automatic device mapping:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("liuhaotian/llava-1.5-7b")
model = AutoModelForCausalLM.from_pretrained(
    "liuhaotian/llava-1.5-7b",
    device_map="auto",
    torch_dtype=torch.float16
)

def llava_query(text, image_b64=None):
    payload = {"question": text}
    if image_b64:
        payload["image"] = image_b64
    prompt = tokenizer.apply_chat_template([payload], tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    generated_ids = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

LLaVA expects a JSON-like prompt structure containing a question field and optional image field with base64-encoded PNG data.

Step 5: Convert Text to Speech

Add optional voice output using a local TTS engine:

import pyttsx3

tts = pyttsx3.init()

def speak(text):
    tts.say(text)
    tts.runAndWait()

Step 6: Integrate the Full Pipeline

Combine all components into a cohesive workflow:

# Capture and transcribe audio

audio = record(duration=4)
user_text = transcribe(audio)

# Optional: include image for multimodal reasoning

img_b64 = encode_image("sample.jpg")
response = llava_query(user_text, img_b64)

# Output results

print("Assistant:", response)
speak(response)  # Optional voice output

Complete Working Example

The repository references a full implementation in the Multimodal-AI-App-using-Llava-7B project. This external repository contains:

• app.py: Streamlit interface that captures audio, loads Whisper, and calls LLaVA
• requirements.txt: Dependency specifications including whisper, transformers, torch, and pyttsx3
• README.md: Quick-start instructions and model download commands

As noted in resources/60_ai_projects.md lines 82-85, this project includes both a YouTube tutorial video and the complete starter code implementing the Whisper+LLaVA pipeline.

Optimization Strategies

• Model persistence: Load Whisper and LLaVA once at application startup rather than instantiating per request to eliminate GPU memory allocation overhead
• Real-time processing: Deploy whisper.cpp with streaming support for continuous transcription instead of fixed-duration recording
• Safety measures: Filter LLaVA outputs through a moderation layer before TTS conversion to prevent inappropriate spoken responses
• Batch vs. streaming: Use smaller Whisper models (tiny/base) for real-time assistants and larger models (small/large) for batch transcription accuracy

Summary

• Whisper provides state-of-the-art local speech-to-text transcription without requiring external APIs, maintaining user privacy
• LLaVA enables sophisticated multimodal reasoning that combines transcribed speech with visual inputs for context-aware responses
• The implementation sequence follows: audio capture → Whisper transcription → optional image encoding → LLaVA inference → optional TTS output
• Reference code and video tutorials are available in the aishwaryanr/awesome-generative-ai-guide repository under the voice assistant project entry at resources/60_ai_projects.md lines 68-85

Frequently Asked Questions

Can I run this voice assistant without a GPU?

While both models support CPU inference, GPU acceleration is essential for practical use. Whisper runs significantly slower on CPU, and LLaVA-7B requires approximately 14GB of memory that typically exceeds standard RAM capacities. For CPU-only deployment, use quantized 4-bit or 8-bit versions of LLaVA and the tiny Whisper variant.

How do I handle real-time streaming instead of fixed-duration recording?

Replace the fixed-duration record() function with voice activity detection (VAD) that triggers transcription when speech is detected. Use whisper.cpp in streaming mode or implement circular audio buffers that feed chunks to Whisper while maintaining the model in GPU memory to minimize latency between speech and response generation.

What image formats does LLaVA support in this implementation?

The reference code converts all images to PNG format before base64 encoding to ensure compatibility with LLaVA's tokenizer. While LLaVA models can theoretically process various image formats, the implementation in Multimodal-AI-App-using-Llava-7B specifically handles PNG encoding through PIL's Image.convert("RGB") and save(buffered, format="PNG") methods.

Where can I find the starter code and tutorial mentioned in this guide?

The complete implementation resources are linked in resources/60_ai_projects.md (lines 82-85) of the aishwaryanr/awesome-generative-ai-guide repository. This includes a YouTube tutorial video walking through environment setup and the GitHub repository containing the Streamlit application with app.py, requirements.txt, and setup instructions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}