# How to Implement Voice Assistants Using Whisper and LLaVA: A Complete Multimodal Guide

> Build a voice assistant using Whisper and LLaVA. Transcribe speech with Whisper and enable multimodal understanding with LLaVA for advanced command processing. Get started today.

- Repository: [aishwaryanr/awesome-generative-ai-guide](https://github.com/aishwaryanr/awesome-generative-ai-guide)
- Tags: how-to-guide
- Published: 2026-06-21

---

**Combine OpenAI Whisper for speech-to-text transcription with LLaVA's multimodal capabilities to build a voice assistant that understands both spoken commands and visual inputs.**

Building a voice assistant capable of processing speech and images requires integrating specialized AI components. According to the `aishwaryanr/awesome-generative-ai-guide` repository, you can implement voice assistants using Whisper and LLaVA by connecting Whisper's transcription pipeline with LLaVA's vision-language reasoning. This guide walks through the complete implementation based on the reference project documented in [`resources/60_ai_projects.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/resources/60_ai_projects.md).

## Architecture Overview

The multimodal voice assistant follows a sequential data flow where audio input undergoes transcription before multimodal processing. As documented in the repository's project entry "AI Voice Assistant App using Multimodal LLM 'Llava' and Whisper" found at [`resources/60_ai_projects.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/resources/60_ai_projects.md) lines 68-85, the architecture consists of:

- **Whisper**: Converts raw audio into text transcripts using local speech recognition
- **LLaVA**: Processes the transcribed text alongside optional image inputs to generate contextual responses
- **Optional TTS**: Converts text responses back to speech for voice interaction

Both models operate independently, allowing you to swap Whisper for alternative STT solutions or replace LLaVA with newer multimodal architectures.

## Prerequisites and Model Selection

Select model variants based on your hardware constraints and latency requirements. The reference implementation recommends:

- **Whisper**: Use `tiny` or `base` variants for real-time streaming, `small` for balanced accuracy, or `large` for maximum transcription quality
- **LLaVA**: The `liuhaotian/llava-1.5-7b` checkpoint provides robust multimodal reasoning while fitting within single GPU memory constraints
- **Hardware**: Load both models into GPU memory simultaneously to avoid allocation overhead during inference

## Step-by-Step Implementation

### Step 1: Capture Audio Input

Capture microphone input at 16kHz to match Whisper's expected sample rate:

```python
import sounddevice as sd
import numpy as np

def record(duration=5, samplerate=16000):
    return sd.rec(int(duration * samplerate), samplerate=samplerate, channels=1, dtype='float32')

```

### Step 2: Transcribe Speech with Whisper

Load the Whisper model once at startup and transcribe captured audio:

```python
import whisper

model = whisper.load_model("small")  # Choose tiny/base/small/large based on latency budget

def transcribe(waveform, sr=16000):
    result = model.transcribe(np.squeeze(waveform), fp16=False)
    return result["text"]

```

### Step 3: Process Visual Inputs (Optional)

For image-based queries, encode visuals as base64 strings to include in LLaVA's prompt structure:

```python
from PIL import Image
import base64
import io

def encode_image(path):
    img = Image.open(path).convert("RGB")
    buffered = io.BytesIO()
    img.save(buffered, format="PNG")
    return base64.b64encode(buffered.getvalue()).decode()

```

### Step 4: Generate Responses with LLaVA

Initialize LLaVA using Hugging Face transformers with automatic device mapping:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("liuhaotian/llava-1.5-7b")
model = AutoModelForCausalLM.from_pretrained(
    "liuhaotian/llava-1.5-7b",
    device_map="auto",
    torch_dtype=torch.float16
)

def llava_query(text, image_b64=None):
    payload = {"question": text}
    if image_b64:
        payload["image"] = image_b64
    prompt = tokenizer.apply_chat_template([payload], tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    generated_ids = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

```

LLaVA expects a JSON-like prompt structure containing a `question` field and optional `image` field with base64-encoded PNG data.

### Step 5: Convert Text to Speech

Add optional voice output using a local TTS engine:

```python
import pyttsx3

tts = pyttsx3.init()

def speak(text):
    tts.say(text)
    tts.runAndWait()

```

### Step 6: Integrate the Full Pipeline

Combine all components into a cohesive workflow:

```python

# Capture and transcribe audio

audio = record(duration=4)
user_text = transcribe(audio)

# Optional: include image for multimodal reasoning

img_b64 = encode_image("sample.jpg")
response = llava_query(user_text, img_b64)

# Output results

print("Assistant:", response)
speak(response)  # Optional voice output

```

## Complete Working Example

The repository references a full implementation in the **Multimodal-AI-App-using-Llava-7B** project. This external repository contains:

- [`app.py`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/app.py): Streamlit interface that captures audio, loads Whisper, and calls LLaVA
- [`requirements.txt`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/requirements.txt): Dependency specifications including `whisper`, `transformers`, `torch`, and `pyttsx3`
- [`README.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/README.md): Quick-start instructions and model download commands

As noted in [`resources/60_ai_projects.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/resources/60_ai_projects.md) lines 82-85, this project includes both a YouTube tutorial video and the complete starter code implementing the Whisper+LLaVA pipeline.

## Optimization Strategies

- **Model persistence**: Load Whisper and LLaVA once at application startup rather than instantiating per request to eliminate GPU memory allocation overhead
- **Real-time processing**: Deploy [`whisper.cpp`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/whisper.cpp) with streaming support for continuous transcription instead of fixed-duration recording
- **Safety measures**: Filter LLaVA outputs through a moderation layer before TTS conversion to prevent inappropriate spoken responses
- **Batch vs. streaming**: Use smaller Whisper models (`tiny`/`base`) for real-time assistants and larger models (`small`/`large`) for batch transcription accuracy

## Summary

- **Whisper** provides state-of-the-art local speech-to-text transcription without requiring external APIs, maintaining user privacy
- **LLaVA** enables sophisticated multimodal reasoning that combines transcribed speech with visual inputs for context-aware responses
- The implementation sequence follows: audio capture → Whisper transcription → optional image encoding → LLaVA inference → optional TTS output
- Reference code and video tutorials are available in the `aishwaryanr/awesome-generative-ai-guide` repository under the voice assistant project entry at [`resources/60_ai_projects.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/resources/60_ai_projects.md) lines 68-85

## Frequently Asked Questions

### Can I run this voice assistant without a GPU?

While both models support CPU inference, GPU acceleration is essential for practical use. Whisper runs significantly slower on CPU, and LLaVA-7B requires approximately 14GB of memory that typically exceeds standard RAM capacities. For CPU-only deployment, use quantized 4-bit or 8-bit versions of LLaVA and the `tiny` Whisper variant.

### How do I handle real-time streaming instead of fixed-duration recording?

Replace the fixed-duration `record()` function with voice activity detection (VAD) that triggers transcription when speech is detected. Use [`whisper.cpp`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/whisper.cpp) in streaming mode or implement circular audio buffers that feed chunks to Whisper while maintaining the model in GPU memory to minimize latency between speech and response generation.

### What image formats does LLaVA support in this implementation?

The reference code converts all images to PNG format before base64 encoding to ensure compatibility with LLaVA's tokenizer. While LLaVA models can theoretically process various image formats, the implementation in `Multimodal-AI-App-using-Llava-7B` specifically handles PNG encoding through PIL's `Image.convert("RGB")` and `save(buffered, format="PNG")` methods.

### Where can I find the starter code and tutorial mentioned in this guide?

The complete implementation resources are linked in [`resources/60_ai_projects.md`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/resources/60_ai_projects.md) (lines 82-85) of the `aishwaryanr/awesome-generative-ai-guide` repository. This includes a YouTube tutorial video walking through environment setup and the GitHub repository containing the Streamlit application with [`app.py`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/app.py), [`requirements.txt`](https://github.com/aishwaryanr/awesome-generative-ai-guide/blob/main/requirements.txt), and setup instructions.