# VoiceStudio Dictation Widget Architecture: Tauri, FastAPI, and Sherpa-ONNX Integration

> Explore the VoiceStudio dictation widget architecture. Learn how Tauri, FastAPI, and Sherpa-ONNX integrate for live transcriptions and optional LLM refinement.

- Repository: [Palash Debnath/VoiceStudio](https://github.com/debpalash/VoiceStudio)
- Tags: architecture
- Published: 2026-09-04

---

**The VoiceStudio dictation widget architecture consists of a Tauri-based desktop shell, a React frontend, and a FastAPI backend that streams live transcriptions from Sherpa-ONNX via WebSocket, optionally refining results through a local LLM.**

The VoiceStudio dictation widget is a core component of the open-source [debpalash/VoiceStudio](https://github.com/debpalash/VoiceStudio) repository, bridging desktop system shortcuts with real-time speech recognition. Its architecture separates concerns across four distinct layers—from Rust-based global hotkeys to Python ASR services—enabling low-latency dictation with optional post-processing capabilities.

## Architecture Layers

The widget implements a layered architecture that moves from hardware input capture to neural network inference. Each layer operates independently but communicates through well-defined interfaces.

### Desktop Shell (Tauri v2 Rust)

The **desktop shell** resides in `frontend/src-tauri/` and serves as the system integration layer. A Rust side-car process ([`dictation_shortcut.rs`](https://github.com/debpalash/VoiceStudio/blob/main/dictation_shortcut.rs)) registers the system-wide shortcut (⌘+E on macOS, Ctrl+E on Linux/Windows) using Tauri’s global shortcut manager. When triggered, it emits an IPC message to the frontend via `handle.emit_all("dictation-start", {})`. The shell also handles text injection into active applications through [`dictation_output.rs`](https://github.com/debpalash/VoiceStudio/blob/main/dictation_output.rs).

### Frontend Interface (React and Vite)

The **frontend layer** in `frontend/src/` provides the visual interface via the `DictationWidget` React component. It consumes a WebSocket stream from the backend at `ws://localhost:3900/v1/audio/transcriptions/stream`. The utility module [`dictationCapture.js`](https://github.com/debpalash/VoiceStudio/blob/main/dictationCapture.js) manages the connection lifecycle, parsing partial and final transcription chunks while forwarding user commands (start/stop, language selection, LLM cleanup toggles) back to the API.

### Backend API (FastAPI)

The **backend API** exposes the streaming endpoint in [`backend/api/routers/dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/api/routers/dictation.py). The WebSocket route `WS /v1/audio/transcriptions/stream` accepts incoming connections and delegates audio processing to the Sherpa-ONNX service layer. The backend also serves preference data via `GET /dictation/prefs` and `POST /dictation/prefs`, defined in [`backend/api/schemas/dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/api/schemas/dictation.py).

### ASR Engine (Sherpa-ONNX)

The **engine layer** in [`backend/services/sherpa_dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/sherpa_dictation.py) implements the live-dictation pipeline. It initializes a `SherpaOnnxRecognizer` instance that processes microphone audio chunks in real-time, yielding partial results during speech and final utterances upon silence detection. This layer optionally interfaces with a local LLM for *dictation refinement*—post-processing raw ASR output to fix punctuation and formatting before delivery.

## Data Flow Pipeline

The **VoiceStudio dictation widget** processes audio through a six-stage pipeline:

1. **Shortcut Capture**: The Tauri side-car detects the global hotkey and emits the `dictation-start` event via IPC.
2. **UI Activation**: The React frontend receives the IPC message and mounts the `DictationWidget` overlay.
3. **WebSocket Handshake**: The widget opens a persistent connection to `ws://localhost:3900/v1/audio/transcriptions/stream`.
4. **Audio Streaming**: The backend router accepts the connection and initiates the Sherpa-ONNX recognizer, streaming audio chunks from the system microphone.
5. **Live Rendering**: Partial transcription chunks (type: `partial`) push to the React component in real-time, updating the overlay text.
6. **Finalization**: Upon stopping dictation, final results (type: `final`) return to the frontend. If enabled, the **dictation refinement** skill processes the text through a local LLM before the Tauri shell injects it into the active text field.

## Key Implementation Details

### Registering Global Shortcuts in Rust

The system-wide shortcut registration occurs in [`frontend/src-tauri/src/dictation_shortcut.rs`](https://github.com/debpalash/VoiceStudio/blob/main/frontend/src-tauri/src/dictation_shortcut.rs). The implementation uses Tauri’s `global_shortcut_manager` to bind the Meta+E combination:

```rust
// frontend/src-tauri/src/dictation_shortcut.rs
// Registers the global hotkey and notifies the UI
tauri::Builder::default()
    .setup(|app| {
        let handle = app.handle();
        let shortcut = Shortcut::new(Key::E, Modifiers::META); // Cmd+E on macOS
        app.global_shortcut_manager().register(shortcut, move || {
            handle.emit_all("dictation-start", {}).unwrap();
        })
    })
    .run(tauri::generate_context!())
    .expect("error while running tauri application");

```

### Consuming Live Transcription in React

The frontend utility in [`frontend/src/utils/dictationCapture.js`](https://github.com/debpalash/VoiceStudio/blob/main/frontend/src/utils/dictationCapture.js) establishes the WebSocket connection and handles message protocols:

```javascript
// frontend/src/utils/dictationCapture.js
export function startDictation(onPartial, onFinal) {
  const ws = new WebSocket('ws://localhost:3900/v1/audio/transcriptions/stream');
  ws.onmessage = ev => {
    const data = JSON.parse(ev.data);
    if (data.type === 'partial') onPartial(data.text);
    if (data.type === 'final')   onFinal(data.text);
  };
  ws.onerror = err => console.error('Dictation WS error', err);
  return () => ws.close(); // stop callback
}

```

### Streaming Endpoint in FastAPI

The backend router in [`backend/api/routers/dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/api/routers/dictation.py) manages WebSocket lifecycle and backpressure:

```python

# backend/api/routers/dictation.py

@router.websocket("/v1/audio/transcriptions/stream")
async def dictation_stream(ws: WebSocket):
    await ws.accept()
    async for chunk in sherpa_dictation.stream():
        await ws.send_json(chunk)

```

### Sherpa-ONNX Integration

The core recognition logic in [`backend/services/sherpa_dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/sherpa_dictation.py) wraps the Sherpa-ONNX runtime:

```python

# backend/services/sherpa_dictation.py

def stream():
    recognizer = SherpaOnnxRecognizer(model_path=MODEL_PATH)
    for audio_chunk in microphone.yield_chunks():
        result = recognizer.decode(audio_chunk)
        yield {"type": "partial", "text": result.partial}
        if result.is_final:
            yield {"type": "final", "text": result.text}

```

## Configuration and State Management

User preferences for the **dictation widget** persist in the `omnivoice_data/` directory. The backend exposes REST endpoints to retrieve and update settings such as enabled models, default languages, and LLM cleanup preferences. The `DictationWidget` queries these preferences on mount to initialize UI controls correctly.

## Summary

- The **VoiceStudio dictation widget architecture** spans four layers: a Tauri Rust shell for system integration, a React frontend for UI, a FastAPI backend for routing, and Sherpa-ONNX for speech recognition.
- **Global shortcuts** trigger the widget via IPC from [`dictation_shortcut.rs`](https://github.com/debpalash/VoiceStudio/blob/main/dictation_shortcut.rs), ensuring system-wide availability without focus requirements.
- **Live transcription** streams through a WebSocket connection to `WS /v1/audio/transcriptions/stream`, delivering partial results in real-time and final results upon completion.
- **Post-processing** occurs optionally through a local LLM refinement skill before text injection into active applications.
- State persistence uses the `omnivoice_data/` storage accessed via `/dictation/prefs` endpoints.

## Frequently Asked Questions

### What technologies power the VoiceStudio dictation widget?

The widget combines **Tauri v2** (Rust) for the desktop shell, **React with Vite** for the frontend, **FastAPI** for the backend, and **Sherpa-ONNX** for on-device automatic speech recognition. This stack enables cross-platform deployment with low-latency audio processing.

### How does the widget capture system-wide keyboard shortcuts?

The Tauri side-car in [`frontend/src-tauri/src/dictation_shortcut.rs`](https://github.com/debpalash/VoiceStudio/blob/main/frontend/src-tauri/src/dictation_shortcut.rs) registers a global hotkey (Meta+E) using Tauri’s `global_shortcut_manager`. When pressed, it emits a `dictation-start` event to the React frontend via IPC, triggering the widget overlay without requiring the application window to be focused.

### What ASR engine processes the live audio stream?

The backend delegates audio processing to **Sherpa-ONNX** through [`backend/services/sherpa_dictation.py`](https://github.com/debpalash/VoiceStudio/blob/main/backend/services/sherpa_dictation.py). This service streams partial transcription chunks while audio is being captured, then emits final results when silence detection indicates utterance completion.

### Can the dictation results be refined by an LLM?

Yes. The architecture supports optional **dictation refinement** where final ASR output passes through a local LLM before injection. This skill corrects punctuation, capitalization, and formatting based on user preferences stored in `omnivoice_data/` and accessed via the `/dictation/prefs` API routes.