VoiceStudio Dictation Widget Architecture: Tauri, FastAPI, and Sherpa-ONNX Integration

The VoiceStudio dictation widget architecture consists of a Tauri-based desktop shell, a React frontend, and a FastAPI backend that streams live transcriptions from Sherpa-ONNX via WebSocket, optionally refining results through a local LLM.

The VoiceStudio dictation widget is a core component of the open-source debpalash/VoiceStudio repository, bridging desktop system shortcuts with real-time speech recognition. Its architecture separates concerns across four distinct layers—from Rust-based global hotkeys to Python ASR services—enabling low-latency dictation with optional post-processing capabilities.

Architecture Layers

The widget implements a layered architecture that moves from hardware input capture to neural network inference. Each layer operates independently but communicates through well-defined interfaces.

Desktop Shell (Tauri v2 Rust)

The desktop shell resides in frontend/src-tauri/ and serves as the system integration layer. A Rust side-car process (dictation_shortcut.rs) registers the system-wide shortcut (⌘+E on macOS, Ctrl+E on Linux/Windows) using Tauri’s global shortcut manager. When triggered, it emits an IPC message to the frontend via handle.emit_all("dictation-start", {}). The shell also handles text injection into active applications through dictation_output.rs.

Frontend Interface (React and Vite)

The frontend layer in frontend/src/ provides the visual interface via the DictationWidget React component. It consumes a WebSocket stream from the backend at ws://localhost:3900/v1/audio/transcriptions/stream. The utility module dictationCapture.js manages the connection lifecycle, parsing partial and final transcription chunks while forwarding user commands (start/stop, language selection, LLM cleanup toggles) back to the API.

Backend API (FastAPI)

The backend API exposes the streaming endpoint in backend/api/routers/dictation.py. The WebSocket route WS /v1/audio/transcriptions/stream accepts incoming connections and delegates audio processing to the Sherpa-ONNX service layer. The backend also serves preference data via GET /dictation/prefs and POST /dictation/prefs, defined in backend/api/schemas/dictation.py.

ASR Engine (Sherpa-ONNX)

The engine layer in backend/services/sherpa_dictation.py implements the live-dictation pipeline. It initializes a SherpaOnnxRecognizer instance that processes microphone audio chunks in real-time, yielding partial results during speech and final utterances upon silence detection. This layer optionally interfaces with a local LLM for dictation refinement—post-processing raw ASR output to fix punctuation and formatting before delivery.

Data Flow Pipeline

The VoiceStudio dictation widget processes audio through a six-stage pipeline:

1. Shortcut Capture: The Tauri side-car detects the global hotkey and emits the dictation-start event via IPC.
2. UI Activation: The React frontend receives the IPC message and mounts the DictationWidget overlay.
3. WebSocket Handshake: The widget opens a persistent connection to ws://localhost:3900/v1/audio/transcriptions/stream.
4. Audio Streaming: The backend router accepts the connection and initiates the Sherpa-ONNX recognizer, streaming audio chunks from the system microphone.
5. Live Rendering: Partial transcription chunks (type: partial) push to the React component in real-time, updating the overlay text.
6. Finalization: Upon stopping dictation, final results (type: final) return to the frontend. If enabled, the dictation refinement skill processes the text through a local LLM before the Tauri shell injects it into the active text field.

Key Implementation Details

Registering Global Shortcuts in Rust

The system-wide shortcut registration occurs in frontend/src-tauri/src/dictation_shortcut.rs. The implementation uses Tauri’s global_shortcut_manager to bind the Meta+E combination:

// frontend/src-tauri/src/dictation_shortcut.rs
// Registers the global hotkey and notifies the UI
tauri::Builder::default()
    .setup(|app| {
        let handle = app.handle();
        let shortcut = Shortcut::new(Key::E, Modifiers::META); // Cmd+E on macOS
        app.global_shortcut_manager().register(shortcut, move || {
            handle.emit_all("dictation-start", {}).unwrap();
        })
    })
    .run(tauri::generate_context!())
    .expect("error while running tauri application");

Consuming Live Transcription in React

The frontend utility in frontend/src/utils/dictationCapture.js establishes the WebSocket connection and handles message protocols:

// frontend/src/utils/dictationCapture.js
export function startDictation(onPartial, onFinal) {
  const ws = new WebSocket('ws://localhost:3900/v1/audio/transcriptions/stream');
  ws.onmessage = ev => {
    const data = JSON.parse(ev.data);
    if (data.type === 'partial') onPartial(data.text);
    if (data.type === 'final')   onFinal(data.text);
  };
  ws.onerror = err => console.error('Dictation WS error', err);
  return () => ws.close(); // stop callback
}

Streaming Endpoint in FastAPI

The backend router in backend/api/routers/dictation.py manages WebSocket lifecycle and backpressure:

# backend/api/routers/dictation.py

@router.websocket("/v1/audio/transcriptions/stream")
async def dictation_stream(ws: WebSocket):
    await ws.accept()
    async for chunk in sherpa_dictation.stream():
        await ws.send_json(chunk)

Sherpa-ONNX Integration

The core recognition logic in backend/services/sherpa_dictation.py wraps the Sherpa-ONNX runtime:

# backend/services/sherpa_dictation.py

def stream():
    recognizer = SherpaOnnxRecognizer(model_path=MODEL_PATH)
    for audio_chunk in microphone.yield_chunks():
        result = recognizer.decode(audio_chunk)
        yield {"type": "partial", "text": result.partial}
        if result.is_final:
            yield {"type": "final", "text": result.text}

Configuration and State Management

User preferences for the dictation widget persist in the omnivoice_data/ directory. The backend exposes REST endpoints to retrieve and update settings such as enabled models, default languages, and LLM cleanup preferences. The DictationWidget queries these preferences on mount to initialize UI controls correctly.

Summary

• The VoiceStudio dictation widget architecture spans four layers: a Tauri Rust shell for system integration, a React frontend for UI, a FastAPI backend for routing, and Sherpa-ONNX for speech recognition.
• Global shortcuts trigger the widget via IPC from dictation_shortcut.rs, ensuring system-wide availability without focus requirements.
• Live transcription streams through a WebSocket connection to WS /v1/audio/transcriptions/stream, delivering partial results in real-time and final results upon completion.
• Post-processing occurs optionally through a local LLM refinement skill before text injection into active applications.
• State persistence uses the omnivoice_data/ storage accessed via /dictation/prefs endpoints.

Frequently Asked Questions

What technologies power the VoiceStudio dictation widget?

The widget combines Tauri v2 (Rust) for the desktop shell, React with Vite for the frontend, FastAPI for the backend, and Sherpa-ONNX for on-device automatic speech recognition. This stack enables cross-platform deployment with low-latency audio processing.

How does the widget capture system-wide keyboard shortcuts?

The Tauri side-car in frontend/src-tauri/src/dictation_shortcut.rs registers a global hotkey (Meta+E) using Tauri’s global_shortcut_manager. When pressed, it emits a dictation-start event to the React frontend via IPC, triggering the widget overlay without requiring the application window to be focused.

What ASR engine processes the live audio stream?

The backend delegates audio processing to Sherpa-ONNX through backend/services/sherpa_dictation.py. This service streams partial transcription chunks while audio is being captured, then emits final results when silence detection indicates utterance completion.

Can the dictation results be refined by an LLM?

Yes. The architecture supports optional dictation refinement where final ASR output passes through a local LLM before injection. This skill corrects punctuation, capitalization, and formatting based on user preferences stored in omnivoice_data/ and accessed via the /dictation/prefs API routes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}