How to Use `gevRealtime.js` for Voice Control in Gods‑Eye‑View

The gevRealtime.js module exposes the GevRealtimeController class that manages WebRTC sessions with OpenAI's Realtime API, handling microphone streaming, push-to-talk shortcuts, and tool execution for voice-controlled mapping interactions.

gevRealtime.js serves as the core voice control client for the Gods‑Eye‑View (GEV) mapping interface, bridging browser-based WebRTC streams with OpenAI's Realtime API to enable hands‑free navigation and data queries. Located at src/voice/gevRealtime.js, this self‑contained controller encapsulates session lifecycle management, cost metering, and radio audio ducking while exposing a simple global interface for UI integration. Understanding how to use gevRealtime.js for voice control requires familiarity with its initialization patterns, session methods, and event handling architecture.

Initializing the Voice Controller

The entry point for voice functionality is the initGevVoiceCommands factory function, which instantiates the controller and registers it as window.__gevVoiceCommands for global access.

According to the source code in src/voice/gevRealtime.js (lines 96‑103), this function wires the controller to core application objects including the map viewer, style manager, and data layer:

// src/main.js – bootstrap voice control
import { initGevVoiceCommands } from './voice/gevRealtime.js';

const controller = initGevVoiceCommands({
  viewer,                // the map viewer instance
  styleManager,         // style utilities
  dataManager,          // data‑layer manager
  sceneDirector: null,  // optional scene director
  annotations: null,    // optional annotation engine
});

After initialization, the controller automatically binds keyboard listeners for push‑to‑talk and exposes the GevRealtimeController instance globally, allowing UI components to trigger voice sessions without direct imports.

Starting and Stopping Voice Sessions

The start() method (defined at lines 36‑78 in src/voice/gevRealtime.js) orchestrates the full WebRTC handshake:

1. Fetches a short‑lived Realtime token via fetchRealtimeToken
2. Creates an RTCPeerConnection and data channel labeled oai-events
3. Attaches the local microphone stream and initializes the audio visualizer
4. Posts the SDP offer to https://api.openai.com/v1/realtime/calls

Conversely, stop() (lines 70‑95) tears down the connection, resets UI state, and pauses cost tracking.

To programmatically toggle voice control:

// Start a continuous listening session
window.__gevVoiceCommands?.start({ pushToTalk: false });

// Gracefully terminate the session
window.__gevVoiceCommands?.stop();

Push-to-Talk and Audio Management

For scenarios requiring manual activation, the controller implements bindPushToTalkShortcut() (lines 75‑122), which registers global keydown and keyup listeners on the Space key. When enabled, holding Space un‑mutes the microphone while preserving the underlying WebRTC connection, and releasing the key triggers releasePushToTalkKey() to silence the input.

The controller also manages background audio mixing through radio integration helpers (lines 84‑90). During active voice sessions, the system can duck or pause the audio‑radio layer to prevent interference, then restore playback after tool execution completes.

Enable push‑to‑talk mode programmatically via:

// Enable Space‑key activation
window.__gevVoiceCommands.pushToTalkMode = true;

Handling Realtime Events and Tool Calls

Incoming messages traverse the handleRealtimeEvent() method (lines 101‑200), which parses JSON events from the oai-events data channel. This handler:

• Updates UI status indicators for input_audio_buffer.speech_started and response.done events
• Deduplicates function calls to prevent redundant execution
• Forwards function_call payloads to the GevActionRunner (src/voice/gevActions.js)
• Returns tool outputs via sendToolOutput

For text‑based interaction without audio, use sendTextCommand() (lines 188‑197):

function sendCommand(text) {
  window.__gevVoiceCommands?.sendTextCommand(text);
}

Cost Tracking and Model Tiers

The controller enforces budget controls through integration with src/voice/voiceCost.js. After obtaining a Realtime token (lines 80‑86), the controller initializes a costTracker instance that meters usage per model tier and terminates the session gracefully if a user‑defined spend cap is exceeded.

Toggle between voice model tiers (e.g., "standard" vs. "premium") for subsequent sessions:

// Cycle to the next tier
window.__gevVoiceCommands?.toggleVoiceTier();

// Monitor current spend
const state = window.__gevVoiceCommands?.costTracker?.state();
console.log('USD spent:', state?.costUsd?.toFixed(2));

The cost state persists via VOICE_TIER_STORAGE_KEY, ensuring preferences survive page reloads.

Summary

• initGevVoiceCommands bootstraps the controller and exposes it globally as window.__gevVoiceCommands
• start() and stop() manage WebRTC lifecycles, microphone access, and data channel initialization
• Push‑to‑talk binds to the Space key via bindPushToTalkShortcut(), with programmatic toggling available
• handleRealtimeEvent() processes Realtime API streams and delegates tool calls to GevActionRunner
• Cost tracking integrates voiceCost.js to meter usage and enforce spending caps per session

Frequently Asked Questions

How do I enable push-to-talk mode programmatically?

Set the pushToTalkMode property to true on the global controller instance. This activates the Space‑key listeners defined in bindPushToTalkShortcut() (lines 75‑122) without requiring UI button interaction. Users can then hold Space to transmit audio and release to mute.

What happens when the cost cap is reached during a voice session?

The costTracker monitors cumulative spend in real‑time. When the user‑defined cap is exceeded, the controller automatically invokes stop() to terminate the WebRTC connection and resets the UI state, preventing further API charges. Current spend is accessible via costTracker.state().costUsd.

Can I send text commands instead of using voice input?

Yes. The sendTextCommand() method (lines 188‑197) accepts a string parameter and forwards it through the Realtime API's data channel as a text input event. This allows chat‑style interaction or scripted command injection alongside microphone input.

How does the controller integrate with the map viewer actions?

When the Realtime API emits function_call events, handleRealtimeEvent() forwards them to the GevActionRunner defined in src/voice/gevActions.js. This runner maps voice intents—such as "zoom to location" or "highlight layer"—to concrete viewer methods using the dependencies injected during initGevVoiceCommands.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}