How the Gods Eye View Voice Control System Uses WebRTC and the OpenAI Realtime API

The Gods Eye View voice control system establishes a WebRTC peer connection with a bidirectional data channel to stream encrypted audio and exchange JSON events with the OpenAI Realtime API, enabling low-latency voice commands and tool execution.

The voice control feature in the bilawalsidhu/gods-eye-view repository implements a production-grade WebRTC pipeline that connects browser-based audio streams directly to OpenAI's Realtime API. Located within src/voice/, this architecture bypasses traditional HTTP polling in favor of persistent peer-to-peer media channels, delivering sub-second latency for conversational AI interactions within the 3D visualization interface.

WebRTC Architecture Overview

The voice control system operates through four distinct stages orchestrated by GevRealtimeController in src/voice/gevRealtime.js:

1. Token Acquisition – The client requests a JWT from /api/realtime/token that authorizes the session and specifies the target model.
2. Peer Connection Setup – Creation of an RTCPeerConnection with local microphone tracks and a data channel named "oai-events".
3. SDP Exchange – The client generates an SDP offer, transmits it to https://api.openai.com/v1/realtime/calls, and applies the returned SDP answer.
4. Real-time Messaging – Bidirectional JSON event flow over the data channel for conversation items, function calls, and system events.

This flow leverages the browser's native WebRTC stack to establish encrypted, low-latency communication without intermediary servers handling the media stream.

Token Acquisition and Session Initialization

Before establishing any peer connection, the controller must obtain authorization. The fetchRealtimeToken() function (lines 14-35 in src/voice/gevRealtime.js) retrieves a short-lived JWT from the application's backend:

// Internal flow within GevRealtimeController.start()
const token = await fetchRealtimeToken();
// Returns: { token: string, model: string }

The token response includes both the authentication credential and the specific OpenAI model identifier (e.g., gpt-4o-realtime-preview). This value is immediately passed to the voice cost tracker in src/voice/voiceCost.js to ensure accurate billing calculations against the correct rate table.

Establishing the WebRTC Peer Connection

The start() method in src/voice/gevRealtime.js (lines 23-49) manages the complex WebRTC initialization sequence:

// Create peer connection
this.pc = new RTCPeerConnection({
  iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
});

// Attach local microphone stream
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getTracks().forEach(track => this.pc.addTrack(track, stream));

// Initialize data channel for OpenAI events
this.dc = this.pc.createDataChannel('oai-events', { ordered: true });

Once the data channel and tracks are attached, the controller generates an SDP offer and completes the signaling exchange with OpenAI's servers (lines 71-97). The remote SDP answer is applied via this.pc.setRemoteDescription(), finalizing the encrypted media path.

Data Channel Protocol and Event Handling

All communication with the Realtime API occurs over the "oai-events" data channel using JSON payloads conforming to the OpenAI Realtime specification. The sendRealtimeEvent() method wraps outgoing messages, while handleRealtimeEvent() (lines 102-165 in src/voice/gevRealtime.js) parses and dispatches incoming events:

handleRealtimeEvent(event) {
  const payload = JSON.parse(event.data);
  
  switch (payload.type) {
    case 'input_audio_buffer.speech_started':
      // Mark user turn beginning
      break;
    case 'response.function_call_arguments.done':
      // Extract and execute tool calls
      this.runTool(payload);
      break;
    case 'response.done':
      // Finalize turn and trigger UI updates
      break;
  }
}

The data channel enables server-initiated function calls. When the model emits response.function_call_arguments.done, the controller extracts the function name and arguments through extractFunctionCalls(), then delegates execution to the tool runner in src/voice/gevActions.js.

Session Lifecycle and Connection State Management

The controller implements a robust state machine monitoring the RTCPeerConnection.connectionState. Critical logic resides in the connection state listener:

• Failed State: Immediately triggers fatalError() and terminates the session.
• Disconnected State: Initiates a grace period (DISCONNECT_GRACE_MS, 6 seconds) to tolerate temporary network fluctuations before calling fatalError().

The stop() method provides graceful teardown by incrementing an internal startEpoch to cancel in-flight operations, closing the data channel, stopping peer connection transceivers, and releasing microphone tracks:

// Graceful shutdown example
controller.stop({ preserveRadioPlayback: true });

Push-to-Talk vs. Click-to-Talk Interaction Modes

The system supports two distinct user interaction patterns managed through bindPushToTalkShortcut() and shouldHandlePushToTalkKeyDown:

Click-to-Talk: The UI button handler toggles the session state via controller.buttonHandler, starting or stopping the connection based on current state.

Push-to-Talk: Keyboard event listeners intercept Space keydown/keyup events. Holding Space enables the microphone stream (setMicrophoneEnabled(true)) and maintains the session only while depressed, suitable for quick commands without full session overhead.

Function Calling and Tool Execution

When the Realtime API returns function calls, GevRealtimeController invokes the action runner with abort signal support and session validation:

// Inside handleRealtimeEvent for function calls
const result = await this.runner(call.name, parsedArguments, {
  signal: toolController.signal,
  isCurrent: () => this.startEpoch === currentEpoch
});

The runner imported from src/voice/gevActions.js maps function names to application-specific operations:

// Example tool implementation in gevActions.js
export async function runTool(name, args, ctx) {
  switch (name) {
    case 'highlight_runway':
      await viewer.highlightRunway(args.runwayId);
      return { ok: true, action: name };
  }
}

Cost Tracking and Audio Visualization

Every session instantiates a cost tracker from src/voice/voiceCost.js using the model identifier returned during token acquisition. This ensures usage accounting against the correct pricing tier for the specific Realtime model deployed.

Simultaneously, the system runs dual audio visualizers built on the Web Audio API (AudioContext, AnalyserNode):

• Microphone Visualizer: Analyzes the local MediaStream to provide visual feedback during user speech.
• Assistant Visualizer: Processes the remote audio stream from the AI to indicate when the model is speaking.

Both visualizers update DOM elements via startVoiceVisualizer() and startAssistantVoiceVisualizer() to create synchronized visual feedback.

Practical Implementation Examples

Initialize the voice controller in your application entry point:

import { initGevVoiceCommands } from './voice/gevRealtime.js';

const controller = initGevVoiceCommands({
  viewer,
  styleManager,
  dataManager,
  sceneDirector,
  annotations
});

Send typed commands programmatically while the session is active:

controller.sendTextCommand('Show me the nearest runway');

Handle incoming function calls by registering tools in src/voice/gevActions.js:

export async function runTool(name, args, ctx) {
  if (name === 'navigate_to') {
    await ctx.viewer.flyTo(args.coordinates);
    return { status: 'success' };
  }
}

Summary

• The voice control system in bilawalsidhu/gods-eye-view uses WebRTC peer connections to establish encrypted, low-latency audio streams directly with the OpenAI Realtime API.
• Authentication requires a JWT from /api/realtime/token before initiating the SDP exchange with https://api.openai.com/v1/realtime/calls.
• The "oai-events" data channel carries JSON messages for conversation items, speech detection, and function calling between the browser and OpenAI servers.
• GevRealtimeController manages connection state with a 6-second grace period for temporary disconnections and proper cleanup of media tracks on teardown.
• Function calls from the Realtime API are delegated to src/voice/gevActions.js, enabling voice-controlled manipulation of the 3D visualization interface.
• Each session tracks costs via src/voice/voiceCost.js and provides visual feedback through Web Audio API-based microphone and assistant visualizers.

Frequently Asked Questions

How does the voice control system authenticate with the OpenAI Realtime API?

The system requests a short-lived JWT from the backend endpoint /api/realtime/token via the fetchRealtimeToken() function. This token authorizes the specific session and identifies which model variant to use for billing purposes. The JWT is then included in the SDP offer sent to OpenAI's signaling endpoint at https://api.openai.com/v1/realtime/calls.

What happens if the WebRTC connection drops during a voice session?

The controller monitors connectionState changes on the RTCPeerConnection. Temporary disconnections trigger a 6-second grace period (DISCONNECT_GRACE_MS). If the connection remains disconnected beyond this window, the controller calls fatalError() and terminates the session. Users can restart the connection via the push-to-talk shortcut or click-to-talk button.

How are voice commands converted into application actions?

Voice audio flows to the OpenAI Realtime API, which processes speech and can return function_call_arguments events. The handleRealtimeEvent() method in src/voice/gevRealtime.js extracts these calls and passes them to the tool runner in src/voice/gevActions.js. This runner maps function names (like highlight_runway or navigate_to) to specific viewer methods, executing the requested action within the 3D scene.

Can the system handle text input alongside voice commands?

Yes. The sendTextCommand() method on the controller constructs a conversation.item.create message with role set to user and transmits it over the established "oai-events" data channel. The Realtime API processes this text identically to transcribed speech, allowing mixed-mode interaction where users can type commands while maintaining an active voice session.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}