# How the Gods Eye View Voice Control System Uses WebRTC and the OpenAI Realtime API

> Discover how the Gods Eye View voice control system uses WebRTC and OpenAI Realtime API for low-latency commands. Stream audio and execute tools effortlessly.

- Repository: [Bilawal Sidhu/gods-eye-view](https://github.com/bilawalsidhu/gods-eye-view)
- Tags: how-to-guide
- Published: 2026-08-29

---

**The Gods Eye View voice control system establishes a WebRTC peer connection with a bidirectional data channel to stream encrypted audio and exchange JSON events with the OpenAI Realtime API, enabling low-latency voice commands and tool execution.**

The voice control feature in the [bilawalsidhu/gods-eye-view](https://github.com/bilawalsidhu/gods-eye-view) repository implements a production-grade WebRTC pipeline that connects browser-based audio streams directly to OpenAI's Realtime API. Located within `src/voice/`, this architecture bypasses traditional HTTP polling in favor of persistent peer-to-peer media channels, delivering sub-second latency for conversational AI interactions within the 3D visualization interface.

## WebRTC Architecture Overview

The voice control system operates through four distinct stages orchestrated by `GevRealtimeController` in [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js):

1. **Token Acquisition** – The client requests a JWT from `/api/realtime/token` that authorizes the session and specifies the target model.
2. **Peer Connection Setup** – Creation of an `RTCPeerConnection` with local microphone tracks and a data channel named `"oai-events"`.
3. **SDP Exchange** – The client generates an SDP offer, transmits it to `https://api.openai.com/v1/realtime/calls`, and applies the returned SDP answer.
4. **Real-time Messaging** – Bidirectional JSON event flow over the data channel for conversation items, function calls, and system events.

This flow leverages the browser's native WebRTC stack to establish encrypted, low-latency communication without intermediary servers handling the media stream.

## Token Acquisition and Session Initialization

Before establishing any peer connection, the controller must obtain authorization. The `fetchRealtimeToken()` function (lines 14-35 in [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js)) retrieves a short-lived JWT from the application's backend:

```javascript
// Internal flow within GevRealtimeController.start()
const token = await fetchRealtimeToken();
// Returns: { token: string, model: string }

```

The token response includes both the authentication credential and the specific OpenAI model identifier (e.g., `gpt-4o-realtime-preview`). This value is immediately passed to the voice cost tracker in [`src/voice/voiceCost.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/voiceCost.js) to ensure accurate billing calculations against the correct rate table.

## Establishing the WebRTC Peer Connection

The `start()` method in [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js) (lines 23-49) manages the complex WebRTC initialization sequence:

```javascript
// Create peer connection
this.pc = new RTCPeerConnection({
  iceServers: [{ urls: 'stun:stun.l.google.com:19302' }]
});

// Attach local microphone stream
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getTracks().forEach(track => this.pc.addTrack(track, stream));

// Initialize data channel for OpenAI events
this.dc = this.pc.createDataChannel('oai-events', { ordered: true });

```

Once the data channel and tracks are attached, the controller generates an SDP offer and completes the signaling exchange with OpenAI's servers (lines 71-97). The remote SDP answer is applied via `this.pc.setRemoteDescription()`, finalizing the encrypted media path.

## Data Channel Protocol and Event Handling

All communication with the Realtime API occurs over the `"oai-events"` data channel using JSON payloads conforming to the OpenAI Realtime specification. The `sendRealtimeEvent()` method wraps outgoing messages, while `handleRealtimeEvent()` (lines 102-165 in [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js)) parses and dispatches incoming events:

```javascript
handleRealtimeEvent(event) {
  const payload = JSON.parse(event.data);
  
  switch (payload.type) {
    case 'input_audio_buffer.speech_started':
      // Mark user turn beginning
      break;
    case 'response.function_call_arguments.done':
      // Extract and execute tool calls
      this.runTool(payload);
      break;
    case 'response.done':
      // Finalize turn and trigger UI updates
      break;
  }
}

```

The data channel enables server-initiated function calls. When the model emits `response.function_call_arguments.done`, the controller extracts the function name and arguments through `extractFunctionCalls()`, then delegates execution to the tool runner in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js).

## Session Lifecycle and Connection State Management

The controller implements a robust state machine monitoring the `RTCPeerConnection.connectionState`. Critical logic resides in the connection state listener:

- **Failed State**: Immediately triggers `fatalError()` and terminates the session.
- **Disconnected State**: Initiates a grace period (`DISCONNECT_GRACE_MS`, 6 seconds) to tolerate temporary network fluctuations before calling `fatalError()`.

The `stop()` method provides graceful teardown by incrementing an internal `startEpoch` to cancel in-flight operations, closing the data channel, stopping peer connection transceivers, and releasing microphone tracks:

```javascript
// Graceful shutdown example
controller.stop({ preserveRadioPlayback: true });

```

## Push-to-Talk vs. Click-to-Talk Interaction Modes

The system supports two distinct user interaction patterns managed through `bindPushToTalkShortcut()` and `shouldHandlePushToTalkKeyDown`:

**Click-to-Talk**: The UI button handler toggles the session state via `controller.buttonHandler`, starting or stopping the connection based on current state.

**Push-to-Talk**: Keyboard event listeners intercept Space keydown/keyup events. Holding Space enables the microphone stream (`setMicrophoneEnabled(true)`) and maintains the session only while depressed, suitable for quick commands without full session overhead.

## Function Calling and Tool Execution

When the Realtime API returns function calls, `GevRealtimeController` invokes the action runner with abort signal support and session validation:

```javascript
// Inside handleRealtimeEvent for function calls
const result = await this.runner(call.name, parsedArguments, {
  signal: toolController.signal,
  isCurrent: () => this.startEpoch === currentEpoch
});

```

The runner imported from [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js) maps function names to application-specific operations:

```javascript
// Example tool implementation in gevActions.js
export async function runTool(name, args, ctx) {
  switch (name) {
    case 'highlight_runway':
      await viewer.highlightRunway(args.runwayId);
      return { ok: true, action: name };
  }
}

```

## Cost Tracking and Audio Visualization

Every session instantiates a cost tracker from [`src/voice/voiceCost.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/voiceCost.js) using the model identifier returned during token acquisition. This ensures usage accounting against the correct pricing tier for the specific Realtime model deployed.

Simultaneously, the system runs dual audio visualizers built on the Web Audio API (`AudioContext`, `AnalyserNode`):

- **Microphone Visualizer**: Analyzes the local `MediaStream` to provide visual feedback during user speech.
- **Assistant Visualizer**: Processes the remote audio stream from the AI to indicate when the model is speaking.

Both visualizers update DOM elements via `startVoiceVisualizer()` and `startAssistantVoiceVisualizer()` to create synchronized visual feedback.

## Practical Implementation Examples

Initialize the voice controller in your application entry point:

```javascript
import { initGevVoiceCommands } from './voice/gevRealtime.js';

const controller = initGevVoiceCommands({
  viewer,
  styleManager,
  dataManager,
  sceneDirector,
  annotations
});

```

Send typed commands programmatically while the session is active:

```javascript
controller.sendTextCommand('Show me the nearest runway');

```

Handle incoming function calls by registering tools in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js):

```javascript
export async function runTool(name, args, ctx) {
  if (name === 'navigate_to') {
    await ctx.viewer.flyTo(args.coordinates);
    return { status: 'success' };
  }
}

```

## Summary

- The voice control system in `bilawalsidhu/gods-eye-view` uses WebRTC peer connections to establish encrypted, low-latency audio streams directly with the OpenAI Realtime API.
- Authentication requires a JWT from `/api/realtime/token` before initiating the SDP exchange with `https://api.openai.com/v1/realtime/calls`.
- The `"oai-events"` data channel carries JSON messages for conversation items, speech detection, and function calling between the browser and OpenAI servers.
- `GevRealtimeController` manages connection state with a 6-second grace period for temporary disconnections and proper cleanup of media tracks on teardown.
- Function calls from the Realtime API are delegated to [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js), enabling voice-controlled manipulation of the 3D visualization interface.
- Each session tracks costs via [`src/voice/voiceCost.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/voiceCost.js) and provides visual feedback through Web Audio API-based microphone and assistant visualizers.

## Frequently Asked Questions

### How does the voice control system authenticate with the OpenAI Realtime API?

The system requests a short-lived JWT from the backend endpoint `/api/realtime/token` via the `fetchRealtimeToken()` function. This token authorizes the specific session and identifies which model variant to use for billing purposes. The JWT is then included in the SDP offer sent to OpenAI's signaling endpoint at `https://api.openai.com/v1/realtime/calls`.

### What happens if the WebRTC connection drops during a voice session?

The controller monitors `connectionState` changes on the `RTCPeerConnection`. Temporary disconnections trigger a 6-second grace period (`DISCONNECT_GRACE_MS`). If the connection remains disconnected beyond this window, the controller calls `fatalError()` and terminates the session. Users can restart the connection via the push-to-talk shortcut or click-to-talk button.

### How are voice commands converted into application actions?

Voice audio flows to the OpenAI Realtime API, which processes speech and can return `function_call_arguments` events. The `handleRealtimeEvent()` method in [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js) extracts these calls and passes them to the tool runner in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js). This runner maps function names (like `highlight_runway` or `navigate_to`) to specific viewer methods, executing the requested action within the 3D scene.

### Can the system handle text input alongside voice commands?

Yes. The `sendTextCommand()` method on the controller constructs a `conversation.item.create` message with role set to `user` and transmits it over the established `"oai-events"` data channel. The Realtime API processes this text identically to transcribed speech, allowing mixed-mode interaction where users can type commands while maintaining an active voice session.