How to Integrate OpenAI Realtime API for Voice Control in a CesiumJS App

Integrate OpenAI Realtime API with CesiumJS by implementing a secure token proxy server, establishing a WebRTC data channel for bidirectional audio streaming, and mapping voice intents to Cesium camera and entity actions through a modular action runner.

The God's Eye View repository demonstrates a production-ready pattern for adding voice control to CesiumJS applications using the OpenAI Realtime API. This implementation keeps API keys server-side while enabling natural language commands to manipulate the 3D globe, fly to locations, and control scene layers in real time.

Architecture Overview

The integration splits responsibilities across three distinct layers to ensure security and cost control. In src/voice/gevRealtime.js, the client-side layer handles microphone capture, WebRTC negotiation, and UI state management. The server layer in vite.config.js proxies short-lived tokens to OpenAI without exposing your API key to the browser. Finally, src/voice/voiceCost.js enforces spending caps and model selection to prevent runaway costs during voice sessions.

This architecture ensures that sensitive credentials never reach the client while allowing the voice assistant full access to manipulate the Cesium viewer instance through a secure action dispatch system.

Server-Side Token Management

The token proxy endpoint prevents API key exposure while enforcing tier-based model selection. Located in vite.config.js, the /api/realtime/token route validates user-selected tiers and forwards requests to OpenAI's Realtime endpoint.

// vite.config.js - Server-side token endpoint
export default defineConfig({
  server: {
    proxy: {
      '/api/realtime/token': {
        target: 'https://api.openai.com',
        changeOrigin: true,
        rewrite: (path) => {
          // Tier validation and model resolution happens here
          // before forwarding to /v1/realtime/calls
        }
      }
    }
  }
})

The server uses resolveVoiceModel from src/voice/voiceCost.js to map user tier selections (standard or mini) to specific model IDs before requesting the ephemeral token. This ensures users cannot arbitrarily select expensive models while giving you granular control over which Realtime models your application exposes.

Client-Side Voice Integration

The src/voice/gevRealtime.js module manages the WebRTC connection and audio streaming lifecycle. It creates an RTCPeerConnection, negotiates the SDP offer/answer exchange with OpenAI's Realtime API, and attaches the user's microphone stream to the connection.

Key responsibilities include:

• Disconnect throttling to prevent rapid reconnection loops
• Viewport screenshot limits to control context window costs
• Audio playback of assistant responses through the browser's audio context

The module exposes initGevVoiceCommands, which you call from src/main.js after initializing your Cesium viewer, passing the viewer instance and other managers to enable scene-aware voice control.

Cost Guardrails and Model Selection

The src/voice/voiceCost.js file implements a critical safety layer for production deployments. It defines default spending limits (warnUsd at 2 USD and capUsd at 5 USD) and persists user tier preferences in localStorage under the keys godsEyeView.voiceCost.tier and godsEyeView.voiceCost.limits.

During active sessions, the module tracks token usage against OpenAI's pricing tables (updated as of 2026-08-18). When the soft warning threshold triggers, the UI can notify the user; when the hard cap hits, the session terminates cleanly to prevent billing surprises.

Implementing Voice Commands for Cesium

Voice intents translate to Cesium actions through src/voice/gevActions.js, which registers functions accessible to the GevActionRunner. These actions receive the Cesium viewer instance as their first parameter, enabling direct manipulation of the camera, entities, and data layers.

// src/voice/gevActions.js - Example voice action
export async function flyToAndSelect({ viewer }, { airportCode }) {
  const airport = await lookupAirport(airportCode);
  viewer.camera.flyTo({ destination: airport.position });
  
  const nearest = findNearestAircraft(viewer.entities, airport.position);
  viewer.selectedEntity = nearest;
}

This pattern allows complex multi-step commands like "take me to LAX and select the nearest aircraft" to execute as atomic operations that combine camera movement with entity selection.

Step-by-Step Integration Walkthrough

Initializing the Voice System

Call initGevVoiceCommands from your main application entry point after the Cesium viewer initializes. Pass references to the viewer and any scene managers that voice actions might need to access.

// src/main.js
import { initGevVoiceCommands } from './voice/gevRealtime.js';
import { viewer } from './cesiumSetup.js';

initGevVoiceCommands({
  viewer,
  styleManager: window.styleManager,
  dataManager: window.dataManager,
  sceneDirector: window.sceneDirector,
  annotations: window.annotations,
});

This initialization creates the GEV MIC button in your Cesium UI and prepares the action runner to accept voice commands.

Requesting Tokens and Establishing WebRTC

When the user activates the microphone, request an ephemeral token from your secure endpoint, then establish the WebRTC connection to OpenAI's Realtime API.

// Client-side token request
async function getRealtimeToken(tier = 'standard') {
  const resp = await fetch(`/api/realtime/token?tier=${tier}`);
  if (!resp.ok) throw new Error('Failed to get OpenAI token');
  const { token } = await resp.json();
  return token;
}

// WebRTC session initiation
async function startRealtimeSession(tier) {
  const token = await getRealtimeToken(tier);
  const pc = new RTCPeerConnection();
  
  // Setup data channel for realtime events
  const channel = pc.createDataChannel('realtime');
  
  // Add microphone track
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  stream.getAudioTracks().forEach(track => pc.addTrack(track, stream));
  
  // Create and send offer
  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);
  // Send offer to OpenAI with the ephemeral token
}

The gevRealtime.js implementation handles the full SDP handshake, ICE candidate management, and graceful disconnects when the cost cap triggers or the user closes the session.

Handling Voice Actions

The action runner processes natural language responses from OpenAI and maps them to registered functions in gevActions.js. Each action receives a context object containing the Cesium viewer and any additional managers you passed during initialization, allowing voice commands to:

• Fly the camera to specific coordinates or named locations
• Toggle layer visibility or styling
• Select and highlight entities in the scene
• Control auxiliary features like the built-in radio player

Summary

• Secure token proxy: Use a server-side endpoint in vite.config.js to request ephemeral tokens from OpenAI, keeping your API key hidden from client-side code.
• WebRTC audio streaming: Implement the connection logic in src/voice/gevRealtime.js to stream microphone input and receive assistant audio responses in real time.
• Cost protection: Leverage src/voice/voiceCost.js to enforce spending limits and tier-based model selection, preventing unexpected API charges.
• Cesium integration: Pass the Cesium viewer instance to voice actions via GevActionRunner, enabling natural language control over camera movements, entity selection, and scene state.

Frequently Asked Questions

How do I prevent my OpenAI API key from being exposed in the browser?

Deploy a server-side token proxy. The God's Eye View implementation uses a Vite development server proxy endpoint at /api/realtime/token that forwards the request to OpenAI using your secret key stored in environment variables. The browser only receives a short-lived ephemeral token (valid for one session) that cannot be reused to access your account or incur charges beyond the current connection.

What CesiumJS features can be controlled through voice commands?

Any feature accessible through the Cesium viewer instance. The action system in src/voice/gevActions.js demonstrates camera flights to locations (viewer.camera.flyTo), entity selection (viewer.selectedEntity), and layer toggling. You can extend this pattern to control entity properties, trigger data loading, or manipulate the timeline clock—all by passing the viewer reference to your voice action functions.

How does the cost-guard system protect against unexpected billing?

Through soft warnings and hard caps persisted in localStorage. The voiceCost.js module tracks cumulative session costs against configurable limits (defaulting to 2 USD warning and 5 USD cap). When the warning threshold hits, the UI can alert the user; when the cap hits, the WebRTC connection terminates automatically. The tier system also restricts users to cheaper models (like "mini" variants) unless explicitly permitted.

Can I use this pattern with other 3D frameworks besides CesiumJS?

Yes, with minimal modifications. While the specific actions in gevActions.js target Cesium's API, the WebRTC connection logic in gevRealtime.js and the token proxy in vite.config.js are framework-agnostic. Replace the Cesium-specific action implementations with equivalents for Three.js, Babylon.js, or other WebGL frameworks while maintaining the same voice-to-action mapping architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}