# How to Use the OpenAI API for Voice Control in God’s Eye View

> Learn how to integrate the OpenAI API for voice control in God's Eye View. Discover how to stream audio, execute tool calls, and generate spoken summaries using a secure Node.js proxy.

- Repository: [Bilawal Sidhu/gods-eye-view](https://github.com/bilawalsidhu/gods-eye-view)
- Tags: how-to-guide
- Published: 2026-09-13

---

**God’s Eye View implements voice control by integrating the OpenAI Realtime API into a client-server architecture that streams microphone audio, executes tool calls for in-app actions, and generates spoken HUD summaries via a secure Node.js proxy.**

God’s Eye View (GEV) is an open-source geospatial visualization platform that enables hands-free operation through the OpenAI API for voice control. This guide explains the implementation details, configuration steps, and code patterns required to enable realtime voice interaction in your own GEV deployment.

## Architecture Overview

The voice system uses a secure client-server split to protect API credentials while maintaining low-latency audio streaming. The browser manages the user interface and audio capture, while the server handles authentication and proxies sensitive requests to OpenAI.

### Client-Side Voice Module

The [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js) file initializes a WebSocket connection to OpenAI’s realtime endpoint and manages bidirectional audio streaming. This module handles microphone input, plays back model responses, and dispatches incoming events to the appropriate handlers.

### Server-Side Proxy

The server-side implementation in [`server/providers/openai/realtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/realtime.js) generates short-lived client secrets and forwards session configuration to OpenAI. This proxy ensures your `OPENAI_API_KEY` never exposes to the browser. The server also hosts the HUD summary endpoint in [`server/providers/openai/hud-summary.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/hud-summary.js), which provides concise five-word intelligence briefings.

## Configuring the OpenAI API Integration

Before enabling voice control, you must configure the API credentials and environment variables that authenticate your server with OpenAI’s realtime services.

### Environment Variables

Copy `.env.example` to `.env` in your project root and add your OpenAI API key. The server also supports storing credentials in the macOS keychain under `openai-api/api-key`.

```bash

# .env

OPENAI_API_KEY=sk-YOUR_OPENAI_KEY

# Optional: limit calls per minute

OPENAI_REALTIMET_TOKEN_RATE_LIMIT=30

```

### Server Initialization

Start the development server to load the environment configuration and initialize the OpenAI middlewares. The server registers endpoints for `/api/realtime/token` and `/api/openai/hud-summary` during startup.

```bash
npm install
npm run dev

```

## Implementing Realtime Voice Connections

The voice interaction flow follows a three-step process: token generation, WebSocket handshake, and continuous audio streaming.

### Requesting a Realtime Token

The browser requests a session token from the `/api/realtime/token` endpoint. The server creates this token by POSTing the full session configuration—including instructions from [`server/providers/openai/instructions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/instructions.js) and tool schemas from [`server/providers/openai/tools.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/tools.js)—to `https://api.openai.com/v1/realtime/client_secrets`.

```javascript
import { getRealtimeToken } from './api';

async function initVoice() {
  const { token, model, instructions, tools } = await getRealtimeToken();
  // Token is a short-lived client secret for the WebSocket
}

```

### Establishing the WebSocket

With the token returned, the client constructs a WebSocket URL pointing to `wss://api.openai.com/v1/realtime` and appends the model name and authentication parameters.

```javascript
const ws = new WebSocket(
  `wss://api.openai.com/v1/realtime?model=${encodeURIComponent(model)}`,
  [token]
);

const gev = new GEVRealtime(ws, { instructions, tools });
gev.start(); // Begins microphone streaming

```

### Streaming Audio and Events

The `GEVRealtime` class streams PCM audio to the WebSocket and listens for JSON events such as `response`, `tool_call`, and `error`. When the model returns a tool call, the dispatcher routes it to [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js) for execution.

## Handling Voice Commands with Tool Calls

Voice commands materialize as structured tool calls from the OpenAI model. The system maps these calls to specific functions within the God’s Eye View interface.

### Mapping Tools to Actions

The [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js) file translates tool calls into in-app actions like toggling map layers or adjusting camera altitude. The available tools are defined in [`server/providers/openai/tools.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/tools.js) inside the `GEV_REALTIME_TOOLS` object, which exports JSON schemas describing each function’s parameters.

When the model emits a `tool_call` event, `gevActions` looks up the corresponding implementation, executes the action locally, and returns a result object to the model to continue the conversation.

### Creating Custom Voice Tools

Extend the voice capabilities by adding entries to [`server/providers/openai/tools.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/tools.js) and implementing handlers in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js).

```javascript
// server/providers/openai/tools.js
export const GEV_REALTIME_TOOLS = {
  ...existingTools,
  focusLayer: {
    description: "Focus the camera on a specific map layer",
    parameters: {
      type: "object",
      properties: {
        layerId: { type: "string" }
      },
      required: ["layerId"]
    }
  }
};

// src/voice/gevActions.js
export async function handleToolCall(call) {
  switch (call.name) {
    case 'focusLayer':
      await focusMapLayer(call.parameters.layerId);
      return { result: `Focused on layer ${call.parameters.layerId}` };
    default:
      throw new Error(`Unknown tool: ${call.name}`);
  }
}

```

## Generating HUD Summaries

The system provides concise situational awareness through the `/api/openai/hud-summary` endpoint implemented in [`server/providers/openai/hud-summary.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/hud-summary.js). When a user requests a summary via voice, the client queries this endpoint, which forwards a prompt to the `gpt‑5‑nano` model (default configuration) and returns a terse five-word response.

The action dispatcher in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js) handles the `hudSummary` intent by fetching from this endpoint and returning the text to be spoken back to the pilot.

```javascript
case 'hudSummary':
  const resp = await fetch('/api/openai/hud-summary');
  const { summary } = await resp.json();
  return { result: summary };

```

## Summary

- **Secure Authentication**: The server generates short-lived tokens via [`server/providers/openai/realtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/realtime.js) to protect your `OPENAI_API_KEY` from client exposure.
- **Realtime Streaming**: [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js) manages the WebSocket connection to `wss://api.openai.com/v1/realtime` and handles bidirectional audio streaming.
- **Tool Integration**: Voice commands map to JavaScript functions through [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js) using schemas defined in [`server/providers/openai/tools.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/tools.js).
- **HUD Summaries**: The `/api/openai/hud-summary` endpoint provides spoken intelligence briefings powered by OpenAI language models.

## Frequently Asked Questions

### What OpenAI model does God’s Eye View use for voice control?

God’s Eye View connects to OpenAI’s Realtime API for voice interactions, using the realtime models for bidirectional audio. For HUD summaries, the default configuration in [`server/providers/openai/hud-summary.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/hud-summary.js) uses `gpt‑5‑nano`, though this can be modified in the server configuration.

### How does God’s Eye View secure the OpenAI API key?

The system never exposes the `OPENAI_API_KEY` to the browser. Instead, the server in [`server/providers/openai/realtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/realtime.js) creates ephemeral client secrets via the OpenAI API and passes only these short-lived tokens to the WebSocket client, maintaining secure credential isolation.

### Can I add custom voice commands to God’s Eye View?

Yes, extend `GEV_REALTIME_TOOLS` in [`server/providers/openai/tools.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/server/providers/openai/tools.js) with your JSON schema, then add the corresponding handler logic in [`src/voice/gevActions.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevActions.js). The model will automatically recognize and invoke your custom tools during voice conversations.

### What audio format does the realtime API use?

The [`src/voice/gevRealtime.js`](https://github.com/bilawalsidhu/gods-eye-view/blob/main/src/voice/gevRealtime.js) module streams PCM audio to the WebSocket connection. The implementation handles the encoding and decoding required by the OpenAI Realtime API, allowing you to work with standard web audio APIs on the client side.