OmniRoute Audio Processing Capabilities: Speech-to-Text and Text-to-Speech APIs

OmniRoute provides OpenAI-compatible audio processing with Speech-to-Text (transcription) and Text-to-Speech (synthesis) endpoints, supporting 100 MiB uploads, remote provider routing controls, and automatic failover across audio-capable providers.

OmniRoute treats audio as a first-class modality within its unified routing layer, exposing standard REST endpoints that mirror the OpenAI Audio API specification. The diegosouzapw/OmniRoute repository implements a pluggable provider system that automatically selects appropriate backend nodes based on declared model capabilities, enabling seamless audio transcription and voice synthesis without client-side provider management.

Core Audio Processing Services

OmniRoute exposes two primary audio endpoints defined in the API-category map at src/shared/constants/endpointCategories.ts. The generic OpenAI-compatible router automatically exposes these routes when the requested modality is "audio".

Speech-to-Text Transcription

The transcription endpoint accepts multipart audio uploads and returns recognized text. Clients send POST requests to /v1/audio/transcriptions with audio files (MP3, WAV, or other supported formats) and a model identifier. The request body must include a model parameter (e.g., whisper-1) that corresponds to entries in the provider catalog. According to the source at src/shared/middleware/bodySizeGuard.ts (lines 25-70), this endpoint specifically permits uploads up to 100 MiB to accommodate large audio files.

Text-to-Speech Synthesis

The speech synthesis endpoint streams binary audio data in response to text input. Clients POST to /v1/audio/speech with a JSON payload specifying the model (e.g., tts-1), input text (or SSML), and optional voice parameters. The router returns a binary audio stream (typically MP3) that can be saved directly to disk or passed to audio players.

Routing Architecture and Model Discovery

The platform wires audio capabilities through a capability-driven routing layer that validates requests against provider metadata before forwarding.

Provider Catalog Registration

Every audio-capable provider is registered in src/lib/providerModels/catalog.ts (line 142) under the audio category. Each catalog entry specifies the provider ID, supported endpoints (audio-transcriptions, audio-speech), and provider-specific quirks. When OmniRoute initializes, it builds an internal index of these entries to determine which backend nodes can fulfill incoming audio requests.

Capability Validation

Before routing, OmniRoute validates that the requested model actually supports the audio operation. The src/lib/modelCapabilities.ts file (line 74) enumerates capabilities for each model, exposing "audio" as a supported modality. This validation layer prevents requests from reaching providers that lack transcription or speech synthesis implementations.

Endpoint-to-Modality Mapping

The mapping logic resides in src/shared/constants/modelSupportedEndpoints.ts (lines 7-13), which links logical endpoint names (audio-speech, audio-transcriptions) to the generic audio modality. When a request targets /v1/audio/*, the router consults this mapping to determine which sub-endpoints to activate and which provider nodes are eligible for selection.

Security and Performance Controls

OmniRoute implements specific guards for audio processing due to the sensitive nature of voice data and the large payload sizes involved.

Upload Size Management

Because audio files can be substantially larger than text payloads, the body-size guard in src/shared/middleware/bodySizeGuard.ts raises the upload limit to 100 MiB specifically for the transcription endpoint. This prevents standard request size limits from blocking legitimate audio uploads while still protecting against denial-of-service attacks via excessive payload sizes.

Remote Provider Access Control

By default, OmniRoute restricts audio processing to localhost nodes to prevent accidental egress of raw audio data to external providers. The feature flag audioRemoteProviderNodes in src/shared/constants/featureFlagDefinitions.ts (lines 150-151) controls this behavior and is disabled by default. Administrators must explicitly enable this flag to route audio requests to external provider nodes, ensuring compliance with data residency and privacy requirements.

Bridge Configuration

The modality bridge layer manages streaming audio sessions through configurable defaults defined in src/shared/constants/modalityBridgeDefaults.ts (lines 25-28). These settings control session timeout values and maximum clip counts per request, ensuring long-running transcription or synthesis operations complete without hanging connections.

Client Implementation Examples

The following examples demonstrate calling the audio endpoints against a local OmniRoute server running on port 20128.

Transcription Example

This TypeScript snippet uploads an audio file for speech-to-text conversion:

import fs from "node:fs";
import fetch from "node-fetch";

async function transcribe() {
  const file = fs.readFileSync("speech.mp3");
  const form = new FormData();
  form.append("file", new Blob([file]), "speech.mp3");
  form.append("model", "whisper-1");

  const resp = await fetch("http://localhost:20128/v1/audio/transcriptions", {
    method: "POST",
    body: form,
    headers: { 
      Authorization: `Bearer ${process.env.OMNIROUTE_API_KEY}` 
    },
  });

  const data = await resp.json();
  console.log("Transcribed text:", data.text);
}

transcribe();

The request uses multipart/form-data encoding and relies on the generic API middleware for authentication and injection protection as implemented in src/middleware/promptInjectionGuard.ts.

Speech Synthesis Example

This example converts text to speech and saves the binary audio stream:

import fetch from "node-fetch";
import fs from "node:fs";

async function speak() {
  const payload = {
    model: "tts-1",
    input: "Hello, OmniRoute!",
    voice: "alloy",
  };

  const resp = await fetch("http://localhost:20128/v1/audio/speech", {
    method: "POST",
    body: JSON.stringify(payload),
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.OMNIROUTE_API_KEY}`,
    },
  });

  const buffer = await resp.arrayBuffer();
  fs.writeFileSync("output.mp3", Buffer.from(buffer));
  console.log("Audio saved to output.mp3");
}

speak();

Both examples utilize the error sanitization helpers in src/open-sse/utils/error.ts to prevent internal stack traces from leaking to clients.

Resilience and Provider Management

Audio requests benefit from the same resilience mechanisms that power chat and embeddings routing. The provider layer implements circuit breakers that temporarily remove failing audio providers from rotation, connection cooldown periods to prevent thundering herds, and combo-routing capabilities that can distribute requests across multiple audio-capable providers. If a provider fails during transcription or synthesis, OmniRoute automatically falls back to alternative providers listed in src/shared/constants/providers/audio.ts without requiring client-side retry logic.

Summary

• OmniRoute exposes OpenAI-compatible /v1/audio/transcriptions and /v1/audio/speech endpoints for speech-to-text and text-to-speech processing.
• The platform validates audio capabilities against src/lib/modelCapabilities.ts before routing to providers registered in src/lib/providerModels/catalog.ts.
• Audio uploads support files up to 100 MiB through the specialized body-size guard in src/shared/middleware/bodySizeGuard.ts.
• By default, audio processing is restricted to localhost nodes via the audioRemoteProviderNodes feature flag in src/shared/constants/featureFlagDefinitions.ts.
• Audio requests receive full resilience support including circuit breakers, automatic failover, and combo-routing across multiple providers.

Frequently Asked Questions

What audio file formats does OmniRoute support for transcription?

OmniRoute delegates format support to the underlying provider models registered in the provider catalog. While the API accepts standard formats like MP3 and WAV, the specific codec support depends on the backend provider (e.g., Whisper-based models or FishAudio). The routing layer in src/shared/constants/endpointCategories.ts treats the audio file as an opaque binary blob and passes it through to the selected provider without transcoding.

How does OmniRoute handle large audio file uploads?

The platform raises the request body limit to 100 MiB specifically for the transcription endpoint via the body-size guard defined in src/shared/middleware/bodySizeGuard.ts (lines 25-70). This limit accommodates high-fidelity, long-duration audio files while still protecting the server from memory exhaustion attacks. Administrators can adjust these limits by modifying the middleware configuration.

Can I route audio requests to external cloud providers?

Yes, but this requires explicitly enabling the audioRemoteProviderNodes feature flag in src/shared/constants/featureFlagDefinitions.ts (lines 150-151). By default, this flag is disabled to prevent accidental transmission of raw audio data to external nodes. When enabled, OmniRoute can route requests to external provider nodes listed under the audio category in the provider catalog, subject to standard authentication and TLS encryption.

What happens if an audio provider fails during processing?

OmniRoute applies provider-circuit-breaker logic and connection-cooldown mechanisms to audio endpoints just as it does for chat completions. If a provider fails mid-request, the router can automatically retry the request against alternative audio-capable providers defined in src/lib/providerModels/catalog.ts without the client needing to implement retry logic. The modality bridge defaults in src/shared/constants/modalityBridgeDefaults.ts control timeouts to ensure failed requests fail fast rather than hanging indefinitely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}