How Real-Time Speech Recognition (ASR) Streaming Works in Bella OpenAPI

Bella OpenAPI implements real-time ASR streaming as a full-duplex WebSocket bridge that forwards binary audio frames to third-party providers and returns incremental transcription results in a standardized RealTimeMessage format.

The lianjiatech/bella-openapi repository provides a production-ready architecture for real-time speech recognition that abstracts provider-specific protocols behind a unified WebSocket interface. Clients connect to a single endpoint, stream audio chunks, and receive partial and final transcripts without managing multiple provider connections.

WebSocket Endpoint and Controller Entry Point

The streaming service exposes two equivalent paths: /v1/audio/asr/stream and the alias /realtime. The AudioController handles the WebSocket upgrade and initializes the processing pipeline.

// src/main/java/com/ke/bella/openapi/endpoints/AudioController.java
@RequestMapping({ "/realtime", "/asr/stream" })
public void realtime(HttpServletRequest request, HttpServletResponse response) {
    // Creates a WebSocketSession and registers RealTimeHandler
}

Source: AudioController.java

The controller extracts the request URI, invokes the ChannelRouter to select a provider, and delegates the WebSocket lifecycle to RealTimeHandler.

Channel Routing and Provider Selection

Before establishing the downstream connection, the system selects the appropriate ASR provider using the ChannelRouter. This component evaluates the requested model, cost constraints, and load-balancing configuration to choose a concrete implementation (e.g., Ke, Tencent, or Huoshan).

ChannelDB channel = router.route(endpoint, model, apikey, processData.isMock());
String protocol = processData.getProtocol();
String url = processData.getForwardUrl();
String channelInfo = channel.getChannelInfo();

RealTimeAdaptor<AsrProperty> adaptor = adaptorManager.getProtocolAdaptor(
        endpoint, protocol, RealTimeAdaptor.class);
AsrProperty property = JacksonUtils.deserialize(channelInfo,
        adaptor.getPropertyClass());

Source: Same file, lines 118-132.

The AsrProperty object contains provider-specific credentials and configuration options required to authenticate with the third-party service.

The RealTimeAdaptor Interface

All ASR providers implement the RealTimeAdaptor interface, which defines the contract for establishing connections, sending audio, and managing the transcription lifecycle.

// src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeAdaptor.java
public interface RealTimeAdaptor<T extends AsrProperty> extends IProtocolAdaptor {
    default String endpoint() { return "/v1/audio/realtime"; }

    WebSocket startTranscription(String url, T property,
                                 RealTimeMessage request,
                                 WebSocketCallback callback);
    boolean sendAudioData(WebSocket ws, byte[] audio, WebSocketCallback cb);
    boolean stopTranscription(WebSocket ws, RealTimeMessage request,
                              WebSocketCallback cb);
    void closeConnection(WebSocket ws);
    WebSocketCallback createCallback(Sender sender, EndpointProcessData pd,
                                     EndpointLogger logger,
                                     String taskId, RealTimeMessage req,
                                     T property);
    Class<T> getPropertyClass();
}

Source: RealTimeAdaptor.java

This abstraction allows Bella OpenAPI to support multiple ASR backends without modifying the core WebSocket handling logic.

Provider-Specific Implementation: KeAdaptor

The default internal provider uses KeAdaptor, which implements the RealTimeAdaptor interface to communicate with the "Ke" ASR service.

// src/main/java/com/ke/bella/openapi/protocol/realtime/KeAdaptor.java
@Override
public WebSocket startTranscription(String url, RealtimeProperty property,
                                    RealTimeMessage request,
                                    WebSocketCallback callback) {
    Request.Builder builder = new Request.Builder()
            .url(url)
            .header("Authorization", apikey);
    WebSocket ws = HttpUtils.websocketRequest(builder.build(),
                     new BellaWebSocketListener(callback));
    ws.send(JacksonUtils.serialize(request));
    callback.started();
    return ws;
}

Source: KeAdaptor.java

The adaptor creates an OkHttp WebSocket connection to the external service, sends the initial StartTranscription payload, and returns the socket for subsequent audio streaming.

Audio frames are forwarded as binary messages:

@Override
public boolean sendAudioData(WebSocket ws, byte[] audio, WebSocketCallback cb) {
    return ws.send(ByteString.of(audio));
}

Source: Same file, lines 52-54.

Core WebSocket Handler and Message Flow

The RealTimeHandler manages the full-duplex communication between the client and the selected ASR provider. It processes both text control messages and binary audio data.

// src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeHandler.java
@Override
protected void handleTextMessage(WebSocketSession session, TextMessage message) {
    // parse control messages (START_TRANSCRIPTION / STOP_TRANSCRIPTION)
}

@Override
protected void handleBinaryMessage(WebSocketSession session, BinaryMessage message) {
    // forward binary audio to external ASR WebSocket
}

private void handleStartTranscription(...){
    callback = adaptor.createCallback(...);
    ws = adaptor.startTranscription(url, property, request, callback);
}

private void handleStopTranscription(...){
    adaptor.stopTranscription(ws, request, callback);
}

Source: RealTimeHandler.java

The handler maintains the taskId, the external WebSocket connection (ws), and delegates all I/O operations to the provider-specific adaptor.

Callback Processing and Metrics

The KeRealtimeCallback processes incoming messages from the ASR provider, translates them to the client format, and records performance metrics.

// src/main/java/com/ke/bella/openapi/protocol/realtime/KeRealtimeCallback.java
@Override
public void onMessage(WebSocket webSocket, String text) {
    RealTimeMessage msg = JacksonUtils.deserialize(text, RealTimeMessage.class);
    // forward most messages to client; intercept STARTED, TTS_TTFT, SESSION_CLOSE, TASK_FAILED
}

private void complete() {
    // record total latency, close client session, log metrics
}

Source: KeRealtimeCallback.java

The callback pushes incremental transcription results back to the client via the Sender interface, captures metrics such as total latency (TTLT) and time-to-first-token (TTS), and ensures clean session termination when the provider signals completion.

Client Integration Example

Clients connect to Bella OpenAPI using standard WebSocket APIs. The following JavaScript example demonstrates connecting to the real-time ASR streaming endpoint, sending audio, and handling incremental results.

// Connect to Bella OpenAPI real-time ASR
const socket = new WebSocket(
  "wss://api.example.com/v1/audio/asr/stream?model=gpt-4o-asr"
);

// 1️⃣ Start transcription
socket.addEventListener("open", () => {
  const startMsg = {
    header: { name: "StartTranscription", taskId: crypto.randomUUID() },
    payload: { /* optional TTS/LLM options */ }
  };
  socket.send(JSON.stringify(startMsg));
});

// 2️⃣ Stream binary audio (e.g., PCM 16-bit, 16 kHz)
function sendAudioChunk(chunkArrayBuffer) {
  socket.send(chunkArrayBuffer); // binary frame
}

// 3️⃣ Handle incremental results
socket.addEventListener("message", event => {
  const msg = JSON.parse(event.data);
  if (msg.header.name === "SentenceEnd") {
    console.log("Transcribed:", msg.payload.text);
  } else if (msg.header.name === "TranscriptionFinished") {
    console.log("Final result:", msg.payload);
    socket.close();
  }
});

The client sends a StartTranscription control message, followed by binary PCM audio chunks. Bella OpenAPI forwards these to the selected provider, and each incremental result (such as SentenceEnd or TranscriptionFinished) is pushed back to the client in the standardized RealTimeMessage format.

Summary

• Bella OpenAPI exposes real-time ASR streaming through WebSocket endpoints /v1/audio/asr/stream and /realtime, handled by AudioController.
• Channel routing via ChannelRouter selects the appropriate ASR provider (Ke, Tencent, Huoshan) based on model and cost configuration.
• Protocol abstraction is achieved through the RealTimeAdaptor interface, allowing provider-specific implementations like KeAdaptor to manage external WebSocket connections.
• Message handling occurs in RealTimeHandler, which processes client text commands and binary audio, forwarding them to the provider via the adaptor.
• Result translation is handled by callbacks such as KeRealtimeCallback, which convert provider responses into RealTimeMessage DTOs and record performance metrics.
• Client integration requires only standard WebSocket support, sending JSON control messages and binary audio frames to receive incremental transcriptions.

Frequently Asked Questions

What audio format does Bella OpenAPI expect for real-time ASR streaming?

Bella OpenAPI accepts binary audio frames in standard formats such as PCM 16-bit at 16 kHz. The RealTimeHandler forwards these binary messages directly to the provider's WebSocket via the sendAudioData method in the RealTimeAdaptor implementation, without transcoding. Specific format requirements depend on the selected provider channel configured in AsrProperty.

How does Bella OpenAPI handle provider failover during an active streaming session?

The ChannelRouter selects the provider channel at the start of the session when handleStartTranscription is invoked in RealTimeHandler. If the downstream provider connection fails, the WebSocketCallback (such as KeRealtimeCallback) detects the failure via onFailure or onClosing events, records the error metrics, and terminates the session. Clients must initiate a new WebSocket connection to trigger a fresh channel selection for retry.

What is the difference between SentenceEnd and TranscriptionFinished messages?

SentenceEnd indicates an intermediate result where the provider has detected a complete sentence boundary but the session remains active for additional audio. TranscriptionFinished signals the final result after the client sends StopTranscription or the provider closes the session. The KeRealtimeCallback intercepts these message types in its onMessage method and forwards them to the client via the Sender interface, allowing applications to render partial transcripts while waiting for completion.

Can I use Bella OpenAPI with custom ASR providers not included in the default distribution?

Yes, the architecture supports custom providers through the RealTimeAdaptor interface. You must implement startTranscription, sendAudioData, stopTranscription, and createCallback methods to handle the provider's specific WebSocket protocol and authentication. Register your adaptor implementation with the adaptorManager using the endpoint and protocol identifiers. The RealTimeHandler will then delegate all audio streaming and message translation to your custom implementation without requiring changes to the core WebSocket handling logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}