# How Real-Time Speech Recognition (ASR) Streaming Works in Bella OpenAPI > Discover how Bella OpenAPI handles real-time ASR streaming. Learn about its WebSocket bridge, binary audio frame forwarding, and standardized transcription messages for efficient audio processing. - Repository: [Ke Technologies/bella-openapi](https://github.com/lianjiatech/bella-openapi) - Tags: deep-dive - Published: 2026-03-06 --- **Bella OpenAPI implements real-time ASR streaming as a full-duplex WebSocket bridge that forwards binary audio frames to third-party providers and returns incremental transcription results in a standardized `RealTimeMessage` format.** The `lianjiatech/bella-openapi` repository provides a production-ready architecture for real-time speech recognition that abstracts provider-specific protocols behind a unified WebSocket interface. Clients connect to a single endpoint, stream audio chunks, and receive partial and final transcripts without managing multiple provider connections. ## WebSocket Endpoint and Controller Entry Point The streaming service exposes two equivalent paths: `/v1/audio/asr/stream` and the alias `/realtime`. The `AudioController` handles the WebSocket upgrade and initializes the processing pipeline. ```java // src/main/java/com/ke/bella/openapi/endpoints/AudioController.java @RequestMapping({ "/realtime", "/asr/stream" }) public void realtime(HttpServletRequest request, HttpServletResponse response) { // Creates a WebSocketSession and registers RealTimeHandler } ``` *Source:* [AudioController.java](https://github.com/lianjiatech/bella-openapi/blob/develop/api/server/src/main/java/com/ke/bella/openapi/endpoints/AudioController.java#L97-L104) The controller extracts the request URI, invokes the `ChannelRouter` to select a provider, and delegates the WebSocket lifecycle to `RealTimeHandler`. ## Channel Routing and Provider Selection Before establishing the downstream connection, the system selects the appropriate ASR provider using the `ChannelRouter`. This component evaluates the requested model, cost constraints, and load-balancing configuration to choose a concrete implementation (e.g., Ke, Tencent, or Huoshan). ```java ChannelDB channel = router.route(endpoint, model, apikey, processData.isMock()); String protocol = processData.getProtocol(); String url = processData.getForwardUrl(); String channelInfo = channel.getChannelInfo(); RealTimeAdaptor adaptor = adaptorManager.getProtocolAdaptor( endpoint, protocol, RealTimeAdaptor.class); AsrProperty property = JacksonUtils.deserialize(channelInfo, adaptor.getPropertyClass()); ``` *Source:* Same file, lines 118-132. The `AsrProperty` object contains provider-specific credentials and configuration options required to authenticate with the third-party service. ## The RealTimeAdaptor Interface All ASR providers implement the `RealTimeAdaptor` interface, which defines the contract for establishing connections, sending audio, and managing the transcription lifecycle. ```java // src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeAdaptor.java public interface RealTimeAdaptor extends IProtocolAdaptor { default String endpoint() { return "/v1/audio/realtime"; } WebSocket startTranscription(String url, T property, RealTimeMessage request, WebSocketCallback callback); boolean sendAudioData(WebSocket ws, byte[] audio, WebSocketCallback cb); boolean stopTranscription(WebSocket ws, RealTimeMessage request, WebSocketCallback cb); void closeConnection(WebSocket ws); WebSocketCallback createCallback(Sender sender, EndpointProcessData pd, EndpointLogger logger, String taskId, RealTimeMessage req, T property); Class getPropertyClass(); } ``` *Source:* [RealTimeAdaptor.java](https://github.com/lianjiatech/bella-openapi/blob/develop/api/server/src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeAdaptor.java) This abstraction allows Bella OpenAPI to support multiple ASR backends without modifying the core WebSocket handling logic. ## Provider-Specific Implementation: KeAdaptor The default internal provider uses `KeAdaptor`, which implements the `RealTimeAdaptor` interface to communicate with the "Ke" ASR service. ```java // src/main/java/com/ke/bella/openapi/protocol/realtime/KeAdaptor.java @Override public WebSocket startTranscription(String url, RealtimeProperty property, RealTimeMessage request, WebSocketCallback callback) { Request.Builder builder = new Request.Builder() .url(url) .header("Authorization", apikey); WebSocket ws = HttpUtils.websocketRequest(builder.build(), new BellaWebSocketListener(callback)); ws.send(JacksonUtils.serialize(request)); callback.started(); return ws; } ``` *Source:* [KeAdaptor.java](https://github.com/lianjiatech/bella-openapi/blob/develop/api/server/src/main/java/com/ke/bella/openapi/protocol/realtime/KeAdaptor.java#L18-L48) The adaptor creates an OkHttp `WebSocket` connection to the external service, sends the initial `StartTranscription` payload, and returns the socket for subsequent audio streaming. Audio frames are forwarded as binary messages: ```java @Override public boolean sendAudioData(WebSocket ws, byte[] audio, WebSocketCallback cb) { return ws.send(ByteString.of(audio)); } ``` *Source:* Same file, lines 52-54. ## Core WebSocket Handler and Message Flow The `RealTimeHandler` manages the full-duplex communication between the client and the selected ASR provider. It processes both text control messages and binary audio data. ```java // src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeHandler.java @Override protected void handleTextMessage(WebSocketSession session, TextMessage message) { // parse control messages (START_TRANSCRIPTION / STOP_TRANSCRIPTION) } @Override protected void handleBinaryMessage(WebSocketSession session, BinaryMessage message) { // forward binary audio to external ASR WebSocket } private void handleStartTranscription(...){ callback = adaptor.createCallback(...); ws = adaptor.startTranscription(url, property, request, callback); } private void handleStopTranscription(...){ adaptor.stopTranscription(ws, request, callback); } ``` *Source:* [RealTimeHandler.java](https://github.com/lianjiatech/bella-openapi/blob/develop/api/server/src/main/java/com/ke/bella/openapi/protocol/realtime/RealTimeHandler.java) The handler maintains the `taskId`, the external WebSocket connection (`ws`), and delegates all I/O operations to the provider-specific adaptor. ## Callback Processing and Metrics The `KeRealtimeCallback` processes incoming messages from the ASR provider, translates them to the client format, and records performance metrics. ```java // src/main/java/com/ke/bella/openapi/protocol/realtime/KeRealtimeCallback.java @Override public void onMessage(WebSocket webSocket, String text) { RealTimeMessage msg = JacksonUtils.deserialize(text, RealTimeMessage.class); // forward most messages to client; intercept STARTED, TTS_TTFT, SESSION_CLOSE, TASK_FAILED } private void complete() { // record total latency, close client session, log metrics } ``` *Source:* [KeRealtimeCallback.java](https://github.com/lianjiatech/bella-openapi/blob/develop/api/server/src/main/java/com/ke/bella/openapi/protocol/realtime/KeRealtimeCallback.java) The callback pushes incremental transcription results back to the client via the `Sender` interface, captures metrics such as total latency (TTLT) and time-to-first-token (TTS), and ensures clean session termination when the provider signals completion. ## Client Integration Example Clients connect to Bella OpenAPI using standard WebSocket APIs. The following JavaScript example demonstrates connecting to the real-time ASR streaming endpoint, sending audio, and handling incremental results. ```javascript // Connect to Bella OpenAPI real-time ASR const socket = new WebSocket( "wss://api.example.com/v1/audio/asr/stream?model=gpt-4o-asr" ); // 1️⃣ Start transcription socket.addEventListener("open", () => { const startMsg = { header: { name: "StartTranscription", taskId: crypto.randomUUID() }, payload: { /* optional TTS/LLM options */ } }; socket.send(JSON.stringify(startMsg)); }); // 2️⃣ Stream binary audio (e.g., PCM 16-bit, 16 kHz) function sendAudioChunk(chunkArrayBuffer) { socket.send(chunkArrayBuffer); // binary frame } // 3️⃣ Handle incremental results socket.addEventListener("message", event => { const msg = JSON.parse(event.data); if (msg.header.name === "SentenceEnd") { console.log("Transcribed:", msg.payload.text); } else if (msg.header.name === "TranscriptionFinished") { console.log("Final result:", msg.payload); socket.close(); } }); ``` The client sends a `StartTranscription` control message, followed by binary PCM audio chunks. Bella OpenAPI forwards these to the selected provider, and each incremental result (such as `SentenceEnd` or `TranscriptionFinished`) is pushed back to the client in the standardized `RealTimeMessage` format. ## Summary - **Bella OpenAPI** exposes real-time ASR streaming through WebSocket endpoints `/v1/audio/asr/stream` and `/realtime`, handled by `AudioController`. - **Channel routing** via `ChannelRouter` selects the appropriate ASR provider (Ke, Tencent, Huoshan) based on model and cost configuration. - **Protocol abstraction** is achieved through the `RealTimeAdaptor` interface, allowing provider-specific implementations like `KeAdaptor` to manage external WebSocket connections. - **Message handling** occurs in `RealTimeHandler`, which processes client text commands and binary audio, forwarding them to the provider via the adaptor. - **Result translation** is handled by callbacks such as `KeRealtimeCallback`, which convert provider responses into `RealTimeMessage` DTOs and record performance metrics. - **Client integration** requires only standard WebSocket support, sending JSON control messages and binary audio frames to receive incremental transcriptions. ## Frequently Asked Questions ### What audio format does Bella OpenAPI expect for real-time ASR streaming? Bella OpenAPI accepts binary audio frames in standard formats such as PCM 16-bit at 16 kHz. The `RealTimeHandler` forwards these binary messages directly to the provider's WebSocket via the `sendAudioData` method in the `RealTimeAdaptor` implementation, without transcoding. Specific format requirements depend on the selected provider channel configured in `AsrProperty`. ### How does Bella OpenAPI handle provider failover during an active streaming session? The `ChannelRouter` selects the provider channel at the start of the session when `handleStartTranscription` is invoked in `RealTimeHandler`. If the downstream provider connection fails, the `WebSocketCallback` (such as `KeRealtimeCallback`) detects the failure via `onFailure` or `onClosing` events, records the error metrics, and terminates the session. Clients must initiate a new WebSocket connection to trigger a fresh channel selection for retry. ### What is the difference between `SentenceEnd` and `TranscriptionFinished` messages? `SentenceEnd` indicates an intermediate result where the provider has detected a complete sentence boundary but the session remains active for additional audio. `TranscriptionFinished` signals the final result after the client sends `StopTranscription` or the provider closes the session. The `KeRealtimeCallback` intercepts these message types in its `onMessage` method and forwards them to the client via the `Sender` interface, allowing applications to render partial transcripts while waiting for completion. ### Can I use Bella OpenAPI with custom ASR providers not included in the default distribution? Yes, the architecture supports custom providers through the `RealTimeAdaptor` interface. You must implement `startTranscription`, `sendAudioData`, `stopTranscription`, and `createCallback` methods to handle the provider's specific WebSocket protocol and authentication. Register your adaptor implementation with the `adaptorManager` using the endpoint and protocol identifiers. The `RealTimeHandler` will then delegate all audio streaming and message translation to your custom implementation without requiring changes to the core WebSocket handling logic.