How the Coco App Transcription Command Processes Audio Input and Returns Text

The transcription command in infinilabs/coco-app captures audio as Base64, routes it through Tauri's Rust backend to a remote Coco server, polls asynchronously for up to 30 seconds, and returns cleaned text with special markup tags stripped.

The transcription command provides end-to-end speech-to-text functionality by bridging React frontend components with Rust backend logic. This implementation handles audio encoding, HTTP-based server communication, and resilient polling to deliver accurate transcripts. Understanding this flow reveals how the Coco App leverages Tauri to combine web UI flexibility with native backend performance.

Step 1: Frontend Audio Capture and Base64 Encoding

Recording and Blob Conversion

The process begins in the AudioRecording component located at src/components/AudioRecording/index.tsx. When a user stops recording, the component triggers a record-end event and converts the captured audio Blob into a Base64 string.

// src/components/AudioRecording/index.tsx (lines 66-78)
const reader = new FileReader();
reader.readAsDataURL(audioBlob);
reader.onloadend = async () => {
  const base64Audio = reader.result as string;
  // base64Audio contains the data URL prefix + Base64 content
};

The Base64 string is then packaged into a JSON payload containing the serverId and audioContent fields required by the backend.

Step 2: Command Invocation Through the Platform Adapter

The frontend invokes the transcription command via the platformAdapter, which abstracts Tauri-specific implementation details. The adapter wraps Tauri's invoke function to bridge TypeScript and Rust.

// Frontend invocation pattern
const response: any = await platformAdapter.commands(
  "transcription",
  {
    serverId: currentService.id,
    audioContent: JSON.stringify({ content: base64Audio }),
  }
);

Source files:

• Platform abstraction: src/utils/platformAdapter.ts
• Tauri bridge implementation: src/utils/tauriAdapter.ts (around line 37)

This call routes through the commands wrapper in tauriAdapter.ts, which ultimately invokes the registered Rust command using Tauri's invoke API.

Step 3: Tauri Backend Command Registration

The Rust backend registers the transcription command in src-tauri/src/lib.rs at line 134:

// src-tauri/src/lib.rs
.invoke_handler(tauri::generate_handler![
    // ... other commands
    server::transcription::transcription,
])

When the frontend calls "transcription", Tauri executes the server::transcription::transcription function defined in src-tauri/src/server/transcription.rs. This Rust module handles all HTTP communication with the remote Coco server.

Step 4: Server Communication and Asynchronous Polling

Initial Task Creation

The backend immediately posts the Base64 audio content to the server's transcription endpoint:

// src-tauri/src/server/transcription.rs (lines 18-25)
let init_response = HttpClient::post(
    &server_id,
    "/services/audio/transcription",
    None,
    Some(audio_content.into()),
).await?;

let task_id = init_response.task_id;

The server responds with a task_id that identifies the asynchronous transcription job.

Polling for Results

The backend enters a polling loop (lines 58-88) that repeatedly queries the task status:

// Polling implementation excerpt
loop {
    let poll_response = HttpClient::get(
        &server_id,
        &format!("/services/audio/task/{}", transcription_task_id),
        None,
    ).await?;
    
    let body = get_response_body_text(poll_response).await?;
    let transcription_response: TranscriptionResponse = serde_json::from_str(&body)?;
    
    if !transcription_response.results.is_empty() {
        return Ok(transcription_response);
    }
    
    if start.elapsed() >= Duration::from_secs(30) {
        return Err("Transcription timeout".into());
    }
    
    tokio::time::sleep(Duration::from_millis(200)).await;
}

Key characteristics:

• Polling interval: 200 milliseconds between requests
• Timeout threshold: 30 seconds maximum wait time
• Completion trigger: Non-empty results array in the response

Step 5: Response Processing and Text Extraction

The shared TypeScript types in src/types/commands.ts (lines 174-176) define the response structure:

export interface TranscriptionResponse {
  text: string;
}

However, the raw response often contains nested arrays and formatting tags. The frontend extracts readable text by flattening the results structure and applying regex cleaning:

// src/components/AudioRecording/index.tsx (lines 85-89)
const text = response?.results
  .flatMap((item: any) => item?.transcription?.transcripts)
  .map((item: any) => item?.text?.replace(/<\|[\/\w]+\|>/g, ""))
  .join(" ");

This processing:

1. Flattens nested transcription.transcripts arrays
2. Strips special markup tags matching the pattern <|...|>
3. Joins all transcript fragments into a single coherent string

Error Handling and Resilience

Errors at any stage bubble up to the invokeWithErrorHandler utility in src/commands/servers.ts (lines 37-101). This centralizes error logging via the addError function and presents user-friendly messages in the UI.

The transcription wrapper function (lines 364-368) in the same file provides a typed interface for components that need direct access without using the platform adapter:

// src/commands/servers.ts
export const transcription = (payload: TranscriptionPayload) => {
  return invokeWithErrorHandler("transcription", payload);
};

Code Implementation Examples

Triggering Transcription from a Custom Component

import platformAdapter from "@/utils/platformAdapter";

async function transcribeUserAudio(serverId: string, audioBlob: Blob) {
  // Convert to Base64
  const reader = new FileReader();
  reader.readAsDataURL(audioBlob);
  
  return new Promise((resolve) => {
    reader.onloadend = async () => {
      const base64Audio = reader.result as string;
      
      const payload = {
        serverId,
        audioContent: JSON.stringify({ content: base64Audio }),
      };
      
      const resp = await platformAdapter.commands("transcription", payload);
      
      // Extract and clean text
      const text = resp?.results
        .flatMap((r: any) => r?.transcription?.transcripts)
        .map((t: any) => t?.text?.replace(/<\|[\/\w]+\|>/g, ""))
        .join(" ");
        
      resolve(text);
    };
  });
}

Direct TypeScript API Call

import { transcription } from "@/commands/servers";

async function simpleTranscribe(serverId: string, base64Data: string) {
  const payload = {
    serverId,
    audioType: "audio/mpeg",
    audioContent: base64Data,
  };
  
  const { results } = await transcription(payload);
  
  return results
    .flatMap((r: any) => r?.transcription?.transcripts)
    .map((t: any) => t?.text?.replace(/<\|[\/\w]+\|>/g, ""))
    .join(" ");
}

Rust Polling Logic (Simplified)

use std::time::{Duration, Instant};

async fn poll_transcription_result(
    server_id: &str, 
    task_id: &str
) -> Result<TranscriptionResponse, String> {
    let start = Instant::now();
    
    loop {
        let response = HttpClient::get(
            server_id,
            &format!("/services/audio/task/{}", task_id),
            None
        ).await.map_err(|e| e.to_string())?;
        
        let body = get_response_body_text(response).await?;
        let resp: TranscriptionResponse = serde_json::from_str(&body)
            .map_err(|e| e.to_string())?;
            
        if !resp.results.is_empty() {
            return Ok(resp);
        }
        
        if start.elapsed() >= Duration::from_secs(30) {
            return Err("Transcription timeout".into());
        }
        
        tokio::time::sleep(Duration::from_millis(200)).await;
    }
}

Summary

• Audio Input: The transcription command accepts Base64-encoded audio content from the AudioRecording component in src/components/AudioRecording/index.tsx
• Command Routing: Frontend calls route through platformAdapter to the Rust backend registered in src-tauri/src/lib.rs
• Server Communication: The Rust implementation in src-tauri/src/server/transcription.rs POSTs audio to /services/audio/transcription and receives a task ID
• Asynchronous Polling: The backend polls /services/audio/task/{task_id} every 200ms with a 30-second timeout
• Text Processing: Results undergo flattening and regex cleaning (/<\|[\/\w]+\|>/g) to remove markup tags before display
• Error Handling: Centralized through invokeWithErrorHandler in src/commands/servers.ts with user-facing error propagation

Frequently Asked Questions

What audio format does the transcription command expect?

The transcription command accepts audio encoded as Base64 strings. While the audioType parameter (e.g., "audio/mpeg") can specify the MIME type, the backend primarily processes the Base64 content posted to the /services/audio/transcription endpoint. The AudioRecording component handles the Blob-to-Base64 conversion automatically.

How long does the transcription command wait for server results?

The Rust backend implements a 30-second timeout with 200-millisecond polling intervals. After submitting audio to the server, the transcription function polls the task status endpoint until results are available or the timeout elapses, whichever occurs first.

How does the app handle transcription errors?

Errors propagate through the invokeWithErrorHandler utility in src/commands/servers.ts. Network failures, server errors, or timeout conditions are caught, logged via the addError function, and presented to users through the application's error notification system. Components can wrap transcription calls in try-catch blocks to implement custom fallback behavior.

Can I use the transcription command outside the AudioRecording component?

Yes. The transcription API is exposed through two interfaces: the platformAdapter (platformAdapter.commands("transcription", payload)) for generic use, and the direct transcription wrapper function exported from src/commands/servers.ts. Both accept a TranscriptionPayload containing serverId and audioContent, allowing any component or utility function to initiate speech-to-text processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}