How Meetily Handles Incremental Transcript Processing During Live Recording

Meetily captures audio in real-time, applies Voice Activity Detection (VAD) to isolate speech segments, processes them through the Whisper engine, and emits partial transcripts incrementally via Tauri events while simultaneously persisting checkpoints to disk for fault tolerance.

Meetily is an open-source meeting transcription application developed by Zackriya-Solutions that transcribes speech live during recording sessions. Unlike traditional batch processing that delays transcription until the meeting ends, this architecture streams partial results to the user interface immediately while maintaining durable checkpoints. This approach ensures users see live captions within seconds of speaking and protects data against application crashes.

The Four-Stage Incremental Pipeline

The incremental transcript processing system consists of four tightly-coupled components that operate concurrently during a live recording session.

Stage 1: Audio Capture and VAD Filtering

The process begins in src/audio/pipeline.rs, where the Pipeline receives PCM frames from the audio capture modules located in src/audio/capture/*. The pipeline buffers the incoming microphone stream and applies a Voice Activity Detection (VAD) filter implemented in src/audio/vad.rs. This filter identifies speech-only segments and discards silence, producing structured AudioChunk objects that contain only relevant audio data. By filtering out non-speech audio before transcription, the system dramatically reduces the computational load on the heavy transcription model.

Stage 2: Async Chunk Dispatch

Each AudioChunk is handed to the transcription worker defined in src/audio/transcription/worker.rs. This worker operates on a separate async task to prevent blocking the main audio capture thread. The worker forwards the chunk to the configured transcription provider, which can be either Whisper-cpp or Parakeet, depending on the system configuration. This asynchronous dispatch ensures that audio capture continues uninterrupted while transcription proceeds in parallel.

Stage 3: Whisper Transcription Processing

The provider—either whisper_provider.rs or parakeet_provider.rs—calls the native Whisper library to process the speech chunk. The engine returns a partial transcript for the received audio segment. The provider wraps this result in a TranscriptUpdate struct and sends it back to the pipeline through an async channel. This immediate return of partial results, rather than waiting for the entire meeting to conclude, is the core mechanism that enables live captioning.

Stage 4: Real-Time Emission and Persistence

Upon receiving the TranscriptUpdate, the pipeline performs two critical operations simultaneously. First, it emits a Tauri event named transcript-update that the React frontend listens for, allowing the UI to append new text to the live transcript view immediately. Second, it passes the same update to the incremental audio-saver in src/audio/incremental_saver.rs, which stores a checkpoint containing both the raw audio segment and its associated transcript fragment in the .checkpoints/ directory. This dual-path emission ensures that even if the application crashes, the partially-saved transcript can be recovered from disk.

Why Incremental Processing Matters

The incremental architecture provides three distinct advantages over batch transcription:

• Responsiveness: Users see captions appear seconds after they speak, mirroring the experience of professional live-captioning services.
• Fault-tolerance: Checkpoints containing audio and transcript fragments are written to disk as soon as they are produced, ensuring that a crash does not discard work already completed.
• Resource Efficiency: The VAD system filters out silence before sending data to Whisper, significantly reducing the amount of audio the transcription model must process and lowering CPU usage.

Implementation Code Examples

Starting a Recording Session

The frontend initiates the incremental pipeline by invoking the Tauri command start_recording, which creates a RecordingManager instance that spawns the audio pipeline described above.

await invoke('start_recording', {
  mic_device_name: 'Built-in Microphone',
  system_device_name: 'BlackHole 2ch',
  meeting_name: 'Team Stand-up',
});

Listening for Live Transcript Updates

The React frontend listens for transcript-update events to append new text to the live view as it becomes available.

listen<TranscriptUpdate>('transcript-update', (event) => {
  setTranscripts(prev => [...prev, event.payload.text]);
});

Each TranscriptUpdate contains the freshly-transcribed text for the latest speech chunk, enabling real-time UI updates.

Recovering from Checkpoints

If the application needs to recover from a crash or when finalizing the recording, the system uses incremental_saver.rs to merge checkpoints into the final output.

let checkpoints = audio::incremental_saver::recover_audio_from_checkpoints(meeting_path)?;
for chk in checkpoints {
    println!("Chunk {} – {} bytes, {} transcript chars", chk.id, chk.audio.len(), chk.transcript.len());
}

When the user stops the recording, the incremental_saver merges all checkpoints into the final audio.mp4 file and a complete transcript file, ensuring no data is lost.

Summary

• Incremental transcript processing in Meetily enables real-time captioning by processing audio chunks as they are captured rather than waiting for recording completion.
• The Voice Activity Detection system in src/audio/vad.rs filters silence before transcription, optimizing resource usage.
• Async workers in src/audio/transcription/worker.rs handle transcription without blocking the audio capture thread.
• Dual-path emission sends transcript updates to both the UI via Tauri events and to disk via incremental_saver.rs for fault tolerance.
• Checkpoint-based persistence ensures that partial meeting transcripts survive application crashes and can be recovered automatically.

Frequently Asked Questions

How does Meetily handle crashes during live recording?

Meetily writes incremental checkpoints to disk immediately upon receiving each transcript update. These checkpoints, stored in the .checkpoints/ directory by src/audio/incremental_saver.rs, contain both the raw audio segment and its corresponding transcript fragment. If the application crashes, the recover_audio_from_checkpoints function can reconstruct the meeting data up to the last successful checkpoint, preventing data loss.

What is the role of Voice Activity Detection in the pipeline?

Voice Activity Detection (VAD) serves as a pre-filter that identifies speech segments within the raw audio stream and discards silence or non-speech noise. Implemented in src/audio/vad.rs and applied in src/audio/pipeline.rs, VAD ensures that only relevant AudioChunk objects are sent to the transcription engine. This reduces CPU usage and improves transcription accuracy by eliminating unnecessary processing of silent periods.

How does the frontend receive real-time transcript updates?

The frontend receives updates through Tauri's event system. When the Rust backend in src/audio/pipeline.rs receives a TranscriptUpdate from the transcription worker, it emits a transcript-update event. The React frontend listens for this event using the listen<TranscriptUpdate>() function and immediately appends the new text to the live transcript view, creating a seamless real-time captioning experience.

Where are incremental checkpoints stored and how are they finalized?

Checkpoints are stored in the .checkpoints/ directory within the meeting's folder path by src/audio/incremental_saver.rs. Each checkpoint contains a serialized audio segment and its transcript. When the user stops the recording via the stop_recording command, the incremental_saver merges all checkpoint files into the final audio.mp4 and a complete transcript file, then cleans up the temporary checkpoint directory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}