How Open-Notebook Generates Multi-Speaker Audio Asynchronously

The Open-Notebook podcast service employs a job-queue pattern that decouples HTTP requests from audio processing, using SurrealDB-backed commands and asynchronous workers to generate multi-speaker episodes without blocking the client.

The lfnovo/open-notebook repository implements a sophisticated podcast generation pipeline that handles complex text-to-speech (TTS) orchestration across multiple speakers. To generate multi-speaker audio asynchronously, the system separates job submission from execution, allowing the API to remain responsive while background workers handle the heavy LLM and TTS processing. This architecture ensures that generating a podcast with multiple voice actors does not timeout or freeze the frontend interface.

The Job Queue Architecture

The foundation of asynchronous generation lies in the command pattern implemented via SurrealDB. When a client requests a new podcast episode, the service does not process the audio immediately. Instead, it queues a command record that a separate worker process consumes later.

Submitting Generation Jobs

The entry point is PodcastService.submit_generation_job in api/podcast_service.py (lines 36-55). This method validates the requested episode profile and speaker profile by calling EpisodeProfile.get_by_name and SpeakerProfile.get_by_name. After validation, it constructs a command payload and delegates to surreal-commands.submit_command, which stores a SurrealDB command record and returns a unique job_id.

The HTTP response returns immediately with this job ID and a "submitted" status, freeing the client to perform other operations while the generation proceeds in the background.

Command Structure

Commands are defined in commands/podcast_commands.py using the @command decorator (lines 69-73). The handler generate_podcast_command is declared as async def, enabling non-blocking execution of I/O-bound TTS operations. This function accepts a PodcastGenerationInput model containing episode parameters, speaker configurations, and source content.

Asynchronous Execution Pipeline

Once queued, the Surreal-Commands worker picks up the job and executes the full generation pipeline asynchronously.

Resolving Model Configurations

The command handler first resolves LLM and TTS configurations through _resolve_model_config, injecting these into the podcast-creator library's profile dictionaries. It then builds a UUID-named output directory using build_episode_output_dir to ensure unique file paths for each episode.

Audio Generation

The core generation happens via create_podcast, an async wrapper around the podcast-creator library. According to the source code in commands/podcast_commands.py, this process:

1. Generates an outline using the configured outline LLM.
2. Produces a full transcript with the transcript LLM.
3. Synthesizes audio segments for each speaker using their specific TTS model defined in the speaker profile.

The library stitches these segments into a single audio file, producing a cohesive multi-speaker episode. Upon completion, the command returns a PodcastGenerationOutput containing success status, episode ID, file path, transcript, outline, and total processing time (lines 70-84).

Multi-Speaker Configuration

Multi-speaker capability is achieved through the speaker profile system. Each SpeakerProfile may contain an array of speakers, each with its own voice_model configuration.

As shown in commands/podcast_commands.py (lines 95-108), the command loops over these entries and resolves TTS configurations per-speaker. The resulting speakers_config dictionary maps each speaker to their designated voice, allowing the podcast-creator library to switch voices appropriately during segment synthesis.

Monitoring Job Status

Clients poll for completion using PodcastService.get_job_status (api/podcast_service.py, lines 15-33). This method queries surreal-commands.get_command_status to return the current state, including status, result, timestamps, and error messages. When the worker completes the command, it returns a PodcastGenerationOutput containing the final audio file path and metadata.

Implementation Examples

The following examples demonstrate how to interact with the podcast service programmatically.

Queue a Generation Job

# Submit a podcast generation request (typically called from a FastAPI endpoint)

job_id = await PodcastService.submit_generation_job(
    episode_profile_name="TechTalk",
    speaker_profile_name="MultiSpeaker",
    episode_name="AI Trends 2024",
    notebook_id="notebook--12345",   # or `content="raw text …"` instead

    briefing_suffix="Add a closing statement about privacy."
)
print(f"Job queued → {job_id}")

Poll for Completion

# Poll the job status until done

import asyncio

while True:
    status = await PodcastService.get_job_status(job_id)
    print(f"Status: {status['status']}")
    if status["status"] in ("completed", "failed"):
        break
    await asyncio.sleep(2)

if status["status"] == "completed":
    print(f"Podcast ready! Audio file: {status['result']['audio_file_path']}")

Direct Command Execution

For testing purposes, you can invoke the underlying command directly:

from commands.podcast_commands import generate_podcast_command, PodcastGenerationInput

input_data = PodcastGenerationInput(
    episode_profile="TechTalk",
    speaker_profile="MultiSpeaker",
    episode_name="AI Trends 2024",
    content="Full transcript text …",
    briefing_suffix="Add a comedic outro."
)

# This runs the full generation synchronously (useful for testing)

output = await generate_podcast_command(input_data)
print(output)

Summary

• The Open-Notebook podcast service uses a job-queue pattern with SurrealDB to handle asynchronous audio generation without blocking HTTP requests.
• Multi-speaker audio is configured through speaker profiles containing multiple voice models, which the system resolves per-speaker during the TTS phase.
• The PodcastService.submit_generation_job method in api/podcast_service.py queues commands, while generate_podcast_command in commands/podcast_commands.py executes the async pipeline.
• Clients monitor progress via PodcastService.get_job_status, receiving a PodcastGenerationOutput upon completion containing the final audio file path and metadata.
• The architecture decouples profile validation, LLM outline generation, transcript creation, and TTS synthesis into discrete, asynchronous steps orchestrated by the Surreal-Commands worker.

Frequently Asked Questions

How does the podcast service handle multiple speakers in a single audio file?

The service achieves multi-speaker output through the speaker profile configuration. Each SpeakerProfile can define multiple speakers with distinct voice_model settings. During execution, generate_podcast_command loops through these configurations (lines 95-108 in commands/podcast_commands.py) and passes a speakers_config dictionary to the podcast-creator library, which handles voice switching during segment synthesis.

What happens if the audio generation fails or times out?

Because generation runs asynchronously via the Surreal-Commands worker, HTTP timeouts are avoided. If processing fails, the worker updates the command record with an error status. When polling PodcastService.get_job_status, the client receives a failed status along with error details in the response, allowing for appropriate error handling without blocking the main application thread.

Can I use different LLM models for outline generation and transcript creation?

Yes. The system resolves separate model configurations for each phase through _resolve_model_config. The episode profile specifies which LLM handles outline generation versus transcript generation. These configurations are injected into the podcast-creator library, allowing you to use lightweight models for outlines and more powerful models for dialogue generation, or vice versa.

How do I check if a podcast generation job is complete?

Call PodcastService.get_job_status(job_id) with the ID returned from submit_generation_job. This queries the SurrealDB command status and returns a dictionary with the current status (pending, processing, completed, or failed), timestamps, and the result payload containing the PodcastGenerationOutput when finished.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}