Implementing Multi-Speaker Podcast Generation in Open-Notebook: A Complete Technical Guide

Open-Notebook supports fully-featured multi-speaker podcast generation by orchestrating distinct voice models per speaker through SurrealDB-backed profiles and the podcast-creator library.

The lfnovo/open-notebook repository provides a production-ready architecture for implementing multi-speaker podcast generation, enabling dynamic audio synthesis with up to four distinct speakers per episode. This system decouples speaker configuration from episode logic, allowing each participant to utilize custom text-to-speech (TTS) models while maintaining a unified generation pipeline.

Architecture Overview

The multi-speaker implementation spans four distinct layers: domain models for data persistence, a service layer for job orchestration, command handlers for background processing, and REST endpoints for client interaction.

Domain Models

In open_notebook/podcasts/models.py, three primary classes define the data structures stored in SurrealDB:

• EpisodeProfile – Links to a speaker configuration via speaker_config and stores model IDs for outline and transcript generation.
• SpeakerProfile – Contains a speakers array supporting up to four entries. Each speaker definition includes name, voice_id, backstory, personality, and an optional voice_model field that overrides the profile-wide TTS model.
• PodcastEpisode – Records episode metadata, the associated command ID, and generated assets including the audio file path and transcript.

Service Layer

The api/podcast_service.py file implements PodcastService, which coordinates job submission and status tracking:

• submit_generation_job validates requested profiles, resolves notebook content when supplied, and queues the background job via surreal-commands.
• list_episodes, get_episode, and get_job_status provide read-only APIs consumed by the frontend to display generation progress.

Command Implementation

Heavy-weight processing occurs in commands/podcast_commands.py within the generate_podcast_command function. This worker:

1. Loads the selected EpisodeProfile and its linked SpeakerProfile.
2. Resolves all model IDs (outline, transcript, TTS) to concrete provider/model/config triples using _resolve_model_config.
3. Mass-converts episode and speaker profiles to the JSON format expected by the podcast-creator library.
4. Iterates over SpeakerProfile.speakers to attach individual voice_model configurations when specified, enabling per-speaker TTS selection.

API Router

HTTP endpoints in api/routers/podcasts.py expose the service to clients:

• POST /podcasts/generate triggers submit_generation_job and returns a job ID.
• GET /podcasts/episodes returns episode lists with job_status and streaming URLs.
• GET /podcasts/episodes/{episode_id}/audio streams the final MP3 file.

Configuring Multi-Speaker Profiles

Implementing multi-speaker synthesis requires defining speaker configurations before episode generation.

Creating a SpeakerProfile

The SpeakerProfile class stores the speakers array where each entry contains:

• name – Human-readable identifier (e.g., "Narrator", "Expert").
• voice_id, backstory, personality – Metadata guiding the LLM to maintain distinct voices.
• voice_model – Optional Model record ID enabling per-speaker TTS overrides.

When voice_model is omitted for a specific speaker, the system falls back to the profile-wide voice_model defined in the SpeakerProfile instance.

Linking to EpisodeProfile

The EpisodeProfile.speaker_config field stores the name of the SpeakerProfile to use. During generation, the command worker fetches this profile and transforms its speakers into the JSON structure required by the podcast-creator library.

The Generation Pipeline

When you submit a generation request, the system executes an asynchronous pipeline that transforms text content into multi-speaker audio.

Job Submission and Validation

Calling PodcastService.submit_generation_job performs the following:

1. Validates that the requested episode_profile and speaker_profile exist in SurrealDB.
2. Resolves the notebook content if notebook_id is provided in the request.
3. Creates a command record and queues it for background processing.

Background Processing with podcast-creator

The generate_podcast_command worker executes the synthesis:

1. Outline Generation – Uses the outline_llm specified in EpisodeProfile to create a structured episode plan.
2. Transcript Generation – Generates the full conversational script using the transcript_llm, applying speaker definitions to maintain distinct voices.
3. Multi-Speaker Audio Synthesis – Splits the transcript into chunks per speaker, resolves individual TTS models (honoring voice_model overrides), and concatenates the resulting audio streams into a single MP3.

Because each speaker can declare its own voice_model, the system can mix voices (e.g., OpenAI's "alloy" for a host and ElevenLabs' English Female V2 for a guest) without code modifications.

Audio Streaming and Retrieval

Upon completion, the episode record updates with audio_url pointing to the generated file. Clients retrieve this via GET /podcasts/episodes/{episode_id}/audio, which streams the MP3 directly from storage.

Implementation Examples

1. Define a Multi-Speaker Profile

from open_notebook.podcasts.models import SpeakerProfile

await SpeakerProfile(
    name="InterviewShow",
    description="Host + Guest conversation",
    voice_model="model:openai:gpt-4o-mini",
    speakers=[
        {
            "name": "Host",
            "voice_id": "alloy",
            "backstory": "A seasoned tech journalist.",
            "personality": "Friendly, inquisitive",
            # Uses the profile-wide TTS model

        },
        {
            "name": "Guest",
            "voice_id": "fable",
            "backstory": "AI researcher with a British accent.",
            "personality": "Thoughtful, measured",
            "voice_model": "model:elevenlabs:eleven_english_female_v2"
        },
    ],
).save()

Source: open_notebook/podcasts/models.py

2. Create an Episode Profile

from open_notebook.podcasts.models import EpisodeProfile

await EpisodeProfile(
    name="AI_Trend_Episode",
    description="Weekly AI trends roundup",
    speaker_config="InterviewShow",
    outline_llm="model:openai:gpt-4o-mini",
    transcript_llm="model:openai:gpt-4o",
    default_briefing="Summarize recent AI research and let the Host interview the Guest.",
    num_segments=6,
).save()

Source: open_notebook/podcasts/models.py

3. Submit a Generation Job

import httpx

payload = {
    "episode_profile": "AI_Trend_Episode",
    "speaker_profile": "InterviewShow",
    "episode_name": "AI Trends – Week 42",
    "notebook_id": "notebook:my_notebook",
    "briefing_suffix": "Include a short outro with a call-to-action."
}

resp = httpx.post("http://localhost:5055/podcasts/generate", json=payload)
job = resp.json()
print("Job submitted:", job["job_id"])

Source: api/routers/podcasts.py

4. Poll for Completion

import time
import httpx

job_id = job["job_id"]
while True:
    status = httpx.get(f"http://localhost:5055/podcasts/jobs/{job_id}").json()
    print(status["status"])
    if status["status"] in ("completed", "failed"):
        break
    time.sleep(5)

Source: api/podcast_service.py

5. Retrieve the Audio URL

episodes = httpx.get("http://localhost:5055/podcasts/episodes").json()
for ep in episodes:
    if ep["name"] == "AI Trends – Week 42":
        print("Audio URL:", ep["audio_url"])
        break

Source: api/routers/podcasts.py

Summary

• Multi-speaker support is implemented through the SpeakerProfile domain model in open_notebook/podcasts/models.py, which stores up to four speaker definitions per profile.
• Per-speaker voice customization occurs via the optional voice_model field, allowing individual TTS models that override the profile default.
• Asynchronous generation is handled by generate_podcast_command in commands/podcast_commands.py, which orchestrates the podcast-creator library.
• Job lifecycle management is provided by PodcastService in api/podcast_service.py, handling submission, validation, and status polling.
• Audio delivery is exposed through REST endpoints in api/routers/podcasts.py, supporting both metadata retrieval and MP3 streaming.

Frequently Asked Questions

How many speakers does Open-Notebook support per podcast episode?

Open-Notebook supports up to four speakers per episode, defined as an array within the SpeakerProfile model. Each speaker requires a unique name, voice characteristics, and optional TTS model configuration.

Can each speaker use a different text-to-speech model?

Yes. While SpeakerProfile defines a default voice_model for the entire profile, individual speakers can override this by specifying their own voice_model record ID. This enables mixing providers (e.g., OpenAI for one speaker, ElevenLabs for another) within a single episode.

What happens if a speaker doesn't specify a voice_model?

If a speaker entry lacks a voice_model field, the system automatically falls back to the profile-wide voice_model defined in the parent SpeakerProfile instance. If neither is specified, the generation command will fail during model resolution in commands/podcast_commands.py.

How do I monitor the status of a podcast generation job?

Submit a POST request to /podcasts/generate to receive a job ID, then poll GET /podcasts/jobs/{job_id} to retrieve status updates. The PodcastService.get_job_status method in api/podcast_service.py returns states including "pending", "running", "completed", or "failed", allowing clients to track progress until the audio file becomes available.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}