How Podcast Generation Works with the Job Queue in Open Notebook

Open Notebook executes podcast generation as asynchronous background jobs through the surreal-commands queue system, allowing the API to return a job ID immediately while long-running audio synthesis processing occurs in the background.

Open Notebook leverages the surreal-commands job queue architecture to decouple resource-intensive text-to-speech processing from HTTP request handling. When users request a podcast episode, the system validates configuration profiles, enqueues a background task, and returns a job identifier that clients poll for completion status. This design ensures the API remains responsive even when generating multi-minute audio content.

Architecture Overview

The Async Job Queue Pattern

The implementation relies on surreal-commands to manage distributed task execution. Rather than blocking the HTTP thread during LLM inference and audio generation, Open Notebook delegates work to background workers. This pattern separates concerns between the FastAPI router layer (api/routers/podcasts.py), the business logic service (api/podcast_service.py), and the actual command implementation (commands/podcast_commands.py).

Key components include:

• Command Registration: Background commands are auto-registered when commands/podcast_commands is imported
• Job Enqueueing: submit_command("open_notebook", "generate_podcast", ...) places work in the queue
• Status Tracking: Jobs maintain state (submitted, running, completed, failed) with timestamps and error messages
• File Isolation: UUID-based output directories prevent collision between concurrent generation jobs

The Podcast Generation Workflow

Step 1: API Request and Validation

Clients initiate generation by POSTing to /podcasts/generate with a PodcastGenerationRequest payload containing episode_profile, speaker_profile, and optionally notebook_id or raw content. The router in api/routers/podcasts.py (lines 41-56) delegates to PodcastService.submit_generation_job.

The service layer validates that referenced profiles exist by calling EpisodeProfile.get_by_name and SpeakerProfile.get_by_name. If validation fails, the API returns a 404 error before any queue operations occur.

Step 2: Content Preparation

When notebook_id is provided without explicit content, the service retrieves notebook context via Notebook.get followed by notebook.get_context (as implemented in api/podcast_service.py lines 46-68). This extracts the text content that will form the basis of the podcast script.

Step 3: Job Enqueueing

After validation, the service ensures the command module is loaded (import commands.podcast_commands to trigger registration), then calls submit_command("open_notebook", "generate_podcast", command_args) as shown in api/podcast_service.py lines 95-104. This returns a job_id that the API immediately returns to the client, enabling non-blocking operation.

Step 4: Background Execution

The queued job invokes generate_podcast_command defined in commands/podcast_commands.py (lines 69-74). This function performs the heavy lifting:

• Loads episode and speaker configurations from SurrealDB
• Resolves LLM and TTS model configurations via _resolve_model_config
• Creates a UUID-based output directory using build_episode_output_dir (lines 26-38) to store generated files safely
• Configures the podcast-creator library with speaker and episode profiles
• Executes await create_podcast(...) (lines 31-35, 44-52) to generate audio
• Persists results to a PodcastEpisode record via episode.save() (lines 28-33, 56-64), storing file paths, transcript, and outline

Step 5: Status Polling and Monitoring

Clients query /podcasts/jobs/{job_id} to track progress. The router calls PodcastService.get_job_status, which forwards to surreal_commands.get_command_status (api/podcast_service.py lines 14-34). The response includes status, result, error_message, created/updated timestamps, and progress indicators.

Step 6: Audio Delivery and Retry Mechanisms

Once completed, episodes appear in /podcasts/episodes. Each entry includes an audio_url path that streams via /podcasts/episodes/{episode_id}/audio using FastAPI's FileResponse (implemented in api/routers/podcasts.py lines 90-115).

If generation fails, clients can POST to /podcasts/episodes/{episode_id}/retry (lines 54-61). This deletes the failed record and its associated audio file, then submits a fresh job using the original profiles. Direct episode deletion (lines 72-80) also cleans up underlying audio files from disk.

API Usage Examples

Submit a Generation Request

curl -X POST http://localhost:5055/api/podcasts/generate \
  -H "Content-Type: application/json" \
  -d '{
        "episode_profile": "MyEpisode",
        "speaker_profile": "MySpeaker",
        "episode_name": "AI Trends 2024",
        "notebook_id": "notebook123"
      }'

Response:

{
  "job_id": "cmd-01HZ2X...",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "MyEpisode",
  "episode_name": "AI Trends 2024"
}

Poll Job Status Programmatically

import httpx

base = "http://localhost:5055/api"
job_id = "cmd-01HZ2X..."

resp = httpx.get(f"{base}/podcasts/jobs/{job_id}")
print(resp.json())

Typical completed payload:

{
  "job_id": "cmd-01HZ2X...",
  "status": "completed",
  "result": null,
  "error_message": null,
  "created": "2026-06-23T12:34:56Z",
  "updated": "2026-06-23T12:35:30Z",
  "progress": null
}

List Generated Episodes

curl http://localhost:5055/api/podcasts/episodes

Each response entry contains audio_url for streaming.

Stream Audio Content

ffplay http://localhost:5055/api/podcasts/episodes/ep-01ABCD/audio

Retry Failed Episodes

curl -X POST http://localhost:5055/api/podcasts/episodes/ep-01ABCD/retry

Returns a new job_id for the retry attempt.

Key Implementation Files

Component	File Path	Purpose
Command Definition	[commands/podcast_commands.py](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)	Implements generate_podcast_command that executes in background workers, handles profile resolution, and manages audio generation via podcast-creator
Service Layer	[api/podcast_service.py](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)	Validates input parameters, orchestrates content fetching, submits jobs to surreal-commands queue, and provides status checking utilities
API Router	[api/routers/podcasts.py](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)	Exposes REST endpoints for job submission (/generate), status polling (/jobs/{id}), episode listing, audio streaming, retry operations, and deletion
Documentation	[docs/2-CORE-CONCEPTS/podcasts-explained.md](https://github.com/lfnovo/open-notebook/blob/main/docs/2-CORE-CONCEPTS/podcasts-explained.md)	High-level architectural documentation explaining podcast concepts and configuration profiles
Test Suite	[tests/test_podcast_path.py](https://github.com/lfnovo/open-notebook/blob/main/tests/test_podcast_path.py)	Unit tests verifying API route integration and job queue behavior

Summary

• Open Notebook uses the surreal-commands job queue to run podcast generation asynchronously, preventing HTTP timeouts during long-running audio synthesis
• The workflow spans validation (EpisodeProfile.get_by_name, SpeakerProfile.get_by_name), content resolution (Notebook.get), job enqueueing (submit_command), and background execution (generate_podcast_command)
• Clients receive immediate job_id responses and poll /podcasts/jobs/{job_id} for completion status
• Generated episodes expose streaming URLs via /podcasts/episodes/{episode_id}/audio with automatic file cleanup on deletion or retry
• All audio files are isolated in UUID-based directories created by build_episode_output_dir to prevent naming collisions

Frequently Asked Questions

What job queue system does Open Notebook use for podcast generation?

Open Notebook uses surreal-commands, a command queue system that integrates with SurrealDB. According to the source code in api/podcast_service.py, the service calls submit_command("open_notebook", "generate_podcast", ...) to enqueue work and get_command_status to poll for updates. This system handles job persistence, worker distribution, and status tracking automatically.

How does Open Notebook handle content preparation when only a notebook ID is provided?

When the API receives a notebook_id without explicit content, the PodcastService loads the notebook via Notebook.get and extracts text using notebook.get_context (lines 46-68 in api/podcast_service.py). This content then populates the script that the podcast-creator library converts to audio during background processing.

Can failed podcast generation jobs be retried without reconfiguring profiles?

Yes. The /podcasts/episodes/{episode_id}/retry endpoint (implemented in api/routers/podcasts.py lines 54-61) allows clients to retry failed episodes. This operation deletes the existing failed record and its associated audio file from the UUID-based output directory, then submits a new generation job using the original episode_profile and speaker_profile configurations.

What happens to audio files when a podcast episode is deleted?

When deleting an episode via the API, Open Notebook removes both the database record and the underlying audio file from disk. As implemented in api/routers/podcasts.py lines 72-80, the deletion endpoint locates the file path stored in the PodcastEpisode record and performs filesystem cleanup to prevent orphaned audio files from consuming storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}