Async Job Queue Architecture for Podcast Generation in Open Notebook

Open Notebook leverages the surreal-commands library built on SurrealDB to execute podcast generation asynchronously, returning a job ID immediately while processing heavy LLM and TTS workloads in the background.

The lfnovo/open-notebook repository implements a robust asynchronous job queue for podcast generation that separates HTTP request handling from resource-intensive audio synthesis tasks. This architecture uses SurrealDB as both the database and job queue backend through the surreal-commands library, enabling non-blocking podcast creation workflows that scale efficiently.

Architecture Overview

The system implements a producer-consumer pattern where the FastAPI application submits jobs to SurrealDB and background workers execute them. The flow spans three main layers: the REST API in api/routers/podcasts.py, the service layer in api/podcast_service.py, and the command definitions in commands/podcast_commands.py.

Job Submission Flow

REST Endpoint and Service Layer

When clients call POST /podcasts/generate, the request enters PodcastService.submit_generation_job in api/podcast_service.py (lines 45-99). This method validates episode profiles and speaker profiles, resolves source content from notebooks or direct text input, and prepares the job parameters for queueing.

Command Registration in SurrealDB

The service layer calls submit_command from the surreal-commands library to create a command record in SurrealDB. This returns a unique job ID (e.g., command:001abcdef) that clients use for polling. The command record initially stores status as pending along with serialized input data, allowing the HTTP response to return instantly while the heavy processing remains queued.

Background Processing Worker

Command Definition and Execution

The actual processing occurs in commands/podcast_commands.py (lines 69-85) via the function decorated with @command("generate_podcast", app="open_notebook"). When dequeued, the worker receives a PodcastGenerationInput model, loads relevant episode and speaker configurations, creates a UUID-based output directory, and resolves all language model settings before execution.

Third-Party Integration

The command invokes the external podcast-creator library to synthesize audio, generate transcripts, and create outlines. This library handles heavy operations including LLM API calls and text-to-speech processing outside the main application thread, preventing blocking of the FastAPI event loop.

Job Status Tracking and Polling

Status Retrieval

Clients poll GET /podcasts/jobs/{job_id} which delegates to PodcastService.get_job_status in api/podcast_service.py (lines 15-33). This calls get_command_status from surreal-commands to retrieve the current state, which can be pending, running, completed, or failed, along with timestamps and progress indicators.

Result Persistence

Upon completion, the command creates a PodcastEpisode record and links it to the original command ID using ensure_record_id. The record stores audio file paths, transcripts, and outlines, enabling retrieval through GET /podcasts/episodes/{episode_id} while maintaining the relationship to the original job status.

Example API Workflows

Submit a generation job:

POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

Immediate response:

{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

Poll for status:

GET /podcasts/jobs/command:001abcdef

Response during processing:

{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

Retry failed episodes:

POST /podcasts/episodes/{episode_id}/retry

This endpoint removes broken records and partial audio files before resubmitting with identical parameters.

Summary

• SurrealDB-backed queue: Uses surreal-commands library to manage job state and background processing through SurrealDB command records
• Immediate response: HTTP requests return job IDs instantly while heavy processing occurs asynchronously in separate worker processes
• Three-layer architecture: FastAPI routers (api/routers/podcasts.py), service layer (api/podcast_service.py), and command workers (commands/podcast_commands.py)
• Status polling: Clients track progress via GET /podcasts/jobs/{job_id} with states including pending, running, completed, and failed
• Result linking: Generated episodes persist as PodcastEpisode records tied to their original command IDs for complete audit trails

Frequently Asked Questions

How does Open Notebook handle long-running podcast generation without blocking HTTP requests?

The architecture delegates heavy processing to background workers through SurrealDB's command queue. When POST /podcasts/generate receives a request, PodcastService.submit_generation_job immediately returns a job ID after creating a command record. The actual LLM inference and audio synthesis run in separate processes managed by the surreal-commands worker pool, keeping the API responsive and preventing timeouts on long-running generation tasks.

What states can a podcast generation job have?

Jobs progress through discrete states managed by surreal-commands: pending when queued, running during execution, completed upon success, or failed if errors occur. The GET /podcasts/jobs/{job_id} endpoint exposes these states along with progress percentages and timestamps through PodcastService.get_job_status, allowing clients to implement polling mechanisms or webhook-based status updates.

Where is the actual podcast generation logic implemented?

The core generation logic resides in commands/podcast_commands.py (lines 69-85) within the generate_podcast_command function. This function, decorated with @command("generate_podcast", app="open_notebook"), loads episode configurations, creates output directories, and invokes the external podcast-creator library. It operates as a SurrealDB command worker, receiving PodcastGenerationInput models and returning results that get stored as PodcastEpisode records linked to the command ID.

How can I retry a failed podcast generation?

Use the POST /podcasts/episodes/{episode_id}/retry endpoint. This operation deletes the failed episode record from SurrealDB, removes any partial audio files from storage, and submits a new generation job using the original episode and speaker profiles. The new job receives a fresh job ID while maintaining the same content parameters, allowing you to regenerate podcasts after fixing underlying issues like LLM configuration errors or TTS service interruptions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}