How Open Notebook Implements an Async Job Queue for Podcast Generation

Open Notebook leverages the surreal-commands library on top of SurrealDB to queue, execute, and track podcast generation work asynchronously, returning an immediate job ID while a background worker handles LLM inference, TTS synthesis, and file I/O.

The lfnovo/open-notebook project offloads long-running podcast creation tasks to an asynchronous job queue so that REST API calls remain fast and non-blocking. In this implementation, the queue is not a custom message broker but rather SurrealDB's command engine, orchestrated through the surreal-commands Python library. This article breaks down exactly how the async job queue for podcast generation is structured, from initial job submission in api/podcast_service.py to background execution in commands/podcast_commands.py.

Architecture Overview

Open Notebook's podcast pipeline separates HTTP request handling from heavy compute by storing commands as SurrealDB records and processing them in separate workers. The flow follows four distinct stages: job submission via PodcastService.submit_generation_job, command registration with submit_command, background execution via the @command("generate_podcast") decorated function, and status retrieval through PodcastService.get_job_status.

Submitting a Generation Job

The REST Endpoint and Service Layer

When a client calls POST /podcasts/generate, the request is handled by the FastAPI router in api/routers/podcasts.py, which delegates to PodcastService.submit_generation_job in api/podcast_service.py (lines 45‑99). This service method validates the selected episode profile and speaker profile, resolves the source content from either a notebook or directly supplied text, and prepares a PodcastGenerationInput model.

Command Registration with SurrealDB

Once validation is complete, the service calls submit_command from the surreal-commands library. This writes a new command record directly into SurrealDB, which immediately yields a unique record ID that serves as the job ID returned to the client. Because the record creation is synchronous but the execution is deferred, the REST response returns instantly while the actual work queues in the background.

POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

The immediate JSON response includes the SurrealDB command ID:

{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

Processing Jobs in the Background

The generate_podcast Command

The worker responsible for dequeuing and running jobs is provided by surreal-commands. It invokes the function decorated with @command("generate_podcast", app="open_notebook") located in commands/podcast_commands.py (lines 69‑85). When triggered, this command receives the PodcastGenerationInput payload, loads the referenced episode and speaker profiles, and resolves all language-model configurations required for the script.

Audio Synthesis and Result Persistence

After setup, the command creates a UUID-based output directory and delegates audio generation, transcript creation, and outline synthesis to the third-party podcast-creator library. Upon successful completion, the command creates a PodcastEpisode record, persists the audio file path, transcript, and outline, and links the episode back to the originating command ID using ensure_record_id. This linkage allows the system to correlate finished episodes with their original job status.

Tracking and Polling Job Status

Querying Command Status

Clients poll for updates using the job ID received at submission. The PodcastService.get_job_status helper in api/podcast_service.py (lines 15‑33) wraps get_command_status from surreal-commands, which reads the current command state directly from SurrealDB. The underlying table stores statuses such as pending, running, completed, and failed, along with timestamps and optional progress metadata.

The Status Endpoint

The FastAPI route GET /podcasts/jobs/{job_id} defined in api/routers/podcasts.py (lines 71‑79) exposes this status to clients. It returns a JSON payload containing the current state, result data when finished, error messages on failure, and creation and update timestamps.

GET /podcasts/jobs/command:001abcdef

A mid-generation response might look like this:

{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

When the status becomes completed, the result field contains the generated episode_id, which can then be fetched with GET /podcasts/episodes/{episode_id}.

Retry Logic for Failed Episodes

If a generation job fails or produces a broken episode, Open Notebook provides a recovery path through POST /podcasts/episodes/{episode_id}/retry. This endpoint deletes the failed episode record, removes any partial audio files from storage, and re-submits a fresh generation job using the original episode profile, speaker profile, and source content. The new job receives its own SurrealDB command ID and follows the same async lifecycle.

Key Files in the Podcast Queue

The following modules implement the async podcast generation queue in open-notebook:

• api/podcast_service.py — Service layer that submits jobs via submit_command and queries status via get_command_status.
• commands/podcast_commands.py — Defines the generate_podcast command that runs the actual podcast creation pipeline.
• api/routers/podcasts.py — FastAPI routes that expose job submission, status polling, episode listing, and retry operations.
• open_notebook/podcasts/models.py — Pydantic models including EpisodeProfile, SpeakerProfile, and PodcastEpisode that are persisted in SurrealDB.

Summary

• Open Notebook uses the surreal-commands library on top of SurrealDB to manage its async job queue for podcast generation, avoiding the need for a separate broker like Redis or RabbitMQ.
• Job submission occurs in PodcastService.submit_generation_job (api/podcast_service.py), which registers a command record and returns a unique job ID instantly.
• Background workers execute the @command("generate_podcast") function in commands/podcast_commands.py, which orchestrates the podcast-creator library and persists a PodcastEpisode record on success.
• Clients poll GET /podcasts/jobs/{job_id} to track progress through SurrealDB statuses (pending, running, completed, failed).
• A dedicated retry endpoint regenerates failed episodes by cleaning up partial state and queuing a new command.

Frequently Asked Questions

What library does Open Notebook use for the async podcast generation queue?

Open Notebook relies on the surreal-commands library, which provides an asynchronous job queue built directly on SurrealDB. This library handles command registration, worker dispatch, and status tracking so the project does not need to run a separate message broker.

How do I check the status of a podcast generation job?

Send a GET /podcasts/jobs/{job_id} request to the REST API. The endpoint delegates to PodcastService.get_job_status in api/podcast_service.py, which calls get_command_status from surreal-commands to return the current state, timestamps, and any result or error data stored in SurrealDB.

What happens when a podcast generation job fails?

If a job fails, the command record in SurrealDB retains a failed status and any error details. You can retry the episode by calling POST /podcasts/episodes/{episode_id}/retry, which deletes the broken record, removes partial audio files, and submits a fresh generation job with the same profiles and content.

Which file contains the actual podcast generation logic?

The core logic lives in commands/podcast_commands.py (lines 69‑85), inside the function decorated with @command("generate_podcast", app="open_notebook"). This function receives the generation input, resolves configurations, and invokes the third-party podcast-creator library to synthesize the final audio, transcript, and outline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}