How the Asynchronous Job Queue Works for Podcast Generation in Open Notebook

Open Notebook leverages the surreal-commands library built on SurrealDB to process podcast generation asynchronously, allowing the API to return immediate job IDs while background workers handle the LLM inference, text-to-speech synthesis, and file I/O.

Open Notebook implements a robust asynchronous job queue for podcast generation using SurrealDB as the persistent command store. This architecture separates the HTTP request lifecycle from resource-intensive audio synthesis tasks, ensuring responsive API endpoints while reliably managing long-running operations in the background.

Job Submission and Command Registration

REST Endpoint Initialization

When a client sends a request to POST /podcasts/generate, the FastAPI router delegates to PodcastService.submit_generation_job in api/podcast_service.py (lines 45-99). This method validates the episode profile, speaker profiles, and resolves the source content from either a notebook or direct text input before queuing the work.

SurrealDB Command Creation

The service layer registers the job by calling submit_command from the surreal-commands library, which creates a persistent command record in SurrealDB. This record receives a unique identifier returned as the job ID to the client, enabling status polling throughout the lifecycle. The command remains in a pending state until a background worker picks it up for execution.

Background Processing and Command Execution

The generate_podcast Command Handler

The actual processing logic resides in commands/podcast_commands.py (lines 69-85), where the function is decorated with @command("generate_podcast", app="open_notebook"). When dequeued, the handler receives a PodcastGenerationInput model, loads the relevant language model configurations, creates a UUID-based output directory, and orchestrates the generation workflow.

Integration with the podcast-creator Library

The command invokes the third-party podcast-creator library to synthesize audio, generate transcripts, and create outlines. This heavy lifting—including LLM API calls and text-to-speech operations—runs entirely within the background worker process managed by SurrealDB's command engine, isolated from the main application thread.

Job Status Tracking and Result Persistence

Polling the Command Status

Clients track progress via GET /podcasts/jobs/{job_id} (defined in api/routers/podcasts.py, lines 71-79), which calls PodcastService.get_job_status. This method queries the SurrealDB command record using get_command_status, returning the current state (pending, running, completed, or failed), timestamps, and any progress indicators.

Episode Creation and Record Linking

Upon successful completion, the command creates a PodcastEpisode record defined in open_notebook/podcasts/models.py, storing the audio file path, transcript, and outline while linking back to the original command ID via ensure_record_id. This linkage enables the episode list endpoint to retrieve jobs alongside their generation status.

API Usage Examples

Submitting a podcast generation job:

POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

Immediate response with job ID:

{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

Polling for job status:

GET /podcasts/jobs/command:001abcdef

Possible status response:

{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

Retrying a failed episode:

POST /podcasts/episodes/{episode_id}/retry

This endpoint deletes the broken episode record, removes any partial audio files, and re-submits a fresh job using the same profiles and content.

Summary

• SurrealDB persistence: The surreal-commands library stores job states in SurrealDB, providing durable, queryable command records that survive application restarts.
• Immediate response: The POST /podcasts/generate endpoint returns instantly with a job_id while delegating heavy processing to background workers.
• Decorated command handlers: The @command decorator in commands/podcast_commands.py registers Python functions as queue workers that SurrealDB automatically invokes when jobs are available.
• Status polling: Clients track progress via GET /podcasts/jobs/{job_id}, which queries the command record's current state and progress metrics.
• Result linking: Generated episodes maintain references to their originating command IDs, enabling comprehensive audit trails and retry functionality.

Frequently Asked Questions

How does Open Notebook handle failures in the podcast generation queue?

When the podcast-creator library throws an exception or the process exits abnormally, the surreal-commands worker updates the SurrealDB command record status to failed and populates the error_message field. Clients polling GET /podcasts/jobs/{job_id} receive this error state and can trigger a retry via POST /podcasts/episodes/{episode_id}/retry, which removes the failed artifacts and submits a fresh command.

What database does the async job queue use for persistence?

The queue uses SurrealDB as its persistent backing store. The surreal-commands library manages the schema for command records, handling state transitions from pending to running to completed or failed while storing timestamps, progress indicators, and serialized results directly in the database.

Can clients retry failed podcast generation jobs?

Yes. The API exposes POST /podcasts/episodes/{episode_id}/retry, which deletes the broken PodcastEpisode record, removes any partial audio files from storage, and re-invokes PodcastService.submit_generation_job with the original episode and speaker profiles. This creates a new command record with a fresh job_id while preserving the original configuration.

How is the job ID generated when submitting a podcast creation request?

The job_id is generated by SurrealDB when the submit_command function creates the command record. Typically formatted as command:<random_string>, this ID is returned immediately to the client by PodcastService.submit_generation_job in api/podcast_service.py (lines 45-99) and serves as the persistent identifier for polling status and debugging execution history.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}