How to Implement Async Podcast Generation with a Job Queue in Open Notebook

Open Notebook implements async podcast generation by queueing jobs in SurrealDB via the surreal-commands library, allowing HTTP endpoints to return immediately while background workers handle compute-intensive text-to-speech processing.

The open-source lfnovo/open-notebook repository provides a production-ready pattern for asynchronous job queues using SurrealDB as the persistence layer. This architecture decouples REST API requests from long-running podcast generation tasks, enabling clients to submit content and poll for completion without blocking server threads.

Architecture Overview: SurrealDB and Command Workers

The system leverages the surreal-commands library to manage distributed task execution. Rather than processing audio generation inline, the application registers command records in SurrealDB that represent pending work units. Background worker processes poll the database for available commands, execute the generation logic asynchronously, and atomically update status fields from pending to running, completed, or failed.

Submitting Async Podcast Generation Jobs

When a client requests podcast creation, the POST /podcasts/generate endpoint invokes PodcastService.submit_generation_job in api/podcast_service.py. This service method validates the EpisodeProfile and SpeakerProfile configurations, resolves source content from either a notebook ID or direct text input, and registers the task via the submit_command function from surreal-commands.

The method returns instantly with a SurrealDB record identifier—formatted as command:{uuid}—which serves as the job ID for subsequent status polling.

POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

Immediate Response:

{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

Processing Queued Jobs with Command Workers

The actual audio synthesis executes outside the HTTP request cycle through the command worker pattern. In commands/podcast_commands.py, the generate_podcast function—decorated with @command("generate_podcast", app="open_notebook")—receives a PodcastGenerationInput Pydantic model when the job is dequeued.

This command handler performs the following steps:

1. Loads episode and speaker profile definitions from open_notebook/podcasts/models.py
2. Resolves language model configurations for the generation pipeline
3. Creates a UUID-based output directory for file storage
4. Invokes the podcast-creator third-party library to synthesize audio, transcripts, and episode outlines
5. Creates a PodcastEpisode record linked to the original command ID via ensure_record_id

Monitoring Job Status and Results

Clients track progress through the GET /podcasts/jobs/{job_id} endpoint defined in api/routers/podcasts.py. The PodcastService.get_job_status method queries SurrealDB using get_command_status to retrieve the current state alongside metadata timestamps.

GET /podcasts/jobs/command:001abcdef

Status Response Examples:

{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

When status transitions to completed, the result field contains the generated episode_id. Fetch the final episode data using GET /podcasts/episodes/{episode_id}, which returns the audio file path, full transcript, and outline content.

Handling Failures and Retry Logic

Failed generation attempts can be requeued without manual data cleanup. The POST /podcasts/episodes/{episode_id}/retry endpoint removes the corrupted episode record from SurrealDB, deletes any partial audio files from storage, and submits a fresh job using the original EpisodeProfile and SpeakerProfile configurations. This returns a new job_id while preserving the user's initial setup parameters.

Summary

• Job Submission: Use PodcastService.submit_generation_job in api/podcast_service.py to queue tasks via submit_command, returning a SurrealDB command ID immediately
• Worker Execution: The @command("generate_podcast") decorator in commands/podcast_commands.py handles background processing through the podcast-creator library
• Status Polling: Query GET /podcasts/jobs/{job_id} to retrieve real-time status, progress percentages, and final episode IDs
• Data Models: Episode and speaker configurations use Pydantic models (EpisodeProfile, SpeakerProfile) stored in open_notebook/podcasts/models.py
• Failure Recovery: The retry endpoint automates cleanup and resubmission for failed generations

Frequently Asked Questions

How does Open Notebook queue podcast generation jobs asynchronously?

The system uses the surreal-commands library to persist job definitions as records in SurrealDB. The submit_command function creates a database entry with status pending, allowing the HTTP endpoint to return a job_id immediately while separate worker processes poll for and execute tasks.

What file contains the actual podcast generation logic?

The command implementation resides in commands/podcast_commands.py, specifically the function decorated with @command("generate_podcast", app="open_notebook"). This handler orchestrates the podcast-creator library, manages file I/O to UUID-based directories, and persists results via PodcastEpisode records.

How do I check if a podcast generation job is complete?

Query the GET /podcasts/jobs/{job_id} endpoint, which invokes PodcastService.get_job_status using get_command_status from surreal-commands. The endpoint returns the SurrealDB command status—pending, running, completed, or failed—along with a progress float and the final episode_id in the result field when finished.

Can I retry a failed podcast generation without reconfiguring profiles?

Yes. Submit a request to POST /podcasts/episodes/{episode_id}/retry. This endpoint automatically deletes the failed episode record, removes partial audio files, and resubmits the job using the original profile configurations, returning a new job_id for tracking.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}