How the Async Job Queue Handles Podcast Generation and Status Polling in Open Notebook

Open Notebook leverages SurrealDB and the surreal-commands library to process podcast generation asynchronously, allowing clients to submit jobs via REST API and poll for status updates while heavy LLM and TTS operations run in the background.

The lfnovo/open-notebook repository implements a robust asynchronous architecture for generating AI podcasts using SurrealDB as both the database and job queue backend. By delegating expensive text-to-speech and language model operations to background workers, the system maintains responsive HTTP endpoints while handling complex multi-step audio synthesis workflows.

Submitting a Podcast Generation Job

Podcast generation begins at the REST endpoint POST /podcasts/generate, defined in [api/routers/podcasts.py](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). When a client submits a request, the service layer invokes PodcastService.submit_generation_job in [api/podcast_service.py](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py#L45-L99).

This method performs several validation steps before queueing:

1. Validates the requested episode profile and speaker profiles against stored configurations
2. Resolves source content from either a referenced notebook or directly supplied text
3. Registers the job via submit_command from the surreal-commands library, which creates a persistent command record in SurrealDB

The SurrealDB record ID returned by submit_command becomes the job ID (e.g., command:001abcdef) that clients use for subsequent status polling. The HTTP response returns immediately with this ID and a status: "submitted" field, without waiting for the actual audio generation to begin.

POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

Background Command Execution

Once queued, SurrealDB's command engine delegates the work to the function decorated with @command("generate_podcast", app="open_notebook") in [commands/podcast_commands.py](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py#L69-L85). This decorator registers the handler with the surreal-commands worker pool.

The command execution flow involves:

• Receiving a PodcastGenerationInput model containing resolved episode parameters
• Loading episode profiles and speaker configurations from SurrealDB
• Resolving language model configurations for the synthesis pipeline
• Creating a UUID-based output directory for audio file storage
• Invoking the third-party podcast-creator library to synthesize audio, generate transcripts, and create content outlines

During execution, the command periodically updates its status in SurrealDB through the surreal-commands framework, transitioning from pending to running and eventually to either completed or failed.

Polling Job Status and Retrieval

Clients monitor progress through the GET /podcasts/jobs/{job_id} endpoint implemented in [api/routers/podcasts.py](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py#L71-L79). This route calls PodcastService.get_job_status, which utilizes the get_command_status helper from surreal-commands to query the current state from SurrealDB.

The endpoint returns a JSON payload containing:

• Current state: One of pending, running, completed, or failed
• Progress indicators: Optional completion percentage (e.g., 0.45 for 45%)
• Result data: When status is completed, includes the generated episode_id
• Timestamps: created and updated fields tracking the job lifecycle
• Error details: error_message field when status is failed

GET /podcasts/jobs/command:001abcdef

{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

When the status becomes completed, the result field contains the episode_id, which can then be fetched via GET /podcasts/episodes/{episode_id} to retrieve the full episode details including audio file paths and transcripts.

Persisting Results and Retry Logic

Upon successful completion, the generate_podcast command creates a PodcastEpisode record in [open_notebook/podcasts/models.py](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py), storing the audio file path, transcript, outline, and linking the episode to the original command ID via ensure_record_id. This linkage enables the system to correlate episodes with their generation jobs for status tracking.

If a job fails or produces unsatisfactory results, the system provides a retry mechanism through POST /podcasts/episodes/{episode_id}/retry. This endpoint:

1. Deletes the broken episode record from SurrealDB
2. Removes any partial audio files from storage
3. Re-submits a fresh generation job using the original episode and speaker profiles

This ensures that transient failures in the podcast-creator library or underlying TTS services do not permanently block content creation.

Summary

• Job submission occurs via POST /podcasts/generate, which uses PodcastService.submit_generation_job and surreal-commands.submit_command to create a SurrealDB command record without blocking the HTTP response.
• Background processing happens through the @command("generate_podcast") decorator in commands/podcast_commands.py, which invokes the podcast-creator library to synthesize audio and transcripts.
• Status polling uses GET /podcasts/jobs/{job_id} and get_command_status to retrieve real-time state updates including progress percentages and completion status.
• Result persistence links generated PodcastEpisode records to their original command IDs, enabling correlation between jobs and final audio outputs.
• Retry functionality allows clients to re-queue failed episodes by deleting partial results and re-submitting with identical parameters.

Frequently Asked Questions

How does the async job queue maintain state between the REST API and background workers?

The system uses SurrealDB as the central state store through the surreal-commands library. When submit_command creates a job record in SurrealDB, both the FastAPI endpoints and background workers read from and write to this same database record. The command status (pending, running, completed, failed) persists in SurrealDB, allowing the get_command_status function to retrieve current state regardless of which worker process is handling the job.

What happens if the podcast generation process crashes or the server restarts?

Because SurrealDB persists the command records and the surreal-commands library implements durable job queuing, incomplete jobs survive server restarts. When the worker pool restarts, it automatically picks up any jobs with pending or running status that were interrupted. The generate_podcast command in commands/podcast_commands.py operates idempotently where possible, using UUID-based output directories to prevent file collisions during retries.

Can clients receive real-time updates without polling the job status endpoint?

The current implementation described in the source code relies on polling via GET /podcasts/jobs/{job_id}, with clients responsible for implementing appropriate polling intervals. The SurrealDB-based queue architecture in surreal-commands supports the infrastructure for future WebSocket or webhook implementations, but the provided code shows a traditional request-response polling pattern for status updates.

How are speaker profiles and episode configurations validated before queuing?

The PodcastService.submit_generation_job method validates inputs before calling submit_command. It loads and verifies the EpisodeProfile and SpeakerProfile models from [open_notebook/podcasts/models.py](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) against SurrealDB records, ensuring that referenced profiles exist and that the source content (notebook or direct text) can be resolved. This prevents invalid jobs from entering the queue and consuming worker resources.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}