# How the Asynchronous Job Queue Works for Podcast Generation in Open Notebook

> Discover how Open Notebook's async job queue efficiently handles podcast generation using surreal-commands and SurrealDB. Learn how background workers manage LLM inference and TTS for seamless API responses.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-12

---

**Open Notebook leverages the `surreal-commands` library built on SurrealDB to process podcast generation asynchronously, allowing the API to return immediate job IDs while background workers handle the LLM inference, text-to-speech synthesis, and file I/O.**

Open Notebook implements a robust asynchronous job queue for podcast generation using SurrealDB as the persistent command store. This architecture separates the HTTP request lifecycle from resource-intensive audio synthesis tasks, ensuring responsive API endpoints while reliably managing long-running operations in the background.

## Job Submission and Command Registration

### REST Endpoint Initialization

When a client sends a request to `POST /podcasts/generate`, the FastAPI router delegates to `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 45-99). This method validates the **episode profile**, **speaker profiles**, and resolves the source content from either a notebook or direct text input before queuing the work.

### SurrealDB Command Creation

The service layer registers the job by calling `submit_command` from the `surreal-commands` library, which creates a persistent command record in SurrealDB. This record receives a unique identifier returned as the **job ID** to the client, enabling status polling throughout the lifecycle. The command remains in a `pending` state until a background worker picks it up for execution.

## Background Processing and Command Execution

### The generate_podcast Command Handler

The actual processing logic resides in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69-85), where the function is decorated with `@command("generate_podcast", app="open_notebook")`. When dequeued, the handler receives a `PodcastGenerationInput` model, loads the relevant language model configurations, creates a UUID-based output directory, and orchestrates the generation workflow.

### Integration with the podcast-creator Library

The command invokes the third-party **`podcast-creator`** library to synthesize audio, generate transcripts, and create outlines. This heavy lifting—including LLM API calls and text-to-speech operations—runs entirely within the background worker process managed by SurrealDB's command engine, isolated from the main application thread.

## Job Status Tracking and Result Persistence

### Polling the Command Status

Clients track progress via `GET /podcasts/jobs/{job_id}` (defined in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py), lines 71-79), which calls `PodcastService.get_job_status`. This method queries the SurrealDB command record using `get_command_status`, returning the current state (`pending`, `running`, `completed`, or `failed`), timestamps, and any progress indicators.

### Episode Creation and Record Linking

Upon successful completion, the command creates a `PodcastEpisode` record defined in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py), storing the audio file path, transcript, and outline while linking back to the original command ID via `ensure_record_id`. This linkage enables the episode list endpoint to retrieve jobs alongside their generation status.

## API Usage Examples

Submitting a podcast generation job:

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

Immediate response with job ID:

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

Polling for job status:

```http
GET /podcasts/jobs/command:001abcdef

```

Possible status response:

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

Retrying a failed episode:

```http
POST /podcasts/episodes/{episode_id}/retry

```

This endpoint deletes the broken episode record, removes any partial audio files, and re-submits a fresh job using the same profiles and content.

## Summary

- **SurrealDB persistence**: The `surreal-commands` library stores job states in SurrealDB, providing durable, queryable command records that survive application restarts.
- **Immediate response**: The `POST /podcasts/generate` endpoint returns instantly with a `job_id` while delegating heavy processing to background workers.
- **Decorated command handlers**: The `@command` decorator in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) registers Python functions as queue workers that SurrealDB automatically invokes when jobs are available.
- **Status polling**: Clients track progress via `GET /podcasts/jobs/{job_id}`, which queries the command record's current state and progress metrics.
- **Result linking**: Generated episodes maintain references to their originating command IDs, enabling comprehensive audit trails and retry functionality.

## Frequently Asked Questions

### How does Open Notebook handle failures in the podcast generation queue?

When the `podcast-creator` library throws an exception or the process exits abnormally, the `surreal-commands` worker updates the SurrealDB command record status to `failed` and populates the `error_message` field. Clients polling `GET /podcasts/jobs/{job_id}` receive this error state and can trigger a retry via `POST /podcasts/episodes/{episode_id}/retry`, which removes the failed artifacts and submits a fresh command.

### What database does the async job queue use for persistence?

The queue uses **SurrealDB** as its persistent backing store. The `surreal-commands` library manages the schema for command records, handling state transitions from `pending` to `running` to `completed` or `failed` while storing timestamps, progress indicators, and serialized results directly in the database.

### Can clients retry failed podcast generation jobs?

Yes. The API exposes `POST /podcasts/episodes/{episode_id}/retry`, which deletes the broken `PodcastEpisode` record, removes any partial audio files from storage, and re-invokes `PodcastService.submit_generation_job` with the original episode and speaker profiles. This creates a new command record with a fresh `job_id` while preserving the original configuration.

### How is the job ID generated when submitting a podcast creation request?

The `job_id` is generated by SurrealDB when the `submit_command` function creates the command record. Typically formatted as `command:<random_string>`, this ID is returned immediately to the client by `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 45-99) and serves as the persistent identifier for polling status and debugging execution history.