# How Open Notebook's Async Job Queue Processes Podcast Generation Background Tasks

> Discover how Open Notebook's async job queue handles background tasks like podcast generation using SurrealDB. Achieve immediate HTTP responses while LLM and TTS processes run efficiently.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-07

---

**Open Notebook leverages SurrealDB and the surreal-commands library to execute podcast generation asynchronously, returning immediate HTTP responses while heavy LLM inference and text-to-speech operations process in the background.**

Open Notebook is an open-source knowledge management system that automates content creation through AI-driven workflows. When users request podcast generation via the REST API, the platform delegates resource-intensive processing to a robust **async job queue** built on SurrealDB, ensuring immediate API feedback while background workers handle the computational load.

## Submitting Jobs to the Async Queue

When a client initiates podcast creation, the system immediately creates a job record and returns a tracking identifier, allowing the lengthy generation process to proceed without blocking the HTTP connection.

### The REST API Endpoint

Clients submit generation requests to `POST /podcasts/generate`, defined in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). This endpoint accepts JSON payloads specifying the **episode profile**, **speaker profile**, source content (via `notebook_id` or direct text), and episode metadata.

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

### Service Layer and Command Submission

The route delegates to `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 45–99). This method performs three critical operations:

1. Validates the episode and speaker profiles against Pydantic models
2. Resolves source content from the referenced notebook or direct input
3. Registers the job via `submit_command` from the surreal-commands library

The `submit_command` function creates a persistent command record in SurrealDB and returns a unique **job ID** (formatted as `command:{uuid}`), which the API returns immediately to the client:

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

## Executing Background Tasks

Once queued, the async job queue processes the task independently of the web server using SurrealDB's background worker infrastructure.

### Command Registration

The actual work is performed by the function decorated with `@command("generate_podcast", app="open_notebook")` in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69–85). When the worker dequeues the job, it invokes this function with a `PodcastGenerationInput` model containing all necessary parameters.

### Audio Generation Pipeline

The command handler performs the following operations:

- Loads episode and speaker profiles from SurrealDB
- Resolves language model configurations
- Creates a UUID-based output directory for file storage
- Invokes the third-party **podcast-creator** library to synthesize audio, transcript, and outline

All heavy I/O and CPU-intensive operations occur within this background process, preventing API server blocking.

## Monitoring Job Status

Clients track progress through the job lifecycle using the status endpoint.

### Polling the Queue State

The endpoint `GET /podcasts/jobs/{job_id}` queries `PodcastService.get_job_status` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 15–33), which calls `get_command_status` from surreal-commands. This retrieves the current state from SurrealDB, which tracks statuses including `pending`, `running`, `completed`, and `failed`.

```http
GET /podcasts/jobs/command:001abcdef

```

Typical response during processing:

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

When `status` becomes `completed`, the `result` field contains the generated `episode_id` for retrieval via `GET /podcasts/episodes/{episode_id}`.

## Persisting Generation Results

After successful audio synthesis, the command persists metadata for future retrieval.

The background worker creates a `PodcastEpisode` record in SurrealDB containing the audio file path, generated transcript, and content outline. This record links to the original command ID via `ensure_record_id`, enabling correlation between job history and final output. If generation fails, the system retains error messages in the command record, and users may retry via `POST /podcasts/episodes/{episode_id}/retry`, which clears partial artifacts and resubmits the job to the async queue.

## Summary

- **Immediate response**: The `POST /podcasts/generate` endpoint returns a `job_id` instantly via `PodcastService.submit_generation_job`, delegating work to the SurrealDB-backed queue.
- **Background processing**: The `@command` decorator in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) registers the `generate_podcast` handler, which executes audio synthesis using the podcast-creator library.
- **Status tracking**: Clients poll `GET /podcasts/jobs/{job_id}` to monitor `pending`, `running`, or `completed` states via `get_command_status`.
- **Result storage**: Completed jobs create `PodcastEpisode` records linked to their command IDs, storing file paths and transcripts in SurrealDB.

## Frequently Asked Questions

### What database powers the async job queue in Open Notebook?

The queue utilizes **SurrealDB** as both the persistence layer and job broker, managed through the surreal-commands library. This integration stores command definitions, tracks job states, and manages background workers that process tasks outside the HTTP request lifecycle.

### How can I check if a podcast generation job has completed?

Poll the `GET /podcasts/jobs/{job_id}` endpoint, which returns the current status and progress percentage. When the status field changes to `completed`, the response includes the `episode_id` in the result field, indicating the audio file and transcript are ready for retrieval.

### What happens if a podcast generation job fails?

The command status updates to `failed` with error details preserved in the SurrealDB record. Users can invoke `POST /podcasts/episodes/{episode_id}/retry`, which deletes the broken episode record, removes partial audio files, and submits a fresh job to the async queue using the original parameters.

### Which external library handles the actual audio synthesis?

The **podcast-creator** library performs the text-to-speech conversion, audio assembly, and transcript generation. The `generate_podcast` command in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) invokes this library after resolving all language model configurations and creating the output directory structure.