# How Open Notebook Implements an Async Job Queue for Podcast Generation

> Discover how Open Notebook uses surreal-commands and SurrealDB for an async job queue, enabling efficient background podcast generation with immediate job IDs.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-05

---

**Open Notebook leverages the `surreal-commands` library on top of SurrealDB to queue, execute, and track podcast generation work asynchronously, returning an immediate job ID while a background worker handles LLM inference, TTS synthesis, and file I/O.**

The `lfnovo/open-notebook` project offloads long-running podcast creation tasks to an asynchronous job queue so that REST API calls remain fast and non-blocking. In this implementation, the queue is not a custom message broker but rather SurrealDB's command engine, orchestrated through the `surreal-commands` Python library. This article breaks down exactly how the async job queue for podcast generation is structured, from initial job submission in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) to background execution in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py).

## Architecture Overview

Open Notebook's podcast pipeline separates HTTP request handling from heavy compute by storing commands as SurrealDB records and processing them in separate workers. The flow follows four distinct stages: job submission via `PodcastService.submit_generation_job`, command registration with `submit_command`, background execution via the `@command("generate_podcast")` decorated function, and status retrieval through `PodcastService.get_job_status`.

## Submitting a Generation Job

### The REST Endpoint and Service Layer

When a client calls `POST /podcasts/generate`, the request is handled by the FastAPI router in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py), which delegates to `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 45‑99). This service method validates the selected **episode profile** and **speaker profile**, resolves the source content from either a notebook or directly supplied text, and prepares a `PodcastGenerationInput` model.

### Command Registration with SurrealDB

Once validation is complete, the service calls `submit_command` from the `surreal-commands` library. This writes a new command record directly into SurrealDB, which immediately yields a unique record ID that serves as the **job ID** returned to the client. Because the record creation is synchronous but the execution is deferred, the REST response returns instantly while the actual work queues in the background.

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

The immediate JSON response includes the SurrealDB command ID:

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

## Processing Jobs in the Background

### The generate_podcast Command

The worker responsible for dequeuing and running jobs is provided by `surreal-commands`. It invokes the function decorated with `@command("generate_podcast", app="open_notebook")` located in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69‑85). When triggered, this command receives the `PodcastGenerationInput` payload, loads the referenced episode and speaker profiles, and resolves all language-model configurations required for the script.

### Audio Synthesis and Result Persistence

After setup, the command creates a UUID-based output directory and delegates audio generation, transcript creation, and outline synthesis to the third-party **`podcast-creator`** library. Upon successful completion, the command creates a `PodcastEpisode` record, persists the audio file path, transcript, and outline, and links the episode back to the originating command ID using `ensure_record_id`. This linkage allows the system to correlate finished episodes with their original job status.

## Tracking and Polling Job Status

### Querying Command Status

Clients poll for updates using the job ID received at submission. The `PodcastService.get_job_status` helper in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 15‑33) wraps `get_command_status` from `surreal-commands`, which reads the current command state directly from SurrealDB. The underlying table stores statuses such as `pending`, `running`, `completed`, and `failed`, along with timestamps and optional progress metadata.

### The Status Endpoint

The FastAPI route `GET /podcasts/jobs/{job_id}` defined in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) (lines 71‑79) exposes this status to clients. It returns a JSON payload containing the current state, result data when finished, error messages on failure, and creation and update timestamps.

```http
GET /podcasts/jobs/command:001abcdef

```

A mid-generation response might look like this:

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

When the status becomes `completed`, the `result` field contains the generated `episode_id`, which can then be fetched with `GET /podcasts/episodes/{episode_id}`.

## Retry Logic for Failed Episodes

If a generation job fails or produces a broken episode, Open Notebook provides a recovery path through `POST /podcasts/episodes/{episode_id}/retry`. This endpoint deletes the failed episode record, removes any partial audio files from storage, and re-submits a fresh generation job using the original episode profile, speaker profile, and source content. The new job receives its own SurrealDB command ID and follows the same async lifecycle.

## Key Files in the Podcast Queue

The following modules implement the async podcast generation queue in `open-notebook`:

- **[`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)** — Service layer that submits jobs via `submit_command` and queries status via `get_command_status`.
- **[`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)** — Defines the `generate_podcast` command that runs the actual podcast creation pipeline.
- **[`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)** — FastAPI routes that expose job submission, status polling, episode listing, and retry operations.
- **[`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)** — Pydantic models including `EpisodeProfile`, `SpeakerProfile`, and `PodcastEpisode` that are persisted in SurrealDB.

## Summary

- Open Notebook uses the **`surreal-commands`** library on top of SurrealDB to manage its async job queue for podcast generation, avoiding the need for a separate broker like Redis or RabbitMQ.
- Job submission occurs in `PodcastService.submit_generation_job` ([`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)), which registers a command record and returns a unique job ID instantly.
- Background workers execute the `@command("generate_podcast")` function in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py), which orchestrates the `podcast-creator` library and persists a `PodcastEpisode` record on success.
- Clients poll `GET /podcasts/jobs/{job_id}` to track progress through SurrealDB statuses (`pending`, `running`, `completed`, `failed`).
- A dedicated retry endpoint regenerates failed episodes by cleaning up partial state and queuing a new command.

## Frequently Asked Questions

### What library does Open Notebook use for the async podcast generation queue?

Open Notebook relies on the **`surreal-commands`** library, which provides an asynchronous job queue built directly on SurrealDB. This library handles command registration, worker dispatch, and status tracking so the project does not need to run a separate message broker.

### How do I check the status of a podcast generation job?

Send a `GET /podcasts/jobs/{job_id}` request to the REST API. The endpoint delegates to `PodcastService.get_job_status` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), which calls `get_command_status` from `surreal-commands` to return the current state, timestamps, and any result or error data stored in SurrealDB.

### What happens when a podcast generation job fails?

If a job fails, the command record in SurrealDB retains a `failed` status and any error details. You can retry the episode by calling `POST /podcasts/episodes/{episode_id}/retry`, which deletes the broken record, removes partial audio files, and submits a fresh generation job with the same profiles and content.

### Which file contains the actual podcast generation logic?

The core logic lives in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69‑85), inside the function decorated with `@command("generate_podcast", app="open_notebook")`. This function receives the generation input, resolves configurations, and invokes the third-party `podcast-creator` library to synthesize the final audio, transcript, and outline.