# How to Implement Async Podcast Generation with a Job Queue in Open Notebook

> Learn how to implement async podcast generation with a job queue in Open Notebook. This approach uses SurrealDB and surreal-commands for efficient background processing of text-to-speech tasks.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-07-02

---

**Open Notebook implements async podcast generation by queueing jobs in SurrealDB via the surreal-commands library, allowing HTTP endpoints to return immediately while background workers handle compute-intensive text-to-speech processing.**

The open-source [lfnovo/open-notebook](https://github.com/lfnovo/open-notebook) repository provides a production-ready pattern for asynchronous job queues using SurrealDB as the persistence layer. This architecture decouples REST API requests from long-running podcast generation tasks, enabling clients to submit content and poll for completion without blocking server threads.

## Architecture Overview: SurrealDB and Command Workers

The system leverages the **surreal-commands** library to manage distributed task execution. Rather than processing audio generation inline, the application registers **command records** in SurrealDB that represent pending work units. Background worker processes poll the database for available commands, execute the generation logic asynchronously, and atomically update status fields from `pending` to `running`, `completed`, or `failed`.

## Submitting Async Podcast Generation Jobs

When a client requests podcast creation, the `POST /podcasts/generate` endpoint invokes `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py). This service method validates the `EpisodeProfile` and `SpeakerProfile` configurations, resolves source content from either a notebook ID or direct text input, and registers the task via the `submit_command` function from surreal-commands.

The method returns instantly with a SurrealDB record identifier—formatted as `command:{uuid}`—which serves as the **job ID** for subsequent status polling.

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

**Immediate Response:**

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

## Processing Queued Jobs with Command Workers

The actual audio synthesis executes outside the HTTP request cycle through the command worker pattern. In [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py), the `generate_podcast` function—decorated with `@command("generate_podcast", app="open_notebook")`—receives a `PodcastGenerationInput` Pydantic model when the job is dequeued.

This command handler performs the following steps:

1. Loads episode and speaker profile definitions from [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)
2. Resolves language model configurations for the generation pipeline
3. Creates a UUID-based output directory for file storage
4. Invokes the **podcast-creator** third-party library to synthesize audio, transcripts, and episode outlines
5. Creates a `PodcastEpisode` record linked to the original command ID via `ensure_record_id`

## Monitoring Job Status and Results

Clients track progress through the `GET /podcasts/jobs/{job_id}` endpoint defined in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). The `PodcastService.get_job_status` method queries SurrealDB using `get_command_status` to retrieve the current state alongside metadata timestamps.

```http
GET /podcasts/jobs/command:001abcdef

```

**Status Response Examples:**

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

When `status` transitions to `completed`, the `result` field contains the generated `episode_id`. Fetch the final episode data using `GET /podcasts/episodes/{episode_id}`, which returns the audio file path, full transcript, and outline content.

## Handling Failures and Retry Logic

Failed generation attempts can be requeued without manual data cleanup. The `POST /podcasts/episodes/{episode_id}/retry` endpoint removes the corrupted episode record from SurrealDB, deletes any partial audio files from storage, and submits a fresh job using the original `EpisodeProfile` and `SpeakerProfile` configurations. This returns a new `job_id` while preserving the user's initial setup parameters.

## Summary

- **Job Submission**: Use `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) to queue tasks via `submit_command`, returning a SurrealDB command ID immediately
- **Worker Execution**: The `@command("generate_podcast")` decorator in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) handles background processing through the podcast-creator library
- **Status Polling**: Query `GET /podcasts/jobs/{job_id}` to retrieve real-time status, progress percentages, and final episode IDs
- **Data Models**: Episode and speaker configurations use Pydantic models (`EpisodeProfile`, `SpeakerProfile`) stored in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)
- **Failure Recovery**: The retry endpoint automates cleanup and resubmission for failed generations

## Frequently Asked Questions

### How does Open Notebook queue podcast generation jobs asynchronously?

The system uses the **surreal-commands** library to persist job definitions as records in SurrealDB. The `submit_command` function creates a database entry with status `pending`, allowing the HTTP endpoint to return a `job_id` immediately while separate worker processes poll for and execute tasks.

### What file contains the actual podcast generation logic?

The command implementation resides in **[`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)**, specifically the function decorated with `@command("generate_podcast", app="open_notebook")`. This handler orchestrates the podcast-creator library, manages file I/O to UUID-based directories, and persists results via `PodcastEpisode` records.

### How do I check if a podcast generation job is complete?

Query the `GET /podcasts/jobs/{job_id}` endpoint, which invokes `PodcastService.get_job_status` using `get_command_status` from surreal-commands. The endpoint returns the SurrealDB command status—`pending`, `running`, `completed`, or `failed`—along with a `progress` float and the final `episode_id` in the result field when finished.

### Can I retry a failed podcast generation without reconfiguring profiles?

Yes. Submit a request to `POST /podcasts/episodes/{episode_id}/retry`. This endpoint automatically deletes the failed episode record, removes partial audio files, and resubmits the job using the original profile configurations, returning a new `job_id` for tracking.