# Async Job Queue Architecture for Podcast Generation in Open Notebook

> Discover the async job queue architecture powering podcast generation in Open Notebook. Understand how surreal-commands and SurrealDB handle LLM and TTS tasks efficiently.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-22

---

**Open Notebook leverages the surreal-commands library built on SurrealDB to execute podcast generation asynchronously, returning a job ID immediately while processing heavy LLM and TTS workloads in the background.**

The lfnovo/open-notebook repository implements a robust asynchronous job queue for podcast generation that separates HTTP request handling from resource-intensive audio synthesis tasks. This architecture uses SurrealDB as both the database and job queue backend through the surreal-commands library, enabling non-blocking podcast creation workflows that scale efficiently.

## Architecture Overview

The system implements a producer-consumer pattern where the FastAPI application submits jobs to SurrealDB and background workers execute them. The flow spans three main layers: the REST API in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py), the service layer in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), and the command definitions in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py).

## Job Submission Flow

### REST Endpoint and Service Layer

When clients call `POST /podcasts/generate`, the request enters `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 45-99). This method validates **episode profiles** and **speaker profiles**, resolves source content from notebooks or direct text input, and prepares the job parameters for queueing.

### Command Registration in SurrealDB

The service layer calls `submit_command` from the surreal-commands library to create a command record in SurrealDB. This returns a unique **job ID** (e.g., `command:001abcdef`) that clients use for polling. The command record initially stores status as `pending` along with serialized input data, allowing the HTTP response to return instantly while the heavy processing remains queued.

## Background Processing Worker

### Command Definition and Execution

The actual processing occurs in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69-85) via the function decorated with `@command("generate_podcast", app="open_notebook")`. When dequeued, the worker receives a `PodcastGenerationInput` model, loads relevant episode and speaker configurations, creates a UUID-based output directory, and resolves all language model settings before execution.

### Third-Party Integration

The command invokes the external **podcast-creator** library to synthesize audio, generate transcripts, and create outlines. This library handles heavy operations including LLM API calls and text-to-speech processing outside the main application thread, preventing blocking of the FastAPI event loop.

## Job Status Tracking and Polling

### Status Retrieval

Clients poll `GET /podcasts/jobs/{job_id}` which delegates to `PodcastService.get_job_status` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) (lines 15-33). This calls `get_command_status` from surreal-commands to retrieve the current state, which can be `pending`, `running`, `completed`, or `failed`, along with timestamps and progress indicators.

### Result Persistence

Upon completion, the command creates a `PodcastEpisode` record and links it to the original command ID using `ensure_record_id`. The record stores audio file paths, transcripts, and outlines, enabling retrieval through `GET /podcasts/episodes/{episode_id}` while maintaining the relationship to the original job status.

## Example API Workflows

Submit a generation job:

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

Immediate response:

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

Poll for status:

```http
GET /podcasts/jobs/command:001abcdef

```

Response during processing:

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

Retry failed episodes:

```http
POST /podcasts/episodes/{episode_id}/retry

```

This endpoint removes broken records and partial audio files before resubmitting with identical parameters.

## Summary

- **SurrealDB-backed queue**: Uses surreal-commands library to manage job state and background processing through SurrealDB command records
- **Immediate response**: HTTP requests return job IDs instantly while heavy processing occurs asynchronously in separate worker processes
- **Three-layer architecture**: FastAPI routers ([`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)), service layer ([`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)), and command workers ([`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py))
- **Status polling**: Clients track progress via `GET /podcasts/jobs/{job_id}` with states including `pending`, `running`, `completed`, and `failed`
- **Result linking**: Generated episodes persist as `PodcastEpisode` records tied to their original command IDs for complete audit trails

## Frequently Asked Questions

### How does Open Notebook handle long-running podcast generation without blocking HTTP requests?

The architecture delegates heavy processing to background workers through SurrealDB's command queue. When `POST /podcasts/generate` receives a request, `PodcastService.submit_generation_job` immediately returns a job ID after creating a command record. The actual LLM inference and audio synthesis run in separate processes managed by the surreal-commands worker pool, keeping the API responsive and preventing timeouts on long-running generation tasks.

### What states can a podcast generation job have?

Jobs progress through discrete states managed by surreal-commands: `pending` when queued, `running` during execution, `completed` upon success, or `failed` if errors occur. The `GET /podcasts/jobs/{job_id}` endpoint exposes these states along with progress percentages and timestamps through `PodcastService.get_job_status`, allowing clients to implement polling mechanisms or webhook-based status updates.

### Where is the actual podcast generation logic implemented?

The core generation logic resides in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69-85) within the `generate_podcast_command` function. This function, decorated with `@command("generate_podcast", app="open_notebook")`, loads episode configurations, creates output directories, and invokes the external `podcast-creator` library. It operates as a SurrealDB command worker, receiving `PodcastGenerationInput` models and returning results that get stored as `PodcastEpisode` records linked to the command ID.

### How can I retry a failed podcast generation?

Use the `POST /podcasts/episodes/{episode_id}/retry` endpoint. This operation deletes the failed episode record from SurrealDB, removes any partial audio files from storage, and submits a new generation job using the original episode and speaker profiles. The new job receives a fresh job ID while maintaining the same content parameters, allowing you to regenerate podcasts after fixing underlying issues like LLM configuration errors or TTS service interruptions.