# How Podcast Generation Works with the Job Queue in Open Notebook

> Learn how Open Notebook uses the job queue for asynchronous podcast generation. Discover how audio synthesis processing happens in the background with a quick API response.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-23

---

**Open Notebook executes podcast generation as asynchronous background jobs through the surreal-commands queue system, allowing the API to return a job ID immediately while long-running audio synthesis processing occurs in the background.**

Open Notebook leverages the surreal-commands job queue architecture to decouple resource-intensive text-to-speech processing from HTTP request handling. When users request a podcast episode, the system validates configuration profiles, enqueues a background task, and returns a job identifier that clients poll for completion status. This design ensures the API remains responsive even when generating multi-minute audio content.

## Architecture Overview

### The Async Job Queue Pattern

The implementation relies on **surreal-commands** to manage distributed task execution. Rather than blocking the HTTP thread during LLM inference and audio generation, Open Notebook delegates work to background workers. This pattern separates concerns between the FastAPI router layer ([`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)), the business logic service ([`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)), and the actual command implementation ([`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)).

Key components include:

- **Command Registration**: Background commands are auto-registered when `commands/podcast_commands` is imported
- **Job Enqueueing**: `submit_command("open_notebook", "generate_podcast", ...)` places work in the queue
- **Status Tracking**: Jobs maintain state (submitted, running, completed, failed) with timestamps and error messages
- **File Isolation**: UUID-based output directories prevent collision between concurrent generation jobs

## The Podcast Generation Workflow

### Step 1: API Request and Validation

Clients initiate generation by POSTing to `/podcasts/generate` with a `PodcastGenerationRequest` payload containing `episode_profile`, `speaker_profile`, and optionally `notebook_id` or raw `content`. The router in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) (lines 41-56) delegates to `PodcastService.submit_generation_job`.

The service layer validates that referenced profiles exist by calling `EpisodeProfile.get_by_name` and `SpeakerProfile.get_by_name`. If validation fails, the API returns a 404 error before any queue operations occur.

### Step 2: Content Preparation

When `notebook_id` is provided without explicit `content`, the service retrieves notebook context via `Notebook.get` followed by `notebook.get_context` (as implemented in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) lines 46-68). This extracts the text content that will form the basis of the podcast script.

### Step 3: Job Enqueueing

After validation, the service ensures the command module is loaded (`import commands.podcast_commands` to trigger registration), then calls `submit_command("open_notebook", "generate_podcast", command_args)` as shown in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) lines 95-104. This returns a `job_id` that the API immediately returns to the client, enabling non-blocking operation.

### Step 4: Background Execution

The queued job invokes `generate_podcast_command` defined in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) (lines 69-74). This function performs the heavy lifting:

- Loads episode and speaker configurations from SurrealDB
- Resolves LLM and TTS model configurations via `_resolve_model_config`
- Creates a **UUID-based output directory** using `build_episode_output_dir` (lines 26-38) to store generated files safely
- Configures the `podcast-creator` library with speaker and episode profiles
- Executes `await create_podcast(...)` (lines 31-35, 44-52) to generate audio
- Persists results to a `PodcastEpisode` record via `episode.save()` (lines 28-33, 56-64), storing file paths, transcript, and outline

### Step 5: Status Polling and Monitoring

Clients query `/podcasts/jobs/{job_id}` to track progress. The router calls `PodcastService.get_job_status`, which forwards to `surreal_commands.get_command_status` ([`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) lines 14-34). The response includes `status`, `result`, `error_message`, `created`/`updated` timestamps, and `progress` indicators.

### Step 6: Audio Delivery and Retry Mechanisms

Once completed, episodes appear in `/podcasts/episodes`. Each entry includes an `audio_url` path that streams via `/podcasts/episodes/{episode_id}/audio` using FastAPI's `FileResponse` (implemented in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) lines 90-115).

If generation fails, clients can POST to `/podcasts/episodes/{episode_id}/retry` (lines 54-61). This deletes the failed record and its associated audio file, then submits a fresh job using the original profiles. Direct episode deletion (lines 72-80) also cleans up underlying audio files from disk.

## API Usage Examples

### Submit a Generation Request

```bash
curl -X POST http://localhost:5055/api/podcasts/generate \
  -H "Content-Type: application/json" \
  -d '{
        "episode_profile": "MyEpisode",
        "speaker_profile": "MySpeaker",
        "episode_name": "AI Trends 2024",
        "notebook_id": "notebook123"
      }'

```

Response:

```json
{
  "job_id": "cmd-01HZ2X...",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "MyEpisode",
  "episode_name": "AI Trends 2024"
}

```

### Poll Job Status Programmatically

```python
import httpx

base = "http://localhost:5055/api"
job_id = "cmd-01HZ2X..."

resp = httpx.get(f"{base}/podcasts/jobs/{job_id}")
print(resp.json())

```

Typical completed payload:

```json
{
  "job_id": "cmd-01HZ2X...",
  "status": "completed",
  "result": null,
  "error_message": null,
  "created": "2026-06-23T12:34:56Z",
  "updated": "2026-06-23T12:35:30Z",
  "progress": null
}

```

### List Generated Episodes

```bash
curl http://localhost:5055/api/podcasts/episodes

```

Each response entry contains `audio_url` for streaming.

### Stream Audio Content

```bash
ffplay http://localhost:5055/api/podcasts/episodes/ep-01ABCD/audio

```

### Retry Failed Episodes

```bash
curl -X POST http://localhost:5055/api/podcasts/episodes/ep-01ABCD/retry

```

Returns a new `job_id` for the retry attempt.

## Key Implementation Files

| Component | File Path | Purpose |
|-----------|-----------|---------|
| **Command Definition** | [[`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) | Implements `generate_podcast_command` that executes in background workers, handles profile resolution, and manages audio generation via `podcast-creator` |
| **Service Layer** | [[`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) | Validates input parameters, orchestrates content fetching, submits jobs to surreal-commands queue, and provides status checking utilities |
| **API Router** | [[`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) | Exposes REST endpoints for job submission (`/generate`), status polling (`/jobs/{id}`), episode listing, audio streaming, retry operations, and deletion |
| **Documentation** | [[`docs/2-CORE-CONCEPTS/podcasts-explained.md`](https://github.com/lfnovo/open-notebook/blob/main/docs/2-CORE-CONCEPTS/podcasts-explained.md)](https://github.com/lfnovo/open-notebook/blob/main/docs/2-CORE-CONCEPTS/podcasts-explained.md) | High-level architectural documentation explaining podcast concepts and configuration profiles |
| **Test Suite** | [[`tests/test_podcast_path.py`](https://github.com/lfnovo/open-notebook/blob/main/tests/test_podcast_path.py)](https://github.com/lfnovo/open-notebook/blob/main/tests/test_podcast_path.py) | Unit tests verifying API route integration and job queue behavior |

## Summary

- **Open Notebook** uses the **surreal-commands** job queue to run podcast generation asynchronously, preventing HTTP timeouts during long-running audio synthesis
- The workflow spans validation (`EpisodeProfile.get_by_name`, `SpeakerProfile.get_by_name`), content resolution (`Notebook.get`), job enqueueing (`submit_command`), and background execution (`generate_podcast_command`)
- Clients receive immediate `job_id` responses and poll `/podcasts/jobs/{job_id}` for completion status
- Generated episodes expose streaming URLs via `/podcasts/episodes/{episode_id}/audio` with automatic file cleanup on deletion or retry
- All audio files are isolated in UUID-based directories created by `build_episode_output_dir` to prevent naming collisions

## Frequently Asked Questions

### What job queue system does Open Notebook use for podcast generation?

Open Notebook uses **surreal-commands**, a command queue system that integrates with SurrealDB. According to the source code in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), the service calls `submit_command("open_notebook", "generate_podcast", ...)` to enqueue work and `get_command_status` to poll for updates. This system handles job persistence, worker distribution, and status tracking automatically.

### How does Open Notebook handle content preparation when only a notebook ID is provided?

When the API receives a `notebook_id` without explicit `content`, the `PodcastService` loads the notebook via `Notebook.get` and extracts text using `notebook.get_context` (lines 46-68 in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)). This content then populates the script that the `podcast-creator` library converts to audio during background processing.

### Can failed podcast generation jobs be retried without reconfiguring profiles?

Yes. The `/podcasts/episodes/{episode_id}/retry` endpoint (implemented in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) lines 54-61) allows clients to retry failed episodes. This operation deletes the existing failed record and its associated audio file from the UUID-based output directory, then submits a new generation job using the original `episode_profile` and `speaker_profile` configurations.

### What happens to audio files when a podcast episode is deleted?

When deleting an episode via the API, Open Notebook removes both the database record and the underlying audio file from disk. As implemented in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) lines 72-80, the deletion endpoint locates the file path stored in the `PodcastEpisode` record and performs filesystem cleanup to prevent orphaned audio files from consuming storage.