# Async Job Queue Architecture for Podcast Generation and Status Tracking in Open‑Notebook

> Discover the async job queue architecture for podcast generation and status tracking in Open-Notebook. Learn how it uses SurrealDB and background workers for efficient audio synthesis and real-time updates.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-19

---

**Open‑Notebook leverages the surreal‑commands library to decouple API requests from CPU‑intensive audio synthesis, using a three‑layer architecture that submits jobs to SurrealDB, processes them in background workers, and exposes real‑time status endpoints for polling.**

The `lfnovo/open‑notebook` repository implements a robust asynchronous pipeline for creating AI‑generated podcasts. By utilizing an **async job queue architecture for podcast generation and status tracking**, the application ensures that long‑running text‑to‑speech operations never block the HTTP API, allowing users to queue episodes and monitor progress through dedicated status endpoints.

## Three‑Layer Async Architecture

The podcast generation flow is divided into three distinct layers, each handled by specific modules in the codebase.

### Service Layer – Job Submission and Validation

Located in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), the `PodcastService` class handles request validation and job enqueueing. The `submit_generation_job` method performs six critical steps before returning a job identifier:

1. **Profile validation** – confirms the requested `EpisodeProfile` and `SpeakerProfile` exist in SurrealDB.
2. **Content resolution** – extracts text from a notebook (via `notebook_id`) or uses explicit `content` arguments.
3. **Command preparation** – builds a `command_args` dictionary containing episode name, profiles, content, and optional briefing suffixes.
4. **Dynamic import** – forces import of `commands.podcast_commands` to register the `generate_podcast` command in the surreal‑commands registry.
5. **Job submission** – calls `surreal_commands.submit_command("open_notebook", "generate_podcast", command_args)`, which returns a SurrealDB **RecordID**.
6. **Return** – converts the RecordID to a string (`job_id`) for the client.

### Command Layer – Background Execution

The heavy lifting occurs in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) inside the `generate_podcast_command` function, decorated with `@command("generate_podcast", ...)` to register it with the queue worker. The command executes eight steps asynchronously:

1. **Load profiles** – fetches episode and speaker configuration from SurrealDB.
2. **Model resolution** – maps profile‑specified LLM/TTS names to concrete provider credentials using `_resolve_model_config`.
3. **Create episode record** – instantiates a `PodcastEpisode` record linked to the command ID for correlation.
4. **Configure engine** – injects resolved model configs into the `podcast‑creator` library.
5. **Prepare output directory** – generates a UUID‑named folder under `DATA_FOLDER/podcasts/episodes/` (defined in [`open_notebook/config.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/config.py)).
6. **Generate audio** – awaits `create_podcast(...)` to produce audio, transcript, and outline data.
7. **Persist results** – updates the `PodcastEpisode` record with `audio_file`, `transcript`, and `outline` fields.
8. **Return output** – produces a `PodcastGenerationOutput` object containing `success`, `episode_id`, file paths, and processing metadata.

If any step fails, the command raises an exception, and the job status is automatically set to **error** with an explanatory message.

### Status Layer – Progress Monitoring

Clients query job state through `PodcastService.get_job_status(job_id)`, which wraps `surreal_commands.get_command_status`. The returned payload includes:

- `status` – `queued`, `running`, `completed`, or `error`
- `result` – serialized `PodcastGenerationOutput` when finished
- `error_message` – populated on failure
- `created` and `updated` timestamps

The FastAPI router in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) exposes this via HTTP endpoints, enabling front‑end polling for live progress bars.

## Submitting a Podcast Generation Job

To queue a new episode, call the service layer with profile names and optional notebook content:

```python
from api.podcast_service import PodcastService

job_id = await PodcastService.submit_generation_job(
    episode_profile_name="TechTalk",
    speaker_profile_name="DefaultSpeaker",
    episode_name="AI in 2026",
    notebook_id="notebook:12345",   # optional – pulls notebook content

    briefing_suffix="Add a short intro about Open‑Notebook",
)

print(f"Job queued with ID: {job_id}")

```

Alternatively, trigger the flow via the REST API:

```bash
curl -X POST http://localhost:5055/podcasts/generate \
  -H "Content-Type: application/json" \
  -d '{
        "episode_profile": "TechTalk",
        "speaker_profile": "DefaultSpeaker",
        "episode_name": "AI in 2026",
        "notebook_id": "notebook:12345",
        "briefing_suffix": "Add a short intro about Open‑Notebook"
      }'

```

## Processing Inside the Background Worker

Once dequeued, the worker executes `generate_podcast_command`. The function ensures filesystem safety by creating a unique directory under `DATA_FOLDER/podcasts/episodes/<uuid>/` before invoking the `podcast‑creator` library. After successful generation, the audio file path and transcript are committed to the `PodcastEpisode` record in SurrealDB, making them immediately queryable via the service layer.

## Tracking Job Status and Retrieving Results

Poll the status endpoint to monitor the asynchronous operation:

```bash
curl http://localhost:5055/podcasts/status/<job_id>

```

Once `status` returns `"completed"`, extract the episode details:

```python
from api.podcast_service import PodcastService

status = await PodcastService.get_job_status(job_id)
if status["status"] == "completed":
    episode_id = status["result"]["episode_id"]
    episode = await PodcastService.get_episode(episode_id)
    print("Audio file:", episode.audio_file)

```

## Summary

- **Decoupled architecture** – The `surreal‑commands` library manages job state in SurrealDB, separating HTTP request handling from long‑running audio synthesis.
- **Service layer** – `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) validates profiles and notebook content before returning a job ID.
- **Background execution** – `generate_podcast_command` in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) runs the heavy lifting, storing results in `PodcastEpisode` records under `DATA_FOLDER/podcasts/episodes/`.
- **Status polling** – `PodcastService.get_job_status` exposes real‑time states (`queued`, `running`, `completed`, `error`) and final output data.

## Frequently Asked Questions

### What job queue system does Open‑Notebook use for podcast generation?

Open‑Notebook uses **surreal‑commands**, a job queue system built on SurrealDB. It stores task metadata in the `commands` table and dispatches work to background workers that execute registered Python functions like `generate_podcast_command`.

### How does the system validate requests before queuing a job?

The `submit_generation_job` method in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) validates that the requested `EpisodeProfile` and `SpeakerProfile` exist in the database and resolves notebook content before invoking `surreal_commands.submit_command`. This ensures only valid configuration reaches the background worker.

### Where are generated podcast files stored?

Audio files, transcripts, and outlines are saved in a UUID‑named folder under `DATA_FOLDER/podcasts/episodes/`, as defined in [`open_notebook/config.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/config.py). The relative path is persisted in the `audio_file` field of the `PodcastEpisode` record stored in SurrealDB.

### What status values can a podcast generation job return?

According to the `surreal_commands.get_command_status` implementation, jobs return statuses including `queued`, `running`, `completed`, or `error`. Completed jobs include a `result` field containing the `PodcastGenerationOutput`, while failed jobs provide an `error_message` detailing the cause.