# Podcast Generation Async Job Queue Architecture with Surreal-Commands

> Explore a robust async job queue architecture for podcast generation using Surreal-Commands. Open Notebook manages long-running tasks, persists job IDs, and offers REST endpoints for status and retries.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-07-04

---

**Open Notebook orchestrates long-running podcast generation tasks by submitting jobs to Surreal-Commands, persisting the command RecordID on the PodcastEpisode model, and exposing REST endpoints for status polling and retry operations.**

The Open Notebook repository implements a robust asynchronous processing pipeline that decouples podcast generation requests from heavy computational workloads. This architecture leverages Surreal-Commands as the underlying job queue backend, providing reliable execution tracking, automatic failure recovery, and seamless integration with SurrealDB for state persistence.

## Stage 1: Submitting a Generation Job

The async flow begins when a client sends a `POST` request to `/podcasts/generate` with a `PodcastGenerationRequest` payload. In [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), the `submit_generation_job` method validates the requested profiles and constructs a command argument dictionary.

Before submitting, the code explicitly imports `commands.podcast_commands` to ensure the command is registered and visible to the Surreal-Commands dispatcher. The method then calls `submit_command("open_notebook", "generate_podcast", args)`, which returns a SurrealDB `RecordID` stored as a string in the `job_id` field. This identifier is returned to the client immediately, allowing the API to respond within milliseconds while the background job executes asynchronously.

## Stage 2: Command Execution and Episode Creation

When Surreal-Commands picks up the job, it executes the `generate_podcast_command` function defined in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py). The function is registered with the `@command("generate_podcast", app="open_notebook", …)` decorator, making it discoverable by the job queue.

Critically, the command creates a `PodcastEpisode` model **before** starting the long-running audio generation pipeline. It stores the Surreal-Commands `command_id` on the episode's `command` field using `ensure_record_id(input_data.execution_context.command_id)`. This early persistence guarantees that the job can be tracked and recovered even if the process crashes during the heavy computational phase.

Once the episode record is saved with the command link, the pipeline proceeds to generate the outline, transcript, and audio file, updating the record as each component completes.

## Stage 3: Persisting the Job Link

The connection between a podcast episode and its background job relies on proper RecordID handling in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py). The `PodcastEpisode` model defines a `command` field that stores the Surreal-Commands job reference as a `RecordID`.

To ensure database consistency, the model overrides `_prepare_save_data` to convert the `command` field to a proper `RecordID` before writing to SurrealDB. This bidirectional linkage allows the system to query job status directly from the episode record and enables the UI to display real-time generation progress alongside episode metadata.

## Stage 4: Querying Status and Results

Clients poll for job status via `GET /podcasts/jobs/{job_id}`, which routes to `PodcastService.get_job_status` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py). This method calls `await get_command_status(job_id)` from Surreal-Commands and returns the raw status, result payload, timestamps, and any error messages.

For episode listings, the `PodcastEpisode` model provides `get_job_detail()`, which embeds the current job status alongside episode metadata. When the command finishes successfully, the episode fields `audio_file`, `transcript`, and `outline` are populated, and the status transitions to `completed`.

## Stage 5: Retry and Cleanup

If a generation job fails, the system provides a clean retry mechanism via `POST /podcasts/episodes/{episode_id}/retry` in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). The endpoint deletes the failed episode record—including any associated audio files—and submits a fresh job via `PodcastService.submit_generation_job`.

This approach guarantees a clean slate by generating a new Surreal-Commands `RecordID` rather than attempting to reuse a corrupted job state. The retry logic ensures that transient failures in the podcast generation pipeline can be recovered without manual database intervention.

## Implementation Examples

### Submitting a Podcast Generation Job

```python
import httpx

payload = {
    "episode_profile": "my‑episode‑profile",
    "speaker_profile": "default‑speaker",
    "episode_name": "TheFutureOfAI",
    "notebook_id": "notebook‑123",
    "briefing_suffix": "Add a short intro."
}

resp = httpx.post("http://localhost:5055/podcasts/generate", json=payload)
job = resp.json()
print(f"Job submitted – ID: {job['job_id']}")

```

### Polling Job Status

```python
import time
import httpx

job_id = "job:open_notebook:generate_podcast:0001"
while True:
    r = httpx.get(f"http://localhost:5055/podcasts/jobs/{job_id}")
    data = r.json()
    print(f"Status: {data['status']}")
    if data["status"] in ("completed", "failed"):
        break
    time.sleep(2)

```

### Retrieving a Completed Episode

```python
import httpx

episode_id = "episode:123456"
r = httpx.get(f"http://localhost:5055/podcasts/episodes/{episode_id}")
episode = r.json()
print("Audio URL:", episode["audio_url"])

```

### Retrying a Failed Episode

```python
import httpx

failed_episode = "episode:bad123"
r = httpx.post(f"http://localhost:5055/podcasts/episodes/{failed_episode}/retry")
print(r.json())

```

## Key Source Files

- **[`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)**: Contains `submit_generation_job` and `get_job_status` methods that interface with Surreal-Commands via `submit_command` and `get_command_status`.

- **[`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)**: Defines the `generate_podcast` command decorated with `@command`, responsible for creating the episode record and executing the podcast generation pipeline.

- **[`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)**: Implements the `PodcastEpisode` model with the `command` field and helper methods `get_job_detail` and `_prepare_save_data` for RecordID handling.

- **[`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)**: FastAPI router exposing endpoints for job submission (`POST /podcasts/generate`), status polling (`GET /podcasts/jobs/{job_id}`), and retry operations (`POST /podcasts/episodes/{episode_id}/retry`).

## Summary

- **Open Notebook** uses **Surreal-Commands** as the async job queue backend for podcast generation tasks.
- Jobs are submitted via `submit_command("open_notebook", "generate_podcast", args)` which returns a `RecordID` stored on the `PodcastEpisode` model.
- The episode record is created **before** the heavy processing begins, ensuring trackability even if the job crashes.
- Status polling uses `get_command_status(job_id)` to return real-time state including `submitted`, `running`, `completed`, or `failed`.
- Failed jobs support clean retries by deleting the old record and submitting a fresh Surreal-Commands job.

## Frequently Asked Questions

### How does the system handle failures during podcast generation?

The `generate_podcast_command` in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) creates and saves the `PodcastEpisode` record before starting the heavy audio generation work. This ensures that even if the process crashes, the job identifier remains persisted in SurrealDB. For explicit failures, the retry endpoint deletes the failed episode and audio files, then submits a completely new job via `submit_generation_job`.

### What is the relationship between the PodcastEpisode and the Surreal-Commands job?

The `PodcastEpisode` model stores the Surreal-Commands `RecordID` in its `command` field, established via `ensure_record_id(input_data.execution_context.command_id)`. This creates a bidirectional link where the episode can query its job status through `get_job_detail()`, which wraps `surreal_commands.get_command_status()`.

### Can I check the status of a podcast generation job without querying the episode record?

Yes. The API exposes `GET /podcasts/jobs/{job_id}` which directly queries Surreal-Commands via `PodcastService.get_job_status`. This returns the raw job state including status, result data, timestamps, and error messages without requiring access to the episode model.

### Why is the command module imported explicitly in the submission code?

The `submit_generation_job` method in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) imports `commands.podcast_commands` to ensure the command is registered and visible to the Surreal-Commands dispatcher before calling `submit_command`. This explicit import guarantees that the job queue can locate and execute the `generate_podcast` command when processing the task.