# How Open Notebook Leverages Surreal-Commands for Podcast Generation Async Job Queues

> Discover how the Open Notebook podcast generation async job queue uses Surreal-Commands for status polling, result retrieval, and automatic retries. Learn more about this efficient workflow.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-21

---

**Open Notebook delegates podcast generation to an asynchronous job queue powered by Surreal-Commands, storing job identifiers on PodcastEpisode records to enable status polling, result retrieval, and automatic retry mechanisms.**

The [lfnovo/open-notebook](https://github.com/lfnovo/open-notebook) repository implements a robust background processing system for AI-generated podcasts. By integrating Surreal-Commands—a job queue system built on SurrealDB—the application offloads long-running podcast creation tasks from the main API thread while maintaining persistent links between episodes and their execution status.

## Submitting a Generation Job

When a client requests podcast generation via `POST /podcasts/generate`, the system initiates an async workflow through the `PodcastService.submit_generation_job` method. Located in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), this function validates the provided episode and speaker profiles, constructs a command argument dictionary, and imports the `commands.podcast_commands` module to make the command visible to the Surreal-Commands dispatcher.

The critical submission occurs via `submit_command("open_notebook", "generate_podcast", args)`, which returns a SurrealDB `RecordID`. This identifier is immediately stored as a string in the `job_id` field and returned to the client, enabling subsequent status checks.

## Executing the Command

Surreal-Commands processes the job by invoking the `generate_podcast_command` function defined in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py). The command uses the `@command("generate_podcast", app="open_notebook", ...)` decorator to register itself with the dispatcher.

Before executing the heavy-weight podcast creator pipeline, the command creates a `PodcastEpisode` model and stores the command's `command_id` on the episode's `command` field using `ensure_record_id(input_data.execution_context.command_id)`. This persistence step occurs **before** the long-running work begins, ensuring that even if the process crashes, the episode record maintains a link to the job for status tracking and retry purposes.

## Persisting the Job Link

The `PodcastEpisode` model in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) treats the `command` field as a `RecordID` that points back to the Surreal-Commands job. The model overrides `_prepare_save_data` to ensure this field always converts to a proper `RecordID` before writing to SurrealDB, maintaining referential integrity between the episode and its background job.

## Querying Job Status

Clients poll for completion via `GET /podcasts/jobs/{job_id}`, handled by the `get_podcast_job_status` router in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). This endpoint delegates to `PodcastService.get_job_status`, which calls `await get_command_status(job_id)` from Surreal-Commands.

The response includes raw status values (`submitted`, `running`, `completed`, `failed`), timestamps, results, and error messages. When listing episodes, the system uses `episode.get_job_detail()` to embed current job status alongside episode metadata, allowing the UI to display real-time generation progress.

## Retry and Cleanup Mechanisms

If a job fails, the `POST /podcasts/episodes/{episode_id}/retry` endpoint provides recovery. The router deletes the failed episode record—including any associated audio files—and invokes `PodcastService.submit_generation_job` to create a fresh Surreal-Commands job with a new `RecordID`. This guarantees a clean execution state while preserving the user's original generation parameters.

## Code Examples

### Submitting a Podcast Generation Job

```python
import httpx

payload = {
    "episode_profile": "my-episode-profile",
    "speaker_profile": "default-speaker",
    "episode_name": "TheFutureOfAI",
    "notebook_id": "notebook-123",
    "briefing_suffix": "Add a short intro."
}

resp = httpx.post("http://localhost:5055/podcasts/generate", json=payload)
job = resp.json()
print(f"Job submitted – ID: {job['job_id']}")

```

### Polling Job Status

```python
import time
import httpx

job_id = "job:open_notebook:generate_podcast:0001"
while True:
    r = httpx.get(f"http://localhost:5055/podcasts/jobs/{job_id}")
    data = r.json()
    print(f"Status: {data['status']}")
    if data["status"] in ("completed", "failed"):
        break
    time.sleep(2)

```

### Retrying a Failed Episode

```python
import httpx

failed_episode = "episode:bad123"
r = httpx.post(f"http://localhost:5055/podcasts/episodes/{failed_episode}/retry")
print(r.json())

```

## Summary

- Open Notebook uses **Surreal-Commands** to manage podcast generation as asynchronous background jobs, preventing API blocking during long-running AI processing.
- The `submit_command` function in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) returns a **SurrealDB RecordID** that links episode records to their execution jobs via the `command` field.
- The `generate_podcast_command` in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) persists episode data before processing begins, ensuring crash recovery and status tracking capabilities.
- Job status polling flows through `get_command_status` from Surreal-Commands, accessed via `GET /podcasts/jobs/{job_id}` endpoints.
- Failed jobs support automatic retry via the retry endpoint, which cleans up corrupted state and submits fresh jobs with new identifiers.

## Frequently Asked Questions

### How does Surreal-Commands integrate with the Open Notebook podcast system?

Surreal-Commands acts as the asynchronous job queue layer built on SurrealDB. When a user requests podcast generation, the system calls `submit_command("open_notebook", "generate_podcast", args)` to queue the work, then polls `get_command_status(job_id)` to track execution. The command decorator `@command("generate_podcast", app="open_notebook", ...)` registers the podcast generation logic with the Surreal-Commands dispatcher.

### What happens if a podcast generation job crashes midway?

The system implements defensive persistence by creating and saving the `PodcastEpisode` record **before** starting the heavy-weight pipeline. The episode stores the Surreal-Commands `command_id` in its `command` field, allowing the system to track job status even if the worker process crashes. Users can query the job status to detect failures and trigger the retry endpoint, which cleans up the failed state and submits a fresh job.

### How is the Surreal-Commands job ID linked to the podcast episode?

The `PodcastEpisode` model in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) maintains a `command` field of type `RecordID` that references the Surreal-Commands job. The model overrides `_prepare_save_data` to ensure proper serialization of this reference to SurrealDB, creating a bidirectional link between the episode and its background execution context.

### Can I check the status of a running podcast generation job?

Yes. The API exposes `GET /podcasts/jobs/{job_id}` which delegates to `PodcastService.get_job_status` and returns the raw Surreal-Commands status including `submitted`, `running`, `completed`, or `failed` states, along with timestamps and error details. The endpoint uses `get_command_status(job_id)` from the Surreal-Commands library to fetch real-time execution data.