# How the Open Notebook Podcast Generation Job Queue Uses Surreal-Commands for Async Processing

> Learn how Open Notebook uses Surreal Commands for async podcast generation. Discover background job queues, job IDs, and status polling for efficient audio processing.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-28

---

**Open Notebook delegates long-running podcast generation to a Surreal-Commands background queue, storing the job ID on the `PodcastEpisode` record and polling status via `get_command_status` until the audio file is ready.**

The `lfnovo/open-notebook` repository implements an asynchronous podcast generation pipeline that offloads heavy text-to-speech and audio processing to a background worker system. By integrating **Surreal-Commands**, the application submits jobs, tracks execution state, and retrieves results without blocking the main API thread.

## Submitting Jobs to the Surreal-Commands Queue

The async lifecycle begins when a client requests podcast generation through the REST API.

### The Submission Endpoint

In [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py), the endpoint `POST /podcasts/generate` accepts a `PodcastGenerationRequest` and delegates to `PodcastService.submit_generation_job`. This service method, defined in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), performs three critical actions:

1. Validates the requested **episode profile** and **speaker profile**
2. Imports the `commands.podcast_commands` module to register the command with Surreal-Commands
3. Invokes `submit_command("open_notebook", "generate_podcast", args)` to enqueue the job

The `submit_command` function returns a `RecordID` that is immediately converted to a string and stored as `job_id` in the response, allowing clients to track progress.

```python

# Conceptual flow inside PodcastService.submit_generation_job

args = {
    "episode_profile": request.episode_profile,
    "speaker_profile": request.speaker_profile,
    "notebook_id": request.notebook_id,
    # ... other parameters

}

# Import ensures command is visible to the queue

import commands.podcast_commands

job_id = submit_command("open_notebook", "generate_podcast", args)
return {"job_id": str(job_id)}  # e.g., "job:open_notebook:generate_podcast:0001"

```

## Command Execution and Record Linking

Once queued, the Surreal-Commands worker picks up the task and executes the decorated function in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py).

### The Command Definition

The `generate_podcast` command is defined with the `@command` decorator:

```python
@command("generate_podcast", app="open_notebook", ...)
def generate_podcast_command(input_data):
    # Heavy-weight processing happens here

    ...

```

Inside `generate_podcast_command`, the code creates a `PodcastEpisode` model instance and **persists the command ID** before starting the long-running work. This ensures that even if the process crashes, the job can be tracked.

### Persisting the Job Reference

In [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py), the `PodcastEpisode` model stores the Surreal-Commands job reference in its `command` field. The model overrides `_prepare_save_data` to ensure the field is always converted to a proper `RecordID` before writing to SurrealDB:

```python
class PodcastEpisode(Model):
    command: Optional[RecordID] = None  # Points to Surreal-Commands job

    
    def _prepare_save_data(self, data):
        # Ensures command field is RecordID, not string

        if self.command:
            data['command'] = ensure_record_id(self.command)
        return data

```

This linkage allows the system to query job status directly from the episode record.

## Querying Job Status and Results

Clients poll for completion using the job ID returned during submission.

### Status Polling Endpoint

The endpoint `GET /podcasts/jobs/{job_id}` in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) forwards to `PodcastService.get_job_status`. This method simply awaits `get_command_status(job_id)` from Surreal-Commands, returning the raw status, result payload, timestamps, and any error messages:

```python
async def get_job_status(job_id: str):
    # Direct proxy to Surreal-Commands

    return await get_command_status(job_id)

```

### Embedding Status in Episode Metadata

When listing episodes, the UI often needs job status alongside episode metadata. The `PodcastEpisode` model provides `get_job_detail()`, which wraps `get_command_status` and returns the job state for completed, failed, or running jobs.

## Handling Failures and Retries

If a generation job fails, the system provides a clean retry mechanism that avoids stale state.

### The Retry Flow

The endpoint `POST /podcasts/episodes/{episode_id}/retry` performs the following actions:

1. Deletes the failed `PodcastEpisode` record from the database
2. Removes any partially generated audio files from storage
3. Re-submits a fresh job via `PodcastService.submit_generation_job`, generating a new Surreal-Commands `RecordID`

This guarantees a clean slate and prevents confusion between the old failed job and the new retry attempt.

## Summary

- **Job Submission**: `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) calls `submit_command` to enqueue work and returns a `job_id` string.
- **Command Execution**: The `@command` decorated function in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) creates a `PodcastEpisode` and stores the `command_id` before processing audio.
- **Record Linking**: The `PodcastEpisode.command` field in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) maintains a `RecordID` reference to the Surreal-Commands job via `_prepare_save_data`.
- **Status Polling**: `GET /podcasts/jobs/{job_id}` uses `get_command_status` to return real-time job state without database polling.
- **Failure Recovery**: The retry endpoint deletes failed records and audio files, then submits a fresh job for clean reprocessing.

## Frequently Asked Questions

### How does Open Notebook track the status of a podcast generation job?

The system stores the Surreal-Commands `RecordID` in the `command` field of the `PodcastEpisode` model. When the UI requests status, the API calls `get_command_status(job_id)` from the Surreal-Commands library, which returns the current state (`submitted`, `running`, `completed`, or `failed`) along with result data or error messages.

### What happens if a podcast generation job fails?

If a job fails, the `POST /podcasts/episodes/{episode_id}/retry` endpoint deletes the failed episode record and any associated audio files, then submits a fresh generation job via `submit_generation_job`. This creates a new Surreal-Commands `RecordID` and ensures no stale data persists from the failed attempt.

### How is the Surreal-Commands job ID linked to a PodcastEpisode record?

During command execution in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py), the `generate_podcast_command` function creates a `PodcastEpisode` and assigns `input_data.execution_context.command_id` to the episode's `command` field. The model's `_prepare_save_data` method ensures this value is persisted as a proper `RecordID` in SurrealDB.

### Can I poll the job status without querying the database directly?

Yes. The API provides the `GET /podcasts/jobs/{job_id}` endpoint, which accepts the job ID string returned during submission and returns the status directly from Surreal-Commands. This endpoint does not require database access from the client, as it delegates to `get_command_status(job_id)` implemented in the Surreal-Commands library.