# How Open Notebook's Podcast Async Job Queue Handles Failures and Retries

> Discover how Open Notebook's async job queue handles failures and retries with surreal-commands, disabling automatic retries and enabling manual re-queuing for robust podcast generation.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-29

---

**Open Notebook manages podcast generation failures through a surreal-commands async queue that captures all exceptions, disables automatic retries via a `max_attempts: 1` configuration, and exposes a manual retry endpoint that deletes failed episodes and re-queues fresh generation jobs.**

Open Notebook implements podcast generation as an asynchronous command processed by the surreal-commands job queue. Understanding how this system handles **podcast async job queue failure handling** is critical for developers building reliable audio generation pipelines. The architecture intentionally avoids automatic retries to prevent expensive reprocessing, instead providing explicit manual recovery mechanisms.

## Job Submission and Queue Registration

Podcast generation begins in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) where the `PodcastService.submit_generation_job` method validates the request and registers the task with the surreal-commands queue.

```python

# From api/podcast_service.py

job_id = submit_command("open_notebook", "generate_podcast", command_args)

```

The `submit_command` function returns a record-ID that serves as the `job_id` exposed to API clients. This identifier allows persistent tracking of the generation task through completion or failure.

## Exception Handling in the Generation Command

The actual processing logic resides in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) within the `generate_podcast_command` function. This function is wrapped with the `@command` decorator that defines the retry policy and failure handling behavior.

```python

# From commands/podcast_commands.py

@command("generate_podcast", app="open_notebook", retry={"max_attempts": 1})
async def generate_podcast_command(input_data: PodcastGenerationInput) -> PodcastGenerationOutput:
    try:
        # Business logic for podcast generation

        ...
    except ValueError as e:
        # Re-raise directly for profile/configuration errors

        raise
    except Exception as e:
        # Log and wrap unexpected errors

        logger.error(f"Podcast generation failed: {e}")
        raise RuntimeError(f"Podcast generation failed: {e}")

```

The implementation distinguishes between error types:

- **ValueError** (e.g., missing profiles) is re-raised immediately, causing the command to terminate with a **failed** status
- **All other exceptions** are logged and re-raised as `RuntimeError`, which the decorator captures to populate the `error_message` field in the job status record

## Automatic Retry Policy

The `@command` decorator explicitly disables automatic retries through the configuration `retry={"max_attempts": 1}`. This setting ensures that each podcast generation job is attempted exactly once.

This design decision prevents expensive reprocessing of potentially flawed inputs. Since podcast generation involves resource-intensive operations and may require user-level corrections (such as fixing a profile configuration), the system leaves failed jobs in a terminal state rather than attempting automatic recovery.

## Manual Retry Endpoint

Because automatic retries are disabled, Open Notebook provides a dedicated HTTP endpoint for manual recovery. Located in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py), the retry route handles the entire recovery workflow:

```python

# From api/routers/podcasts.py

@router.post("/podcasts/episodes/{episode_id}/retry")
async def retry_podcast_episode(episode_id: str):
    # 1. Load the failed episode

    episode = await PodcastService.get_episode(episode_id)
    
    # 2. Verify the job status is failed/error

    detail = await episode.get_job_detail()
    if detail["status"] not in ("failed", "error"):
        raise HTTPException(status_code=400, detail="Episode is not in a failed state")
    
    # 3. Delete the failed episode record

    await episode.delete()
    
    # 4. Re-submit a fresh generation job

    job_id = await PodcastService.submit_generation_job(
        episode_profile_name=ep_profile_name,
        speaker_profile_name=sp_profile_name,
        episode_name=episode_name,
        content=content,
    )
    return {"job_id": job_id, "message": "Retry submitted successfully"}

```

The manual retry process performs four critical steps:

1. **Validation**: Confirms the episode status is either `failed` or `error`
2. **Cleanup**: Removes the failed `PodcastEpisode` record and any orphaned audio files
3. **Re-submission**: Creates a new job via `PodcastService.submit_generation_job`
4. **Response**: Returns the new `job_id` for tracking

## Monitoring Job Status

Clients can poll the job status through the `/api/podcasts/jobs/{job_id}` endpoint implemented in the same router. This endpoint leverages `get_command_status` from [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) to return comprehensive status information.

```python

# Polling example

status = await PodcastService.get_job_status(job_id)
print(status["status"])          # "pending", "running", "completed", "failed"

print(status.get("error_message"))  # Detailed error if failed

```

The status payload includes the current state, result data, error messages, and timestamps, enabling clients to implement appropriate polling strategies and user notifications.

## Complete Workflow Example

The following example demonstrates the full lifecycle of error handling:

```python

# 1. Submit initial job

job_id = await PodcastService.submit_generation_job(
    episode_profile_name="MyProfile",
    speaker_profile_name="MySpeaker",
    episode_name="The Great Adventure",
    notebook_id="notebook:123",
)

# 2. Check status after failure

status = await PodcastService.get_job_status(job_id)
if status["status"] == "failed":
    print(f"Error: {status['error_message']}")
    
    # 3. Manual retry via API

    import httpx
    async with httpx.AsyncClient(base_url="http://localhost:5055") as client:
        resp = await client.post("/api/podcasts/episodes/episode:42/retry")
        new_job_id = resp.json()["job_id"]

```

## Summary

- **Job Registration**: `PodcastService.submit_generation_job` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) queues tasks via surreal-commands and returns a trackable `job_id`
- **Zero Automatic Retries**: The `@command` decorator in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) sets `max_attempts: 1` to prevent expensive reprocessing
- **Exception Capture**: All business logic errors are caught, logged, and stored in the job's `error_message` field
- **Manual Recovery**: The `/podcasts/episodes/{episode_id}/retry` endpoint in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py) validates failures, deletes stale records, and submits fresh jobs
- **Status Visibility**: The `/podcasts/jobs/{job_id}` endpoint provides real-time status, error details, and timestamps

## Frequently Asked Questions

### What happens when a podcast generation job fails?

When `generate_podcast_command` encounters an exception, the `@command` decorator captures the error, marks the job status as **failed**, and persists the error message in the command's status record. The exception is logged via `logger.error` and the job remains in a terminal failed state without automatic retry attempts.

### Why doesn't Open Notebook automatically retry failed podcast jobs?

The system configures `max_attempts: 1` in the command decorator to disable automatic retries. This design prevents expensive reprocessing of podcast generation tasks that may fail due to configuration issues requiring user intervention, such as invalid profile settings or missing content.

### How do I manually retry a failed podcast episode?

Submit a POST request to `/api/podcasts/episodes/{episode_id}/retry`. The endpoint verifies the episode is in a `failed` or `error` state, deletes the existing episode record (including any orphaned audio files), and re-submits a fresh generation job via `PodcastService.submit_generation_job`, returning a new `job_id` for tracking.

### Where can I check the status of a podcast generation job?

Query the `/api/podcasts/jobs/{job_id}` endpoint or use `PodcastService.get_job_status(job_id)` in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py). The response includes the current status (`pending`, `running`, `completed`, `failed`), error messages for failed jobs, and timestamps for each state transition.