How Open Notebook's Podcast Async Job Queue Handles Failures and Retries

Open Notebook manages podcast generation failures through a surreal-commands async queue that captures all exceptions, disables automatic retries via a max_attempts: 1 configuration, and exposes a manual retry endpoint that deletes failed episodes and re-queues fresh generation jobs.

Open Notebook implements podcast generation as an asynchronous command processed by the surreal-commands job queue. Understanding how this system handles podcast async job queue failure handling is critical for developers building reliable audio generation pipelines. The architecture intentionally avoids automatic retries to prevent expensive reprocessing, instead providing explicit manual recovery mechanisms.

Job Submission and Queue Registration

Podcast generation begins in api/podcast_service.py where the PodcastService.submit_generation_job method validates the request and registers the task with the surreal-commands queue.

# From api/podcast_service.py

job_id = submit_command("open_notebook", "generate_podcast", command_args)

The submit_command function returns a record-ID that serves as the job_id exposed to API clients. This identifier allows persistent tracking of the generation task through completion or failure.

Exception Handling in the Generation Command

The actual processing logic resides in commands/podcast_commands.py within the generate_podcast_command function. This function is wrapped with the @command decorator that defines the retry policy and failure handling behavior.

# From commands/podcast_commands.py

@command("generate_podcast", app="open_notebook", retry={"max_attempts": 1})
async def generate_podcast_command(input_data: PodcastGenerationInput) -> PodcastGenerationOutput:
    try:
        # Business logic for podcast generation

        ...
    except ValueError as e:
        # Re-raise directly for profile/configuration errors

        raise
    except Exception as e:
        # Log and wrap unexpected errors

        logger.error(f"Podcast generation failed: {e}")
        raise RuntimeError(f"Podcast generation failed: {e}")

The implementation distinguishes between error types:

• ValueError (e.g., missing profiles) is re-raised immediately, causing the command to terminate with a failed status
• All other exceptions are logged and re-raised as RuntimeError, which the decorator captures to populate the error_message field in the job status record

Automatic Retry Policy

The @command decorator explicitly disables automatic retries through the configuration retry={"max_attempts": 1}. This setting ensures that each podcast generation job is attempted exactly once.

This design decision prevents expensive reprocessing of potentially flawed inputs. Since podcast generation involves resource-intensive operations and may require user-level corrections (such as fixing a profile configuration), the system leaves failed jobs in a terminal state rather than attempting automatic recovery.

Manual Retry Endpoint

Because automatic retries are disabled, Open Notebook provides a dedicated HTTP endpoint for manual recovery. Located in api/routers/podcasts.py, the retry route handles the entire recovery workflow:

# From api/routers/podcasts.py

@router.post("/podcasts/episodes/{episode_id}/retry")
async def retry_podcast_episode(episode_id: str):
    # 1. Load the failed episode

    episode = await PodcastService.get_episode(episode_id)
    
    # 2. Verify the job status is failed/error

    detail = await episode.get_job_detail()
    if detail["status"] not in ("failed", "error"):
        raise HTTPException(status_code=400, detail="Episode is not in a failed state")
    
    # 3. Delete the failed episode record

    await episode.delete()
    
    # 4. Re-submit a fresh generation job

    job_id = await PodcastService.submit_generation_job(
        episode_profile_name=ep_profile_name,
        speaker_profile_name=sp_profile_name,
        episode_name=episode_name,
        content=content,
    )
    return {"job_id": job_id, "message": "Retry submitted successfully"}

The manual retry process performs four critical steps:

1. Validation: Confirms the episode status is either failed or error
2. Cleanup: Removes the failed PodcastEpisode record and any orphaned audio files
3. Re-submission: Creates a new job via PodcastService.submit_generation_job
4. Response: Returns the new job_id for tracking

Monitoring Job Status

Clients can poll the job status through the /api/podcasts/jobs/{job_id} endpoint implemented in the same router. This endpoint leverages get_command_status from api/podcast_service.py to return comprehensive status information.

# Polling example

status = await PodcastService.get_job_status(job_id)
print(status["status"])          # "pending", "running", "completed", "failed"

print(status.get("error_message"))  # Detailed error if failed

The status payload includes the current state, result data, error messages, and timestamps, enabling clients to implement appropriate polling strategies and user notifications.

Complete Workflow Example

The following example demonstrates the full lifecycle of error handling:

# 1. Submit initial job

job_id = await PodcastService.submit_generation_job(
    episode_profile_name="MyProfile",
    speaker_profile_name="MySpeaker",
    episode_name="The Great Adventure",
    notebook_id="notebook:123",
)

# 2. Check status after failure

status = await PodcastService.get_job_status(job_id)
if status["status"] == "failed":
    print(f"Error: {status['error_message']}")
    
    # 3. Manual retry via API

    import httpx
    async with httpx.AsyncClient(base_url="http://localhost:5055") as client:
        resp = await client.post("/api/podcasts/episodes/episode:42/retry")
        new_job_id = resp.json()["job_id"]

Summary

• Job Registration: PodcastService.submit_generation_job in api/podcast_service.py queues tasks via surreal-commands and returns a trackable job_id
• Zero Automatic Retries: The @command decorator in commands/podcast_commands.py sets max_attempts: 1 to prevent expensive reprocessing
• Exception Capture: All business logic errors are caught, logged, and stored in the job's error_message field
• Manual Recovery: The /podcasts/episodes/{episode_id}/retry endpoint in api/routers/podcasts.py validates failures, deletes stale records, and submits fresh jobs
• Status Visibility: The /podcasts/jobs/{job_id} endpoint provides real-time status, error details, and timestamps

Frequently Asked Questions

What happens when a podcast generation job fails?

When generate_podcast_command encounters an exception, the @command decorator captures the error, marks the job status as failed, and persists the error message in the command's status record. The exception is logged via logger.error and the job remains in a terminal failed state without automatic retry attempts.

Why doesn't Open Notebook automatically retry failed podcast jobs?

The system configures max_attempts: 1 in the command decorator to disable automatic retries. This design prevents expensive reprocessing of podcast generation tasks that may fail due to configuration issues requiring user intervention, such as invalid profile settings or missing content.

How do I manually retry a failed podcast episode?

Submit a POST request to /api/podcasts/episodes/{episode_id}/retry. The endpoint verifies the episode is in a failed or error state, deletes the existing episode record (including any orphaned audio files), and re-submits a fresh generation job via PodcastService.submit_generation_job, returning a new job_id for tracking.

Where can I check the status of a podcast generation job?

Query the /api/podcasts/jobs/{job_id} endpoint or use PodcastService.get_job_status(job_id) in api/podcast_service.py. The response includes the current status (pending, running, completed, failed), error messages for failed jobs, and timestamps for each state transition.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}