Podcast Generation Async Job Queue Architecture with Surreal-Commands

Open Notebook orchestrates long-running podcast generation tasks by submitting jobs to Surreal-Commands, persisting the command RecordID on the PodcastEpisode model, and exposing REST endpoints for status polling and retry operations.

The Open Notebook repository implements a robust asynchronous processing pipeline that decouples podcast generation requests from heavy computational workloads. This architecture leverages Surreal-Commands as the underlying job queue backend, providing reliable execution tracking, automatic failure recovery, and seamless integration with SurrealDB for state persistence.

Stage 1: Submitting a Generation Job

The async flow begins when a client sends a POST request to /podcasts/generate with a PodcastGenerationRequest payload. In api/podcast_service.py, the submit_generation_job method validates the requested profiles and constructs a command argument dictionary.

Before submitting, the code explicitly imports commands.podcast_commands to ensure the command is registered and visible to the Surreal-Commands dispatcher. The method then calls submit_command("open_notebook", "generate_podcast", args), which returns a SurrealDB RecordID stored as a string in the job_id field. This identifier is returned to the client immediately, allowing the API to respond within milliseconds while the background job executes asynchronously.

Stage 2: Command Execution and Episode Creation

When Surreal-Commands picks up the job, it executes the generate_podcast_command function defined in commands/podcast_commands.py. The function is registered with the @command("generate_podcast", app="open_notebook", …) decorator, making it discoverable by the job queue.

Critically, the command creates a PodcastEpisode model before starting the long-running audio generation pipeline. It stores the Surreal-Commands command_id on the episode's command field using ensure_record_id(input_data.execution_context.command_id). This early persistence guarantees that the job can be tracked and recovered even if the process crashes during the heavy computational phase.

Once the episode record is saved with the command link, the pipeline proceeds to generate the outline, transcript, and audio file, updating the record as each component completes.

Stage 3: Persisting the Job Link

The connection between a podcast episode and its background job relies on proper RecordID handling in open_notebook/podcasts/models.py. The PodcastEpisode model defines a command field that stores the Surreal-Commands job reference as a RecordID.

To ensure database consistency, the model overrides _prepare_save_data to convert the command field to a proper RecordID before writing to SurrealDB. This bidirectional linkage allows the system to query job status directly from the episode record and enables the UI to display real-time generation progress alongside episode metadata.

Stage 4: Querying Status and Results

Clients poll for job status via GET /podcasts/jobs/{job_id}, which routes to PodcastService.get_job_status in api/podcast_service.py. This method calls await get_command_status(job_id) from Surreal-Commands and returns the raw status, result payload, timestamps, and any error messages.

For episode listings, the PodcastEpisode model provides get_job_detail(), which embeds the current job status alongside episode metadata. When the command finishes successfully, the episode fields audio_file, transcript, and outline are populated, and the status transitions to completed.

Stage 5: Retry and Cleanup

If a generation job fails, the system provides a clean retry mechanism via POST /podcasts/episodes/{episode_id}/retry in api/routers/podcasts.py. The endpoint deletes the failed episode record—including any associated audio files—and submits a fresh job via PodcastService.submit_generation_job.

This approach guarantees a clean slate by generating a new Surreal-Commands RecordID rather than attempting to reuse a corrupted job state. The retry logic ensures that transient failures in the podcast generation pipeline can be recovered without manual database intervention.

Implementation Examples

Submitting a Podcast Generation Job

import httpx

payload = {
    "episode_profile": "my‑episode‑profile",
    "speaker_profile": "default‑speaker",
    "episode_name": "TheFutureOfAI",
    "notebook_id": "notebook‑123",
    "briefing_suffix": "Add a short intro."
}

resp = httpx.post("http://localhost:5055/podcasts/generate", json=payload)
job = resp.json()
print(f"Job submitted – ID: {job['job_id']}")

Polling Job Status

import time
import httpx

job_id = "job:open_notebook:generate_podcast:0001"
while True:
    r = httpx.get(f"http://localhost:5055/podcasts/jobs/{job_id}")
    data = r.json()
    print(f"Status: {data['status']}")
    if data["status"] in ("completed", "failed"):
        break
    time.sleep(2)

Retrieving a Completed Episode

import httpx

episode_id = "episode:123456"
r = httpx.get(f"http://localhost:5055/podcasts/episodes/{episode_id}")
episode = r.json()
print("Audio URL:", episode["audio_url"])

Retrying a Failed Episode

import httpx

failed_episode = "episode:bad123"
r = httpx.post(f"http://localhost:5055/podcasts/episodes/{failed_episode}/retry")
print(r.json())

Key Source Files

• api/podcast_service.py: Contains submit_generation_job and get_job_status methods that interface with Surreal-Commands via submit_command and get_command_status.

• commands/podcast_commands.py: Defines the generate_podcast command decorated with @command, responsible for creating the episode record and executing the podcast generation pipeline.

• open_notebook/podcasts/models.py: Implements the PodcastEpisode model with the command field and helper methods get_job_detail and _prepare_save_data for RecordID handling.

• api/routers/podcasts.py: FastAPI router exposing endpoints for job submission (POST /podcasts/generate), status polling (GET /podcasts/jobs/{job_id}), and retry operations (POST /podcasts/episodes/{episode_id}/retry).

Summary

• Open Notebook uses Surreal-Commands as the async job queue backend for podcast generation tasks.
• Jobs are submitted via submit_command("open_notebook", "generate_podcast", args) which returns a RecordID stored on the PodcastEpisode model.
• The episode record is created before the heavy processing begins, ensuring trackability even if the job crashes.
• Status polling uses get_command_status(job_id) to return real-time state including submitted, running, completed, or failed.
• Failed jobs support clean retries by deleting the old record and submitting a fresh Surreal-Commands job.

Frequently Asked Questions

How does the system handle failures during podcast generation?

The generate_podcast_command in commands/podcast_commands.py creates and saves the PodcastEpisode record before starting the heavy audio generation work. This ensures that even if the process crashes, the job identifier remains persisted in SurrealDB. For explicit failures, the retry endpoint deletes the failed episode and audio files, then submits a completely new job via submit_generation_job.

What is the relationship between the PodcastEpisode and the Surreal-Commands job?

The PodcastEpisode model stores the Surreal-Commands RecordID in its command field, established via ensure_record_id(input_data.execution_context.command_id). This creates a bidirectional link where the episode can query its job status through get_job_detail(), which wraps surreal_commands.get_command_status().

Can I check the status of a podcast generation job without querying the episode record?

Yes. The API exposes GET /podcasts/jobs/{job_id} which directly queries Surreal-Commands via PodcastService.get_job_status. This returns the raw job state including status, result data, timestamps, and error messages without requiring access to the episode model.

Why is the command module imported explicitly in the submission code?

The submit_generation_job method in api/podcast_service.py imports commands.podcast_commands to ensure the command is registered and visible to the Surreal-Commands dispatcher before calling submit_command. This explicit import guarantees that the job queue can locate and execute the generate_podcast command when processing the task.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}