Job Queue Architecture for Async Podcast Generation Using Surreal-Commands in Open Notebook

Open Notebook leverages Surreal-Commands as a distributed task queue to handle podcast generation asynchronously, storing the command RecordID on the PodcastEpisode model to enable status polling, result retrieval, and clean retry operations.

Open Notebook implements a robust job queue architecture for async podcast generation by integrating Surreal-Commands, a distributed task processing system. This design decouples the computationally intensive audio synthesis pipeline from the FastAPI request layer, allowing users to submit generation requests and monitor progress through persistent job identifiers stored on PodcastEpisode records.

Submitting Generation Jobs to the Queue

The async workflow begins when a client sends a POST request to /podcasts/generate. In api/podcast_service.py, the PodcastService.submit_generation_job method validates the requested profiles and constructs a command argument dictionary. It dynamically imports commands.podcast_commands to ensure the command is visible to the Surreal-Commands registry, then invokes submit_command("open_notebook", "generate_podcast", args).

This function returns a SurrealDB RecordID that is immediately stored as a string in the job_id field and returned to the client. This establishes the initial link between the API request and the background job queue, allowing the client to begin polling for status while the heavy processing moves to the background worker.

Command Execution and Episode Persistence

When Surreal-Commands picks up the queued task, it executes the generate_podcast_command function defined in commands/podcast_commands.py. The function is decorated with @command("generate_podcast", app="open_notebook", …) to register it with the command framework.

Inside the command execution, the code creates a PodcastEpisode model and stores the Surreal-Commands command_id on the episode's command field using ensure_record_id(input_data.execution_context.command_id). Crucially, the episode record is persisted before the heavyweight podcast-creator pipeline begins processing. This ensures the job remains trackable even if the process crashes during the long-running audio generation phase.

Persisting the Job Link with RecordID

The connection between the episode and its background job relies on the command field in open_notebook/podcasts/models.py. This field stores a RecordID that points back to the Surreal-Commands job record.

To maintain data integrity, the PodcastEpisode model overrides _prepare_save_data to convert the command field to a proper RecordID instance before writing to SurrealDB. This guarantees that the foreign key relationship remains valid across queries and status checks, preventing serialization mismatches when the episode is retrieved or updated.

Monitoring Job Status and Results

Clients monitor generation progress by calling GET /podcasts/jobs/{job_id}, handled by the get_podcast_job_status router in api/routers/podcasts.py. This endpoint delegates to PodcastService.get_job_status, which simply awaits get_command_status(job_id) from the Surreal-Commands library.

The response includes the raw status (submitted, running, completed, failed), result payload, timestamps, and error messages. When listing episodes, the same get_job_detail() helper method embeds current job status alongside episode metadata, allowing the UI to display real-time generation progress without requiring separate API calls.

Retry and Cleanup Mechanisms

If a job fails, the system provides a clean retry path through POST /podcasts/episodes/{episode_id}/retry. This endpoint, defined in api/routers/podcasts.py, deletes the failed episode record—including any associated audio files—from storage, then re-submits a fresh generation request via PodcastService.submit_generation_job.

This approach guarantees a clean slate by generating a new Surreal-Commands RecordID rather than attempting to resurrect a corrupted or stale job state. This ensures deterministic retry behavior and prevents contamination from previous failed attempts.

Implementation Examples

Submitting a Podcast Generation Job

import httpx

payload = {
    "episode_profile": "my-episode-profile",
    "speaker_profile": "default-speaker",
    "episode_name": "TheFutureOfAI",
    "notebook_id": "notebook-123",
    "briefing_suffix": "Add a short intro."
}

resp = httpx.post("http://localhost:5055/podcasts/generate", json=payload)
job = resp.json()
print(f"Job submitted – ID: {job['job_id']}")

Polling for Job Completion

import time, httpx

job_id = "job:open_notebook:generate_podcast:0001"
while True:
    r = httpx.get(f"http://localhost:5055/podcasts/jobs/{job_id}")
    data = r.json()
    print(f"Status: {data['status']}")
    if data["status"] in ("completed", "failed"):
        break
    time.sleep(2)

Retrying a Failed Episode

import httpx

failed_episode = "episode:bad123"
r = httpx.post(f"http://localhost:5055/podcasts/episodes/{failed_episode}/retry")
print(r.json())

Summary

• Surreal-Commands Integration: Open Notebook uses submit_command and get_command_status from the Surreal-Commands library to queue and monitor podcast generation tasks asynchronously.
• Early Persistence: The PodcastEpisode record is created and saved before audio processing begins, storing the command_id to maintain job visibility even during process crashes.
• Status Polling: The GET /podcasts/jobs/{job_id} endpoint provides real-time access to job states including submitted, running, completed, and failed.
• Clean Retries: Failed jobs trigger deletion of the old episode and submission of a fresh Surreal-Commands job, ensuring no stale state persists.
• RecordID Management: The command field uses _prepare_save_data to ensure proper RecordID serialization when persisting to SurrealDB.

Frequently Asked Questions

How does Open Notebook link a PodcastEpisode to its Surreal-Commands job?

The PodcastEpisode model stores the Surreal-Commands RecordID in its command field. When the generate_podcast_command executes in commands/podcast_commands.py, it captures the command_id from the execution context and stores it using ensure_record_id(). The model's _prepare_save_data method ensures this value is properly serialized as a RecordID when saving to SurrealDB.

What happens if the podcast generation process crashes midway?

Because the PodcastEpisode record is persisted before the long-running audio synthesis begins, the job remains trackable even if the process crashes. The episode exists in the database with its command field populated, allowing the system to query the job status (which would show as failed or remain running depending on the crash timing) and enabling users to retry via the dedicated endpoint.

How does the retry mechanism ensure clean job state?

When retrying a failed episode via POST /podcasts/episodes/{episode_id}/retry, the endpoint first deletes the failed episode record and any associated audio files from storage. It then calls PodcastService.submit_generation_job to create a completely new Surreal-Commands job with a fresh RecordID, preventing contamination from the previous failed attempt and ensuring deterministic execution.

Can I check the status of a podcast generation job without querying the episode record?

Yes. The GET /podcasts/jobs/{job_id} endpoint allows direct status queries by calling get_command_status(job_id) from the Surreal-Commands library. This returns the raw job status, timestamps, and error messages without requiring you to fetch the full episode record, though the episode list view also embeds this status via the get_job_detail() helper method.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}