How the Podcast Generation Async Job Queue Works with Surreal-Commands in Open Notebook

Open Notebook handles long-running podcast generation by submitting jobs to Surreal-Commands, storing the command RecordID on the PodcastEpisode model, and polling the job status through dedicated API endpoints.

The open-notebook repository implements a robust asynchronous processing system for generating AI podcast episodes. By leveraging Surreal-Commands as a background job queue, the application can handle long-running audio synthesis tasks without blocking HTTP requests, while maintaining durable links between episode records and their background jobs.

Submitting a Podcast Generation Job

The async workflow begins when a client sends a POST request to /podcasts/generate with a PodcastGenerationRequest payload. In api/podcast_service.py, the submit_generation_job method validates the requested profiles, builds a command argument dictionary, and imports the command module to make it visible to Surreal-Commands.

The service then calls submit_command("open_notebook", "generate_podcast", args), which returns a RecordID that gets stored as a string in the job_id field:

# From api/podcast_service.py - submit_generation_job

job_id = await submit_command(
    "open_notebook", 
    "generate_podcast", 
    args={
        "episode_profile": episode_profile,
        "speaker_profile": speaker_profile,
        "notebook_id": notebook_id,
        "content": content,
        "briefing_suffix": briefing_suffix
    }
)

# job_id is returned as a RecordID string like "job:open_notebook:generate_podcast:0001"

This identifier immediately returns to the client, allowing the UI to begin polling for status while the heavy processing happens in the background.

Executing the Command in Surreal-Commands

When Surreal-Commands picks up the job, it executes the decorated function defined in commands/podcast_commands.py. The @command("generate_podcast", app="open_notebook", ...) decorator registers this function as the handler for the specified queue.

Inside generate_podcast_command, the code creates a PodcastEpisode model and stores the command's command_id on the command field before starting the long-running work:

# From commands/podcast_commands.py

@command("generate_podcast", app="open_notebook", queue="open_notebook")
async def generate_podcast_command(input_data: PodcastInput):
    # Create episode record early so we can track the job even if it crashes

    episode = PodcastEpisode(
        name=input_data.episode_name,
        command=ensure_record_id(input_data.execution_context.command_id),
        # ... other fields

    )
    await episode.save()
    
    # Heavyweight podcast generation happens here

    # Updates episode.audio_file, episode.transcript, etc.

Storing the episode record before the pipeline runs ensures that the job can be tracked and retried even if the process crashes mid-execution.

Linking Episodes to Job Records

The PodcastEpisode model in open_notebook/podcasts/models.py maintains a persistent link to its Surreal-Commands job through the command field. This field holds a RecordID pointing back to the job record, allowing the system to query status at any time.

The model overrides _prepare_save_data to ensure the command field is always converted to a proper RecordID before writing to SurrealDB:

# From open_notebook/podcasts/models.py

class PodcastEpisode(Model):
    command: Optional[RecordID] = None  # Links to Surreal-Commands job

    
    def _prepare_save_data(self, data):
        if self.command and isinstance(self.command, str):
            self.command = RecordID(self.command)
        return super()._prepare_save_data(data)

This bi-directional link enables the system to traverse from an episode to its job status, or from a job ID to its associated episode data.

Querying Job Status and Results

Clients poll for completion using GET /podcasts/jobs/{job_id}, handled by the get_podcast_job_status router in api/routers/podcasts.py. This endpoint forwards to PodcastService.get_job_status, which calls await get_command_status(job_id) from Surreal-Commands:

# From api/podcast_service.py - get_job_status

async def get_job_status(self, job_id: str) -> dict:
    status = await get_command_status(job_id)
    return {
        "status": status.status,  # submitted, running, completed, failed

        "result": status.result,
        "started_at": status.started_at,
        "finished_at": status.finished_at,
        "error_message": status.error_message
    }

The same helper method episode.get_job_detail() is used when listing episodes to embed the current job status alongside episode metadata, providing a unified view of both the database record and the queue state.

Handling Retries and Cleanup

If a job fails, the system provides a clean retry mechanism through POST /podcasts/episodes/{episode_id}/retry. The router deletes the failed episode record—including any partial audio files—then re-submits a fresh job via PodcastService.submit_generation_job:

# From api/routers/podcasts.py - retry endpoint

@router.post("/podcasts/episodes/{episode_id}/retry")
async def retry_episode(episode_id: str):
    episode = await PodcastEpisode.get(episode_id)
    if episode.status == "failed":
        # Clean up failed artifacts

        if episode.audio_file:
            os.remove(episode.audio_file)
        await episode.delete()
        
        # Submit new job with same parameters

        new_job_id = await podcast_service.submit_generation_job(
            episode_profile=episode.episode_profile,
            speaker_profile=episode.speaker_profile,
            # ... other parameters

        )
        return {"job_id": new_job_id, "status": "submitted"}

This guarantees a clean slate with a new Surreal-Commands RecordID, preventing failed state pollution from affecting the retry attempt.

Summary

• Job Submission: The submit_generation_job method in api/podcast_service.py queues work via submit_command and stores the returned RecordID as a string.
• Command Execution: The generate_podcast_command in commands/podcast_commands.py creates the episode record upfront and links it to the Surreal-Commands job ID before running the heavy pipeline.
• Status Tracking: The command field on PodcastEpisode models (in open_notebook/podcasts/models.py) maintains a durable reference to the background job, enabling real-time status queries via get_command_status.
• Retry Logic: Failed episodes can be retried through the dedicated endpoint, which deletes the old record and submits a fresh job to ensure clean state.

Frequently Asked Questions

How does Open Notebook track the status of a podcast generation job?

Open Notebook stores the Surreal-Commands RecordID on the PodcastEpisode.command field during the initial command execution. The API exposes GET /podcasts/jobs/{job_id}, which calls get_command_status from the Surreal-Commands library to return the current status, timestamps, and any error messages.

What happens if a podcast generation job fails during execution?

If the job fails, the episode record remains in the database with a failed status. Users can trigger a retry via POST /podcasts/episodes/{episode_id}/retry, which deletes the failed record (including any partial audio files) and submits a completely new job to Surreal-Commands with a fresh RecordID.

Where is the Surreal-Commands job ID stored in the database?

The job ID is stored in the command field of the PodcastEpisode model, defined in open_notebook/podcasts/models.py. This field is typed as Optional[RecordID] and is converted to a proper RecordID format during the model's _prepare_save_data lifecycle method before persistence.

How does the system prevent blocking HTTP requests during podcast generation?

The system uses Surreal-Commands as an asynchronous job queue. When a client requests podcast generation, the API immediately returns a job identifier and delegates the heavy audio synthesis work to the background command. Clients then poll the job status endpoint to receive updates without holding open HTTP connections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}