# How the Podcast Generation Async Job Queue Works with Surreal-Commands in Open Notebook > Learn how Open Notebook's podcast generation uses Surreal-Commands for async job queues, storing job IDs on PodcastEpisode models and polling status via API endpoints. - Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook) - Tags: internals - Published: 2026-06-14 --- **Open Notebook handles long-running podcast generation by submitting jobs to Surreal-Commands, storing the command RecordID on the PodcastEpisode model, and polling the job status through dedicated API endpoints.** The open-notebook repository implements a robust asynchronous processing system for generating AI podcast episodes. By leveraging Surreal-Commands as a background job queue, the application can handle long-running audio synthesis tasks without blocking HTTP requests, while maintaining durable links between episode records and their background jobs. ## Submitting a Podcast Generation Job The async workflow begins when a client sends a `POST` request to `/podcasts/generate` with a `PodcastGenerationRequest` payload. In [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py), the `submit_generation_job` method validates the requested profiles, builds a command argument dictionary, and imports the command module to make it visible to Surreal-Commands. The service then calls `submit_command("open_notebook", "generate_podcast", args)`, which returns a `RecordID` that gets stored as a string in the `job_id` field: ```python # From api/podcast_service.py - submit_generation_job job_id = await submit_command( "open_notebook", "generate_podcast", args={ "episode_profile": episode_profile, "speaker_profile": speaker_profile, "notebook_id": notebook_id, "content": content, "briefing_suffix": briefing_suffix } ) # job_id is returned as a RecordID string like "job:open_notebook:generate_podcast:0001" ``` This identifier immediately returns to the client, allowing the UI to begin polling for status while the heavy processing happens in the background. ## Executing the Command in Surreal-Commands When Surreal-Commands picks up the job, it executes the decorated function defined in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py). The `@command("generate_podcast", app="open_notebook", ...)` decorator registers this function as the handler for the specified queue. Inside `generate_podcast_command`, the code creates a `PodcastEpisode` model and stores the command's `command_id` on the `command` field before starting the long-running work: ```python # From commands/podcast_commands.py @command("generate_podcast", app="open_notebook", queue="open_notebook") async def generate_podcast_command(input_data: PodcastInput): # Create episode record early so we can track the job even if it crashes episode = PodcastEpisode( name=input_data.episode_name, command=ensure_record_id(input_data.execution_context.command_id), # ... other fields ) await episode.save() # Heavyweight podcast generation happens here # Updates episode.audio_file, episode.transcript, etc. ``` Storing the episode record before the pipeline runs ensures that the job can be tracked and retried even if the process crashes mid-execution. ## Linking Episodes to Job Records The `PodcastEpisode` model in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) maintains a persistent link to its Surreal-Commands job through the `command` field. This field holds a `RecordID` pointing back to the job record, allowing the system to query status at any time. The model overrides `_prepare_save_data` to ensure the command field is always converted to a proper `RecordID` before writing to SurrealDB: ```python # From open_notebook/podcasts/models.py class PodcastEpisode(Model): command: Optional[RecordID] = None # Links to Surreal-Commands job def _prepare_save_data(self, data): if self.command and isinstance(self.command, str): self.command = RecordID(self.command) return super()._prepare_save_data(data) ``` This bi-directional link enables the system to traverse from an episode to its job status, or from a job ID to its associated episode data. ## Querying Job Status and Results Clients poll for completion using `GET /podcasts/jobs/{job_id}`, handled by the `get_podcast_job_status` router in [`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). This endpoint forwards to `PodcastService.get_job_status`, which calls `await get_command_status(job_id)` from Surreal-Commands: ```python # From api/podcast_service.py - get_job_status async def get_job_status(self, job_id: str) -> dict: status = await get_command_status(job_id) return { "status": status.status, # submitted, running, completed, failed "result": status.result, "started_at": status.started_at, "finished_at": status.finished_at, "error_message": status.error_message } ``` The same helper method `episode.get_job_detail()` is used when listing episodes to embed the current job status alongside episode metadata, providing a unified view of both the database record and the queue state. ## Handling Retries and Cleanup If a job fails, the system provides a clean retry mechanism through `POST /podcasts/episodes/{episode_id}/retry`. The router deletes the failed episode record—including any partial audio files—then re-submits a fresh job via `PodcastService.submit_generation_job`: ```python # From api/routers/podcasts.py - retry endpoint @router.post("/podcasts/episodes/{episode_id}/retry") async def retry_episode(episode_id: str): episode = await PodcastEpisode.get(episode_id) if episode.status == "failed": # Clean up failed artifacts if episode.audio_file: os.remove(episode.audio_file) await episode.delete() # Submit new job with same parameters new_job_id = await podcast_service.submit_generation_job( episode_profile=episode.episode_profile, speaker_profile=episode.speaker_profile, # ... other parameters ) return {"job_id": new_job_id, "status": "submitted"} ``` This guarantees a clean slate with a new Surreal-Commands `RecordID`, preventing failed state pollution from affecting the retry attempt. ## Summary - **Job Submission**: The `submit_generation_job` method in [`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py) queues work via `submit_command` and stores the returned `RecordID` as a string. - **Command Execution**: The `generate_podcast_command` in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) creates the episode record upfront and links it to the Surreal-Commands job ID before running the heavy pipeline. - **Status Tracking**: The `command` field on `PodcastEpisode` models (in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)) maintains a durable reference to the background job, enabling real-time status queries via `get_command_status`. - **Retry Logic**: Failed episodes can be retried through the dedicated endpoint, which deletes the old record and submits a fresh job to ensure clean state. ## Frequently Asked Questions ### How does Open Notebook track the status of a podcast generation job? Open Notebook stores the Surreal-Commands `RecordID` on the `PodcastEpisode.command` field during the initial command execution. The API exposes `GET /podcasts/jobs/{job_id}`, which calls `get_command_status` from the Surreal-Commands library to return the current status, timestamps, and any error messages. ### What happens if a podcast generation job fails during execution? If the job fails, the episode record remains in the database with a failed status. Users can trigger a retry via `POST /podcasts/episodes/{episode_id}/retry`, which deletes the failed record (including any partial audio files) and submits a completely new job to Surreal-Commands with a fresh `RecordID`. ### Where is the Surreal-Commands job ID stored in the database? The job ID is stored in the `command` field of the `PodcastEpisode` model, defined in [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py). This field is typed as `Optional[RecordID]` and is converted to a proper RecordID format during the model's `_prepare_save_data` lifecycle method before persistence. ### How does the system prevent blocking HTTP requests during podcast generation? The system uses Surreal-Commands as an asynchronous job queue. When a client requests podcast generation, the API immediately returns a job identifier and delegates the heavy audio synthesis work to the background command. Clients then poll the job status endpoint to receive updates without holding open HTTP connections.