# How the Async Job Queue Handles Podcast Generation and Status Polling in Open Notebook

> Discover how Open Notebook's async job queue uses SurrealDB for podcast generation and status polling. Submit jobs via REST API and get background LLM/TTS updates efficiently.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-15

---

**Open Notebook leverages SurrealDB and the `surreal-commands` library to process podcast generation asynchronously, allowing clients to submit jobs via REST API and poll for status updates while heavy LLM and TTS operations run in the background.**

The `lfnovo/open-notebook` repository implements a robust asynchronous architecture for generating AI podcasts using SurrealDB as both the database and job queue backend. By delegating expensive text-to-speech and language model operations to background workers, the system maintains responsive HTTP endpoints while handling complex multi-step audio synthesis workflows.

## Submitting a Podcast Generation Job

Podcast generation begins at the REST endpoint **`POST /podcasts/generate`**, defined in [[`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py). When a client submits a request, the service layer invokes `PodcastService.submit_generation_job` in [[`api/podcast_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py)](https://github.com/lfnovo/open-notebook/blob/main/api/podcast_service.py#L45-L99).

This method performs several validation steps before queueing:

1. **Validates** the requested episode profile and speaker profiles against stored configurations
2. **Resolves** source content from either a referenced notebook or directly supplied text
3. **Registers** the job via `submit_command` from the `surreal-commands` library, which creates a persistent command record in SurrealDB

The SurrealDB record ID returned by `submit_command` becomes the **job ID** (e.g., `command:001abcdef`) that clients use for subsequent status polling. The HTTP response returns immediately with this ID and a `status: "submitted"` field, without waiting for the actual audio generation to begin.

```http
POST /podcasts/generate
Content-Type: application/json

{
  "episode_profile": "TechTalk",
  "speaker_profile": "DefaultSpeaker",
  "episode_name": "AI Trends 2024",
  "notebook_id": "notebook:12345"
}

```

```json
{
  "job_id": "command:001abcdef",
  "status": "submitted",
  "message": "Podcast generation started for episode 'AI Trends 2024'",
  "episode_profile": "TechTalk",
  "episode_name": "AI Trends 2024"
}

```

## Background Command Execution

Once queued, SurrealDB's command engine delegates the work to the function decorated with `@command("generate_podcast", app="open_notebook")` in [[`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py)](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py#L69-L85). This decorator registers the handler with the `surreal-commands` worker pool.

The command execution flow involves:

- **Receiving** a `PodcastGenerationInput` model containing resolved episode parameters
- **Loading** episode profiles and speaker configurations from SurrealDB
- **Resolving** language model configurations for the synthesis pipeline
- **Creating** a UUID-based output directory for audio file storage
- **Invoking** the third-party **`podcast-creator`** library to synthesize audio, generate transcripts, and create content outlines

During execution, the command periodically updates its status in SurrealDB through the `surreal-commands` framework, transitioning from `pending` to `running` and eventually to either `completed` or `failed`.

## Polling Job Status and Retrieval

Clients monitor progress through the **`GET /podcasts/jobs/{job_id}`** endpoint implemented in [[`api/routers/podcasts.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py)](https://github.com/lfnovo/open-notebook/blob/main/api/routers/podcasts.py#L71-L79). This route calls `PodcastService.get_job_status`, which utilizes the `get_command_status` helper from `surreal-commands` to query the current state from SurrealDB.

The endpoint returns a JSON payload containing:

- **Current state**: One of `pending`, `running`, `completed`, or `failed`
- **Progress indicators**: Optional completion percentage (e.g., `0.45` for 45%)
- **Result data**: When `status` is `completed`, includes the generated `episode_id`
- **Timestamps**: `created` and `updated` fields tracking the job lifecycle
- **Error details**: `error_message` field when status is `failed`

```http
GET /podcasts/jobs/command:001abcdef

```

```json
{
  "job_id": "command:001abcdef",
  "status": "running",
  "result": null,
  "error_message": null,
  "created": "2026-06-05T12:34:56Z",
  "updated": "2026-06-05T12:35:10Z",
  "progress": 0.45
}

```

When the status becomes `completed`, the `result` field contains the `episode_id`, which can then be fetched via `GET /podcasts/episodes/{episode_id}` to retrieve the full episode details including audio file paths and transcripts.

## Persisting Results and Retry Logic

Upon successful completion, the `generate_podcast` command creates a `PodcastEpisode` record in [[`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py), storing the audio file path, transcript, outline, and linking the episode to the original command ID via `ensure_record_id`. This linkage enables the system to correlate episodes with their generation jobs for status tracking.

If a job fails or produces unsatisfactory results, the system provides a retry mechanism through **`POST /podcasts/episodes/{episode_id}/retry`**. This endpoint:

1. Deletes the broken episode record from SurrealDB
2. Removes any partial audio files from storage
3. Re-submits a fresh generation job using the original episode and speaker profiles

This ensures that transient failures in the `podcast-creator` library or underlying TTS services do not permanently block content creation.

## Summary

- **Job submission** occurs via `POST /podcasts/generate`, which uses `PodcastService.submit_generation_job` and `surreal-commands.submit_command` to create a SurrealDB command record without blocking the HTTP response.
- **Background processing** happens through the `@command("generate_podcast")` decorator in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py), which invokes the `podcast-creator` library to synthesize audio and transcripts.
- **Status polling** uses `GET /podcasts/jobs/{job_id}` and `get_command_status` to retrieve real-time state updates including progress percentages and completion status.
- **Result persistence** links generated `PodcastEpisode` records to their original command IDs, enabling correlation between jobs and final audio outputs.
- **Retry functionality** allows clients to re-queue failed episodes by deleting partial results and re-submitting with identical parameters.

## Frequently Asked Questions

### How does the async job queue maintain state between the REST API and background workers?

The system uses SurrealDB as the central state store through the `surreal-commands` library. When `submit_command` creates a job record in SurrealDB, both the FastAPI endpoints and background workers read from and write to this same database record. The command status (pending, running, completed, failed) persists in SurrealDB, allowing the `get_command_status` function to retrieve current state regardless of which worker process is handling the job.

### What happens if the podcast generation process crashes or the server restarts?

Because SurrealDB persists the command records and the `surreal-commands` library implements durable job queuing, incomplete jobs survive server restarts. When the worker pool restarts, it automatically picks up any jobs with `pending` or `running` status that were interrupted. The `generate_podcast` command in [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) operates idempotently where possible, using UUID-based output directories to prevent file collisions during retries.

### Can clients receive real-time updates without polling the job status endpoint?

The current implementation described in the source code relies on polling via `GET /podcasts/jobs/{job_id}`, with clients responsible for implementing appropriate polling intervals. The SurrealDB-based queue architecture in `surreal-commands` supports the infrastructure for future WebSocket or webhook implementations, but the provided code shows a traditional request-response polling pattern for status updates.

### How are speaker profiles and episode configurations validated before queuing?

The `PodcastService.submit_generation_job` method validates inputs before calling `submit_command`. It loads and verifies the `EpisodeProfile` and `SpeakerProfile` models from [[`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py)](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) against SurrealDB records, ensuring that referenced profiles exist and that the source content (notebook or direct text) can be resolved. This prevents invalid jobs from entering the queue and consuming worker resources.