Async Job Queue Architecture for Podcast Generation and Status Tracking in Open‑Notebook

Open‑Notebook leverages the surreal‑commands library to decouple API requests from CPU‑intensive audio synthesis, using a three‑layer architecture that submits jobs to SurrealDB, processes them in background workers, and exposes real‑time status endpoints for polling.

The lfnovo/open‑notebook repository implements a robust asynchronous pipeline for creating AI‑generated podcasts. By utilizing an async job queue architecture for podcast generation and status tracking, the application ensures that long‑running text‑to‑speech operations never block the HTTP API, allowing users to queue episodes and monitor progress through dedicated status endpoints.

Three‑Layer Async Architecture

The podcast generation flow is divided into three distinct layers, each handled by specific modules in the codebase.

Service Layer – Job Submission and Validation

Located in api/podcast_service.py, the PodcastService class handles request validation and job enqueueing. The submit_generation_job method performs six critical steps before returning a job identifier:

1. Profile validation – confirms the requested EpisodeProfile and SpeakerProfile exist in SurrealDB.
2. Content resolution – extracts text from a notebook (via notebook_id) or uses explicit content arguments.
3. Command preparation – builds a command_args dictionary containing episode name, profiles, content, and optional briefing suffixes.
4. Dynamic import – forces import of commands.podcast_commands to register the generate_podcast command in the surreal‑commands registry.
5. Job submission – calls surreal_commands.submit_command("open_notebook", "generate_podcast", command_args), which returns a SurrealDB RecordID.
6. Return – converts the RecordID to a string (job_id) for the client.

Command Layer – Background Execution

The heavy lifting occurs in commands/podcast_commands.py inside the generate_podcast_command function, decorated with @command("generate_podcast", ...) to register it with the queue worker. The command executes eight steps asynchronously:

1. Load profiles – fetches episode and speaker configuration from SurrealDB.
2. Model resolution – maps profile‑specified LLM/TTS names to concrete provider credentials using _resolve_model_config.
3. Create episode record – instantiates a PodcastEpisode record linked to the command ID for correlation.
4. Configure engine – injects resolved model configs into the podcast‑creator library.
5. Prepare output directory – generates a UUID‑named folder under DATA_FOLDER/podcasts/episodes/ (defined in open_notebook/config.py).
6. Generate audio – awaits create_podcast(...) to produce audio, transcript, and outline data.
7. Persist results – updates the PodcastEpisode record with audio_file, transcript, and outline fields.
8. Return output – produces a PodcastGenerationOutput object containing success, episode_id, file paths, and processing metadata.

If any step fails, the command raises an exception, and the job status is automatically set to error with an explanatory message.

Status Layer – Progress Monitoring

Clients query job state through PodcastService.get_job_status(job_id), which wraps surreal_commands.get_command_status. The returned payload includes:

• status – queued, running, completed, or error
• result – serialized PodcastGenerationOutput when finished
• error_message – populated on failure
• created and updated timestamps

The FastAPI router in api/routers/podcasts.py exposes this via HTTP endpoints, enabling front‑end polling for live progress bars.

Submitting a Podcast Generation Job

To queue a new episode, call the service layer with profile names and optional notebook content:

from api.podcast_service import PodcastService

job_id = await PodcastService.submit_generation_job(
    episode_profile_name="TechTalk",
    speaker_profile_name="DefaultSpeaker",
    episode_name="AI in 2026",
    notebook_id="notebook:12345",   # optional – pulls notebook content

    briefing_suffix="Add a short intro about Open‑Notebook",
)

print(f"Job queued with ID: {job_id}")

Alternatively, trigger the flow via the REST API:

curl -X POST http://localhost:5055/podcasts/generate \
  -H "Content-Type: application/json" \
  -d '{
        "episode_profile": "TechTalk",
        "speaker_profile": "DefaultSpeaker",
        "episode_name": "AI in 2026",
        "notebook_id": "notebook:12345",
        "briefing_suffix": "Add a short intro about Open‑Notebook"
      }'

Processing Inside the Background Worker

Once dequeued, the worker executes generate_podcast_command. The function ensures filesystem safety by creating a unique directory under DATA_FOLDER/podcasts/episodes/<uuid>/ before invoking the podcast‑creator library. After successful generation, the audio file path and transcript are committed to the PodcastEpisode record in SurrealDB, making them immediately queryable via the service layer.

Tracking Job Status and Retrieving Results

Poll the status endpoint to monitor the asynchronous operation:

curl http://localhost:5055/podcasts/status/<job_id>

Once status returns "completed", extract the episode details:

from api.podcast_service import PodcastService

status = await PodcastService.get_job_status(job_id)
if status["status"] == "completed":
    episode_id = status["result"]["episode_id"]
    episode = await PodcastService.get_episode(episode_id)
    print("Audio file:", episode.audio_file)

Summary

• Decoupled architecture – The surreal‑commands library manages job state in SurrealDB, separating HTTP request handling from long‑running audio synthesis.
• Service layer – PodcastService.submit_generation_job in api/podcast_service.py validates profiles and notebook content before returning a job ID.
• Background execution – generate_podcast_command in commands/podcast_commands.py runs the heavy lifting, storing results in PodcastEpisode records under DATA_FOLDER/podcasts/episodes/.
• Status polling – PodcastService.get_job_status exposes real‑time states (queued, running, completed, error) and final output data.

Frequently Asked Questions

What job queue system does Open‑Notebook use for podcast generation?

Open‑Notebook uses surreal‑commands, a job queue system built on SurrealDB. It stores task metadata in the commands table and dispatches work to background workers that execute registered Python functions like generate_podcast_command.

How does the system validate requests before queuing a job?

The submit_generation_job method in api/podcast_service.py validates that the requested EpisodeProfile and SpeakerProfile exist in the database and resolves notebook content before invoking surreal_commands.submit_command. This ensures only valid configuration reaches the background worker.

Where are generated podcast files stored?

Audio files, transcripts, and outlines are saved in a UUID‑named folder under DATA_FOLDER/podcasts/episodes/, as defined in open_notebook/config.py. The relative path is persisted in the audio_file field of the PodcastEpisode record stored in SurrealDB.

What status values can a podcast generation job return?

According to the surreal_commands.get_command_status implementation, jobs return statuses including queued, running, completed, or error. Completed jobs include a result field containing the PodcastGenerationOutput, while failed jobs provide an error_message detailing the cause.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}