# How Source Vectorization Works as a Fire-and-Forget Async Operation in Open Notebook

> Discover how Open Notebook uses fire-and-forget async operations for source vectorization. Learn about background embedding generation chunking and database inserts with automatic retries.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-21

---

**Open Notebook implements source vectorization as an asynchronous job using the surreal-commands queue, allowing the API to return immediately while background workers handle embedding generation, chunking, and database inserts with automatic retry logic.**

Open Notebook transforms raw source text into vector embeddings for semantic search without blocking HTTP requests. Instead of processing embeddings synchronously—which would exhaust connection pools and degrade API performance—the system leverages the surreal-commands job queue to offload intensive computation according to the source code in `lfnovo/open-notebook`.

## The Fire-and-Forget Architecture

The vectorization workflow follows a strict fire-and-forget pattern that separates job submission from execution. This design ensures that saving a source and initiating embedding generation are atomic operations from the client's perspective, while the actual computational work happens asynchronously.

### Triggering the Vectorization Job

When a source is saved via `save_source` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), the system checks the `embed` flag at lines 777-795. If enabled, it invokes `Source.vectorize()`, which initiates the async workflow without waiting for the embedding process to complete.

### Non-Blocking Job Submission

The `Source.vectorize()` method constructs a command request and calls `submit_command` from the surreal-commands library:

```python
command_id = submit_command(
    "open_notebook",          # app name

    "embed_source",          # command name

    {"source_id": str(self.id)},
)

```

As implemented in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) at lines 795-805, this call returns a command ID immediately. The API endpoint can finish its response while the actual embedding work queues in the background, completely independent of the original HTTP request.

### Background Worker Processing

The `embed_source` command defined in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) at lines 65-78 executes the heavy lifting. When the worker process picks up the job, it:

- Loads the source record and deletes any existing embeddings
- Detects content type and chunks the text appropriately
- Generates vector embeddings for all chunks
- Bulk-inserts `source_embedding` records into the database

The implementation includes a robust retry strategy with **5 attempts** and exponential-jitter backoff to handle transient failures without requiring additional client interaction.

## Implementation Details

### Source.vectorize() Method

Located in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), this method bridges the domain model and the command queue. It packages the source ID into a command payload and delegates to surreal-commands, returning only the command ID to callers for optional status tracking. This method never awaits the actual embedding computation, preserving the fire-and-forget semantics.

### The embed_source Command Handler

The actual embedding logic resides in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py). This worker-side implementation handles the complete pipeline from raw text to searchable vectors. Unlike the API layer, this component runs outside the request/response cycle, allowing it to process large documents, handle chunking complexity, and manage database transactions without impacting HTTP response times.

### Command Status Monitoring

Clients that need to confirm completion can query the commands API endpoint defined in [`api/command_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/command_service.py) at lines 46-64. By issuing `GET /commands/{job_id}`, applications retrieve the current status—running, completed, or failed—of the embedding job without blocking the original request.

## Practical Code Examples

**Submitting a source and triggering background embedding:**

```python

# In a FastAPI route (api/routers/sources.py)

@router.post("/sources")
async def create_source(payload: SourceCreate):
    # Content extraction happens in the source graph

    source = await source_graph.ainvoke(payload)     # runs content_process → save_source

    # The embed flag was true, so source.vectorize() already ran

    return {"source_id": source.id, "embedding_job": source.id}

```

**Polling the embedding job status:**

```python
import httpx

job_id = "...command id returned by vectorize..."
resp = httpx.get(f"http://localhost:5055/commands/{job_id}")
status = resp.json()["status"]   # e.g. "completed", "failed", "running"

```

**Directly invoking the command service:**

```python
from api.command_service import CommandService

job_id = await CommandService.submit_command_job(
    module_name="open_notebook",
    command_name="embed_source",
    command_args={"source_id": "source:123"},
)
print(f"Embedding job queued: {job_id}")

```

## Why Fire-and-Forget Matters

**Scalability:** Large documents are chunked and embedded in the background, preventing API connection pool exhaustion and maintaining throughput under load.

**Reliability:** The command system handles retries, exponential backoff, and persistent job IDs, making the embedding process resilient to transient failures without client-side complexity.

**Responsiveness:** UI code can display the source immediately while the embeddings become available later, creating a snappy user experience even with computationally expensive vectorization.

## Summary

- **Source.vectorize()** in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) initiates embedding by submitting a command to surreal-commands and returning immediately with a job ID.
- **surreal-commands** provides the queue infrastructure that decouples job submission from execution, enabling true fire-and-forget semantics.
- **embed_source** in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) handles the worker-side processing including chunking, embedding generation, and bulk inserts with 5-attempt retry logic.
- **Command status endpoints** in [`api/command_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/command_service.py) allow asynchronous polling for clients that need completion confirmation.
- The architecture prevents HTTP request blocking while ensuring reliable, retryable background processing of source documents.

## Frequently Asked Questions

### How does Open Notebook prevent blocking during source embedding?

Open Notebook uses the surreal-commands job queue to offload embedding work. When `Source.vectorize()` calls `submit_command()`, it receives a command ID immediately and returns control to the API endpoint. The actual vectorization—chunking, embedding generation, and database inserts—runs in a separate worker process defined in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py), completely independent of the HTTP request lifecycle.

### Where is the embedding command logic implemented?

The background processing logic is implemented in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) at lines 65-78. This file defines the `embed_source` command that workers execute, which includes loading the source, deleting old embeddings, detecting content types, chunking text, generating vectors, and bulk-inserting `source_embedding` records with exponential-jitter retry logic.

### How can I check if a source's embeddings are ready?

Query the command status endpoint at `GET /commands/{job_id}` as implemented in [`api/command_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/command_service.py) (lines 46-64). The `Source.vectorize()` method returns this job ID when submitting the embedding request. The endpoint returns the current status—running, completed, or failed—allowing your application to poll asynchronously or trigger downstream operations once embeddings are available.

### What happens if the embedding job fails?

The `embed_source` command implements a robust retry strategy with 5 attempts using exponential-jitter backoff. Transient failures are automatically retried without client intervention. If all retries exhaust, the command status reflects the failure state, which you can detect via the commands API endpoint to implement application-level error handling or alerting.