How Source Vectorization Works as a Fire-and-Forget Async Operation in Open Notebook

Open Notebook implements source vectorization as an asynchronous job using the surreal-commands queue, allowing the API to return immediately while background workers handle embedding generation, chunking, and database inserts with automatic retry logic.

Open Notebook transforms raw source text into vector embeddings for semantic search without blocking HTTP requests. Instead of processing embeddings synchronously—which would exhaust connection pools and degrade API performance—the system leverages the surreal-commands job queue to offload intensive computation according to the source code in lfnovo/open-notebook.

The Fire-and-Forget Architecture

The vectorization workflow follows a strict fire-and-forget pattern that separates job submission from execution. This design ensures that saving a source and initiating embedding generation are atomic operations from the client's perspective, while the actual computational work happens asynchronously.

Triggering the Vectorization Job

When a source is saved via save_source in open_notebook/domain/notebook.py, the system checks the embed flag at lines 777-795. If enabled, it invokes Source.vectorize(), which initiates the async workflow without waiting for the embedding process to complete.

Non-Blocking Job Submission

The Source.vectorize() method constructs a command request and calls submit_command from the surreal-commands library:

command_id = submit_command(
    "open_notebook",          # app name

    "embed_source",          # command name

    {"source_id": str(self.id)},
)

As implemented in open_notebook/domain/notebook.py at lines 795-805, this call returns a command ID immediately. The API endpoint can finish its response while the actual embedding work queues in the background, completely independent of the original HTTP request.

Background Worker Processing

The embed_source command defined in commands/embedding_commands.py at lines 65-78 executes the heavy lifting. When the worker process picks up the job, it:

• Loads the source record and deletes any existing embeddings
• Detects content type and chunks the text appropriately
• Generates vector embeddings for all chunks
• Bulk-inserts source_embedding records into the database

The implementation includes a robust retry strategy with 5 attempts and exponential-jitter backoff to handle transient failures without requiring additional client interaction.

Implementation Details

Source.vectorize() Method

Located in open_notebook/domain/notebook.py, this method bridges the domain model and the command queue. It packages the source ID into a command payload and delegates to surreal-commands, returning only the command ID to callers for optional status tracking. This method never awaits the actual embedding computation, preserving the fire-and-forget semantics.

The embed_source Command Handler

The actual embedding logic resides in commands/embedding_commands.py. This worker-side implementation handles the complete pipeline from raw text to searchable vectors. Unlike the API layer, this component runs outside the request/response cycle, allowing it to process large documents, handle chunking complexity, and manage database transactions without impacting HTTP response times.

Command Status Monitoring

Clients that need to confirm completion can query the commands API endpoint defined in api/command_service.py at lines 46-64. By issuing GET /commands/{job_id}, applications retrieve the current status—running, completed, or failed—of the embedding job without blocking the original request.

Practical Code Examples

Submitting a source and triggering background embedding:

# In a FastAPI route (api/routers/sources.py)

@router.post("/sources")
async def create_source(payload: SourceCreate):
    # Content extraction happens in the source graph

    source = await source_graph.ainvoke(payload)     # runs content_process → save_source

    # The embed flag was true, so source.vectorize() already ran

    return {"source_id": source.id, "embedding_job": source.id}

Polling the embedding job status:

import httpx

job_id = "...command id returned by vectorize..."
resp = httpx.get(f"http://localhost:5055/commands/{job_id}")
status = resp.json()["status"]   # e.g. "completed", "failed", "running"

Directly invoking the command service:

from api.command_service import CommandService

job_id = await CommandService.submit_command_job(
    module_name="open_notebook",
    command_name="embed_source",
    command_args={"source_id": "source:123"},
)
print(f"Embedding job queued: {job_id}")

Why Fire-and-Forget Matters

Scalability: Large documents are chunked and embedded in the background, preventing API connection pool exhaustion and maintaining throughput under load.

Reliability: The command system handles retries, exponential backoff, and persistent job IDs, making the embedding process resilient to transient failures without client-side complexity.

Responsiveness: UI code can display the source immediately while the embeddings become available later, creating a snappy user experience even with computationally expensive vectorization.

Summary

• Source.vectorize() in open_notebook/domain/notebook.py initiates embedding by submitting a command to surreal-commands and returning immediately with a job ID.
• surreal-commands provides the queue infrastructure that decouples job submission from execution, enabling true fire-and-forget semantics.
• embed_source in commands/embedding_commands.py handles the worker-side processing including chunking, embedding generation, and bulk inserts with 5-attempt retry logic.
• Command status endpoints in api/command_service.py allow asynchronous polling for clients that need completion confirmation.
• The architecture prevents HTTP request blocking while ensuring reliable, retryable background processing of source documents.

Frequently Asked Questions

How does Open Notebook prevent blocking during source embedding?

Open Notebook uses the surreal-commands job queue to offload embedding work. When Source.vectorize() calls submit_command(), it receives a command ID immediately and returns control to the API endpoint. The actual vectorization—chunking, embedding generation, and database inserts—runs in a separate worker process defined in commands/embedding_commands.py, completely independent of the HTTP request lifecycle.

Where is the embedding command logic implemented?

The background processing logic is implemented in commands/embedding_commands.py at lines 65-78. This file defines the embed_source command that workers execute, which includes loading the source, deleting old embeddings, detecting content types, chunking text, generating vectors, and bulk-inserting source_embedding records with exponential-jitter retry logic.

How can I check if a source's embeddings are ready?

Query the command status endpoint at GET /commands/{job_id} as implemented in api/command_service.py (lines 46-64). The Source.vectorize() method returns this job ID when submitting the embedding request. The endpoint returns the current status—running, completed, or failed—allowing your application to poll asynchronously or trigger downstream operations once embeddings are available.

What happens if the embedding job fails?

The embed_source command implements a robust retry strategy with 5 attempts using exponential-jitter backoff. Transient failures are automatically retried without client intervention. If all retries exhaust, the command status reflects the failure state, which you can detect via the commands API endpoint to implement application-level error handling or alerting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}