How Open Notebook Handles Database Migrations with AsyncMigrationManager

Open Notebook automatically executes database migrations during FastAPI startup using an AsyncMigrationManager that runs idempotent SurrealDB scripts asynchronously without blocking the server.

Open Notebook stores all persistent data in SurrealDB, requiring schema and data transformations as the application evolves. The project solves this challenge with an AsyncMigrationManager located in open_notebook/database/migration_manager.py, which integrates directly into the FastAPI lifecycle to ensure migrations complete before the API serves requests.

FastAPI Startup Integration

The migration system hooks into the FastAPI application lifecycle through a startup event defined in api/main.py. When the server initializes, it instantiates the AsyncMigrationManager and awaits its run() coroutine, guaranteeing that the database schema is current before handling traffic.

This design leverages the asynchronous SurrealDB driver, allowing the event loop to process other connections while migrations execute. If any migration raises a fatal error, the server aborts startup, preventing the API from operating on an incompatible schema.

# api/main.py

from fastapi import FastAPI
from open_notebook.database.migration_manager import AsyncMigrationManager

app = FastAPI()

@app.on_event("startup")
async def startup():
    # Run any pending schema and data migrations

    await AsyncMigrationManager().run()

AsyncMigrationManager Architecture

The AsyncMigrationManager class serves as the orchestration layer for all database changes. It discovers migration modules across the codebase and executes them sequentially using the async SurrealDB connection. Because the manager operates entirely within Python's async/await paradigm, it avoids blocking I/O during long-running data transformations.

Migration scripts are standard Python modules that import from the repository layer. The manager does not enforce a specific naming convention beyond module discoverability, allowing developers to colocate migrations with their domain logic—such as open_notebook/podcasts/migration.py for podcast-related schema changes.

Writing Idempotent Migration Scripts

Each migration script must be idempotent, meaning it can safely run multiple times without duplicating data or causing errors. Scripts achieve this by querying the current state before applying changes, typically using guard clauses to skip already-migrated records.

The podcast migration in open_notebook/podcasts/migration.py demonstrates this pattern by checking for the presence of new fields before transformation:

# open_notebook/podcasts/migration.py

async def migrate_podcast_profiles() -> None:
    logger.info("Starting podcast profile data migration...")
    episode_profiles = await repo_query("SELECT * FROM episode_profile")
    
    for raw in episode_profiles:
        needs_outline = raw.get("outline_llm")
        needs_transcript = raw.get("transcript_llm")
        
        # Idempotent guard: skip if already migrated

        if not needs_outline and not needs_transcript:
            continue
            
        # Resolve legacy provider/model strings to Model records

        model_id = await _find_or_create_model(provider, model_name, "language")
        if model_id:
            await repo_update(
                "episode_profile",
                str(raw["id"]),
                {"outline_llm": ensure_record_id(model_id)},
            )

Legacy Data Transformation Patterns

A common migration task involves converting legacy string fields to proper record references. The podcast example resolves historical provider and model strings into formal Model records, auto-creating the model when matching credentials exist. It then updates the profile with the new record ID using the ensure_record_id helper to format the reference correctly for SurrealQL.

Repository Layer Abstraction

Migration scripts never execute raw SurrealQL directly. Instead, they interface with the generic repository layer defined in open_notebook/database/repository.py. This abstraction provides consistent async CRUD operations and handles SurrealDB-specific syntax requirements.

Key helpers include:

• repo_query – Execute SELECT statements and return raw results
• repo_update – Apply partial updates to specific records by ID
• ensure_record_id – Wrap raw SurrealDB IDs in the required <record> syntax for record linking

# open_notebook/database/repository.py

def ensure_record_id(record_id: str) -> str:
    """Wrap a raw SurrealDB ID in the required <record> syntax."""
    return f"<{record_id}>"

Using these helpers ensures that migrations remain readable and maintainable while adhering to the project's database conventions.

Safety and Observability Features

The AsyncMigrationManager provides comprehensive safety guarantees through its integration with the application lifecycle and detailed logging. During execution, the manager tracks statistics for each migration domain—such as episodes migrated, skipped, or failed—and emits concise summaries via loguru.

Because migrations run inside the FastAPI startup event, any unhandled exception prevents the server from starting. This fail-fast behavior ensures that the API never operates on an outdated schema or partially migrated data. The asynchronous nature of the manager means that while migrations run, the event loop remains responsive to other async operations, though in practice migrations complete before the server accepts HTTP requests.

Summary

• Automatic execution – The AsyncMigrationManager.run() coroutine awaits during FastAPI startup in api/main.py, ensuring migrations complete before request handling begins.
• Non-blocking async design – Leverages the async SurrealDB driver to perform schema changes without blocking the event loop.
• Idempotent scripts – Migration modules use guard clauses to skip already-migrated records, making re-runs safe and predictable.
• Repository abstraction – All database interactions flow through open_notebook/database/repository.py helpers like repo_query and repo_update.
• Built-in safety – Fatal migration errors abort server startup, while detailed logging tracks migration success, skips, and failures per domain.

Frequently Asked Questions

What triggers database migrations in Open Notebook?

Migrations trigger automatically when the FastAPI server starts. The startup event handler in api/main.py instantiates AsyncMigrationManager and awaits its run() method, ensuring the database schema is current before the application accepts traffic.

How does AsyncMigrationManager ensure migrations are safe to run multiple times?

Individual migration scripts implement idempotency by checking the current database state before applying changes. For example, the podcast migration queries existing fields and skips records that already contain the new data, preventing duplicate updates or constraint violations on re-runs.

What happens if a migration fails during startup?

If any migration raises an exception, the FastAPI server fails to start. This fail-fast design prevents the API from operating on an inconsistent schema. Errors are logged via loguru with specific details about which records failed, allowing developers to fix the migration script or data issue before restarting.

Can I run migrations manually without starting the FastAPI server?

While the primary entry point is the FastAPI startup hook, you can import AsyncMigrationManager from open_notebook/database/migration_manager.py in a standalone script and await run() directly. This requires a valid SurrealDB connection configured through the project's environment variables, but allows for offline maintenance or CI/CD pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}