How AsyncMigrationManager Automatically Runs SurrealDB Schema Migrations on API Startup

When the FastAPI application starts, the lifespan context manager in api/main.py instantiates AsyncMigrationManager, detects the current schema version from SurrealDB, and executes any pending migrations from open_notebook/database/migrations/ before the API accepts its first request, aborting startup entirely if any migration fails.

The lfnovo/open-notebook repository guarantees database schema consistency by integrating automatic migration logic into the application boot sequence. Using FastAPI’s asynchronous lifespan hooks, the AsyncMigrationManager class orchestrates version detection, migration ordering, and transactional schema updates without requiring manual intervention. This architecture ensures that the SurrealDB schema is always synchronized with the application code before any HTTP endpoints become available.

The FastAPI Lifespan Hook: Where Automation Begins

In api/main.py, the lifespan async context manager serves as the entry point for schema migration. FastAPI executes this context manager exactly once during startup, before any routes are mounted. Inside this hook, the system initializes the migration manager and triggers the version-check workflow, ensuring the database is ready before the server starts accepting traffic.

from open_notebook.database.async_migrate import AsyncMigrationManager

async def lifespan(app: FastAPI):
    # Database connection setup...

    migration_manager = AsyncMigrationManager()
    current_version = await migration_manager.get_current_version()
    logger.info(f"Current database version: {current_version}")

    if await migration_manager.needs_migration():
        logger.warning("Database migrations are pending. Running migrations...")
        await migration_manager.run_migration_up()
        logger.success(
            f"Migrations completed successfully. Database is now at version "
            f"{await migration_manager.get_current_version()}"
        )
    # Application startup continues...

Initializing the Migration Manager and Scanning Migration Files

The AsyncMigrationManager class, defined in open_notebook/database/async_migrate.py, prepares an ordered list of migrations during instantiation. Its __init__ method constructs up_migrations by explicitly loading each SurrealQL file from open_notebook/database/migrations/ into an AsyncMigration object, ensuring migrations run in sequential order.

class AsyncMigrationManager:
    def __init__(self):
        self.up_migrations = [
            AsyncMigration.from_file("open_notebook/database/migrations/1.surrealql"),
            AsyncMigration.from_file("open_notebook/database/migrations/2.surrealql"),
            # ...

            AsyncMigration.from_file("open_notebook/database/migrations/14.surrealql"),
        ]

Detecting Current Schema Version and Migration Requirements

Before executing any SQL, the manager determines the database’s current state. The get_current_version method delegates to get_latest_version, which queries the _sbl_migrations table in SurrealDB. If this table does not exist, the method returns version 0. The needs_migration method then compares this integer against the length of the up_migrations list to determine if the schema is outdated.

Executing Pending Migrations Transactionally

When migrations are required, run_migration_up delegates to the internal runner’s run_all() method. This iterates through pending AsyncMigration objects, executing each SQL statement via the SurrealDB client using connection.query(self.sql). After each successful migration, bump_version updates the schema version in the _sbl_migrations table, ensuring atomic progress tracking.

async def run(self, bump: bool = True) -> None:
    async with db_connection() as connection:
        await connection.query(self.sql)  # Execute the SurrealQL

    if bump:
        await bump_version()  # Record new version in _sbl_migrations

    else:
        await lower_version()

Preventing Startup on Migration Failure

If any migration step raises an exception, the lifespan handler in api/main.py catches the error and raises a RuntimeError, halting the FastAPI startup process. This fail-fast approach prevents the API from serving requests against an incomplete or incompatible database schema, forcing immediate operational attention to the database state.

Summary

• FastAPI lifespan integration: The lifespan context manager in api/main.py triggers AsyncMigrationManager exactly once during application startup.
• Ordered migration scanning: The manager constructor builds up_migrations by loading numbered SurrealQL files from open_notebook/database/migrations/.
• Version detection: get_current_version reads the _sbl_migrations table via get_latest_version to determine the existing schema state.
• Migration requirement check: needs_migration compares the current version against the migration file count to identify pending updates.
• Transactional execution: run_migration_up executes pending statements via connection.query() and increments the version using bump_version.
• Startup safety: Any migration failure raises a RuntimeError that aborts startup, preventing the API from running with an outdated schema.

Frequently Asked Questions

What triggers the SurrealDB migrations to run when the API starts?

The FastAPI lifespan asynchronous context manager defined in api/main.py instantiates AsyncMigrationManager and invokes its migration workflow during the application startup sequence. This executes before the server binds to any ports or accepts HTTP requests, ensuring the SurrealDB schema is fully updated and compatible with the running code.

How does AsyncMigrationManager determine which migrations need to run?

The manager queries the _sbl_migrations table via get_current_version (which delegates to get_latest_version) to retrieve an integer representing the current schema version. The needs_migration method compares this value against the length of the internal up_migrations list, which contains AsyncMigration objects loaded from the filesystem. If the current version is less than the migration count, the system identifies the delta and queues the remaining migrations for execution.

What happens if a migration fails during API startup?

If any AsyncMigration throws an exception during run_migration_up, the error propagates to the lifespan handler in api/main.py, which raises a RuntimeError and halts the FastAPI startup process. This prevents the application from running with a partially applied schema, ensuring data integrity and forcing immediate resolution of the database issue before the service becomes available.

Where are the migration SQL files stored in the repository?

Migration files are stored as numbered SurrealQL scripts in the open_notebook/database/migrations/ directory. The AsyncMigrationManager constructor explicitly references files such as 1.surrealql, 2.surrealql, and so on, creating an ordered sequence that guarantees schema changes apply in the correct chronological order.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}