What Is AsyncMigrationManager? Database Schema Management in Open Notebook

AsyncMigrationManager is the central async coordinator that automates SurrealDB schema migrations by collecting SQL scripts, detecting current versions, and executing pending updates on application startup.

In the Open Notebook project, database schema evolution is handled automatically through the AsyncMigrationManager class. This component ensures that the SurrealDB backend stays synchronized with application code by running migrations safely in an async context whenever the service starts.

Core Responsibilities of AsyncMigrationManager

The AsyncMigrationManager class in open_notebook/database/async_migrate.py orchestrates six critical operations for schema management:

Collecting Migration Scripts

When initialized, the manager scans the open_notebook/database/migrations/ directory to build ordered lists of migration objects. It separates up migrations (forward changes) from down migrations (rollback scripts), storing them in self.up_migrations and self.down_migrations respectively.

Detecting Current Schema Version

The manager queries the _sbl_migrations table using the get_latest_version() method to determine the most recently applied migration. This version check happens in open_notebook/database/async_migrate.py lines 98-105, where it executes a SELECT against the bookkeeping table.

Determining Migration Requirements

The needs_migration() method compares the current database version against the number of available up migrations. If the counts differ, the system knows pending changes exist and returns True to trigger execution.

Executing Pending Migrations

When migrations are needed, run_migration_up() invokes the AsyncMigrationRunner to apply each missing script sequentially. This process includes error handling and progress logging to ensure atomic application of schema changes.

Version Bookkeeping

After each successful migration, the bump_version() helper inserts a new row into _sbl_migrations (lines 20-28 in async_migrate.py). This creates an immutable audit trail of all schema changes applied to the database.

Automatic Startup Integration

The FastAPI lifespan hook in api/main.py (lines 17-33) instantiates AsyncMigrationManager, checks needs_migration(), and automatically runs run_migration_up() before the API begins serving requests.

How AsyncMigrationManager Integrates with FastAPI

Open Notebook ensures zero-downtime deployments by running migrations during the application lifespan. The integration in api/main.py follows this pattern:

from open_notebook.database.async_migrate import AsyncMigrationManager

async def lifespan(app):
    manager = AsyncMigrationManager()
    if await manager.needs_migration():
        await manager.run_migration_up()
    # ... continue with startup

This guarantees that the database schema is always up-to-date before handling traffic, eliminating manual migration steps during deployment.

Manual Migration Operations

While automatic execution handles most cases, you can interact with AsyncMigrationManager directly for maintenance tasks.

Checking Migration Status

To verify current version and pending changes:

async def check_status():
    manager = AsyncMigrationManager()
    current = await manager.get_current_version()
    pending = len(manager.up_migrations) - current
    print(f"Current version: {current}, Pending: {pending}")

Rolling Back Migrations

To reverse the last applied migration using the down scripts:

async def rollback():
    manager = AsyncMigrationManager()
    await manager.runner.run_one_down()

Adding New Migrations

When extending the schema:

1. Create a new SQL file in open_notebook/database/migrations/ (e.g., 015_add_new_table.surrealql)
2. Add the file path to self.up_migrations in AsyncMigrationManager.__init__
3. Include a corresponding down script in self.down_migrations for rollback capability

Synchronous Compatibility Layer

For legacy synchronous code, the repository provides a wrapper in open_notebook/database/migrate.py. The Migration class proxies calls to AsyncMigrationManager:

from open_notebook.database.migrate import Migration

# Synchronous API that delegates to the async manager

Migration().run()

Summary

• AsyncMigrationManager coordinates all SurrealDB schema migrations in Open Notebook through open_notebook/database/async_migrate.py
• It maintains version state in the _sbl_migrations table and uses get_latest_version() to detect the current schema level
• The needs_migration() method determines if pending updates exist by comparing current version against available scripts
• Up migrations apply forward changes via run_migration_up(), while down migrations handle rollbacks through AsyncMigrationRunner
• FastAPI lifespan hooks automatically execute migrations on startup, ensuring schema consistency before serving requests
• Version tracking occurs through bump_version(), which records each successful migration in the database

Frequently Asked Questions

How does AsyncMigrationManager know which migrations to run?

The manager scans the open_notebook/database/migrations/ directory during initialization to build ordered lists of available scripts. It compares the highest available up migration index against the current version stored in the _sbl_migrations table. Any scripts with indices higher than the current version are queued for execution via run_migration_up().

What happens if a migration fails during automatic startup?

If AsyncMigrationRunner encounters an error while executing a migration script, the exception propagates up through run_migration_up() and halts the FastAPI startup process. This prevents the application from serving requests with an incomplete or corrupted schema, ensuring atomic migration application.

Can I run migrations manually without starting the full application?

Yes. You can instantiate AsyncMigrationManager directly in a Python script or REPL and call await manager.run_migration_up() to execute pending migrations. For synchronous environments, use the Migration class from open_notebook/database/migrate.py which provides a blocking wrapper around the async manager.

Where are migration versions stored and tracked?

Version history persists in the _sbl_migrations table within SurrealDB. The get_latest_version() method queries this table to determine the current schema state, while bump_version() inserts new records after each successful migration. This table serves as the single source of truth for schema versioning across all deployment environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}