How to Run Database Migrations and Check the Current Version in Open Notebook

Use the synchronous MigrationManager class from open_notebook/database/migrate.py to manually execute SurrealDB schema migrations or query the current database version without launching the full API server.

Open Notebook stores its data in SurrealDB and manages schema evolution through numbered SurrealQL migration files. While the API service automatically applies pending migrations on startup, you can also run database migrations manually using the synchronous wrapper exposed in the backend codebase. This approach is essential for CI/CD pipelines, local debugging, or maintenance scripts where you require explicit control over the database state.

Migration Architecture and Key Files

The migration system in lfnovo/open-notebook uses a dual-layer architecture that separates the asynchronous execution engine from a synchronous public interface.

The implementation is split across three critical locations:

• open_notebook/database/migrate.py – A thin synchronous façade that exposes the MigrationManager class for direct use in scripts and REPLs.
• open_notebook/database/async_migrate.py – The core asynchronous migration engine that reads and executes SurrealQL files.
• open_notebook/database/migrations/ – The directory containing ordered migration files (e.g., 1.surrealql, 2.surrealql) that define schema changes.

When you instantiate MigrationManager, it handles the asyncio event loop internally, allowing you to call migration methods from standard synchronous Python code without managing async/await syntax.

How to Check the Current Migration Version

To determine which migration version the database is currently running, use the get_current_version() method. This queries the internal _sbl_migrations table in SurrealDB and returns an integer representing the applied version, or 0 if the migrations table does not yet exist.

from open_notebook.database.migrate import MigrationManager

manager = MigrationManager()
current = manager.get_current_version()
print(f"The database is at migration version {current}")

The _sbl_migrations table stores each applied migration as a record containing the version number and an applied_at timestamp, enabling the manager to determine exactly which schema changes are present.

How to Run Database Migrations Manually

The MigrationManager exposes three primary methods for manual control:

• get_current_version() – Returns the integer version stored in _sbl_migrations.
• needs_migration – A boolean property that returns True when the database version is lower than the latest migration file available in the migrations directory.
• run_migration_up() – Executes every pending "up" migration in sequential order, updating the version record after each successful application.

Because this wrapper is synchronous, you can invoke these methods from any Python context without worrying about event loops or coroutine management.

Script-Based Migration Execution

Create a standalone script to apply pending migrations during deployment or setup:

# scripts/run_migrations.py

from open_notebook.database.migrate import MigrationManager

def main() -> None:
    manager = MigrationManager()

    if manager.needs_migration:
        print("Applying pending migrations...")
        manager.run_migration_up()
        print("Migrations applied successfully.")
    else:
        print("Database is already up-to-date.")

if __name__ == "__main__":
    main()

Execute the script after ensuring your SurrealDB instance is accessible:

python scripts/run_migrations.py

Verifying Migration Status Before Execution

For idempotent deployment logic, check the status before deciding whether to migrate:

from open_notebook.database.migrate import MigrationManager

manager = MigrationManager()

if manager.needs_migration:
    print(f"Current version: {manager.get_current_version()}")
    manager.run_migration_up()
    print(f"Updated to version: {manager.get_current_version()}")
else:
    print("No migrations required.")

Interactive REPL Workflow

You can inspect and manipulate migrations directly in an interactive Python session:

>>> from open_notebook.database.migrate import MigrationManager
>>> mm = MigrationManager()
>>> mm.get_current_version()
3
>>> mm.needs_migration
True
>>> mm.run_migration_up()

Integrating Migrations into CI/CD Pipelines

Use the manager in shell scripts or CI workflows to ensure database consistency before running tests or deployments. This example exits with code 1 if migrations are pending, failing the pipeline step:

python -c "from open_notebook.database.migrate import MigrationManager; \
           m = MigrationManager(); \
           import sys; \
           sys.exit(0 if not m.needs_migration else 1)"

Alternatively, automatically apply migrations and verify success:

python -c "from open_notebook.database.migrate import MigrationManager; \
           m = MigrationManager(); \
           m.run_migration_up() if m.needs_migration else None; \
           print(f'Database at version {m.get_current_version()}')"

Where Migration State is Stored

Open Notebook tracks migration state in a special SurrealDB record type called _sbl_migrations. Each time run_migration_up() applies a migration file, it inserts a record containing:

• version: The integer version number of the applied migration (starting at 1).
• applied_at: A timestamp marking when the migration was executed.

The MigrationManager queries this table to calculate needs_migration and to determine the starting point for run_migration_up(). If this table is missing or empty, the manager treats the database as being at version 0.

Summary

• Automatic execution: Migrations run automatically when the FastAPI server starts, but manual execution is available via MigrationManager.
• Synchronous interface: The open_notebook/database/migrate.py module provides a blocking API that wraps the async engine, making it suitable for scripts and REPLs.
• Version tracking: Query get_current_version() to read the current schema version from the _sbl_migrations table.
• Pending detection: Use the needs_migration property to check if the database lags behind the latest migration files.
• Manual execution: Call run_migration_up() to apply all pending SurrealQL migrations in order.
• No server required: All migration operations work with only a SurrealDB connection, without requiring the full application server to run.

Frequently Asked Questions

Can I run database migrations without starting the Open Notebook API server?

Yes. The MigrationManager only requires a valid SurrealDB connection using the same configuration as the API. You can execute migrations from standalone scripts, Python REPLs, or CI environments without launching the FastAPI application.

How does Open Notebook track which migrations have already been applied?

The system stores migration metadata in a SurrealDB table named _sbl_migrations. Each record contains the migration version number and an applied_at timestamp. The get_current_version() method queries this table to determine the current schema state.

What file format are the migrations written in?

Migration files are written in SurrealQL, the query language for SurrealDB. They are stored as numbered files (e.g., 1.surrealql, 2.surrealql) in the open_notebook/database/migrations/ directory and executed sequentially by the migration engine.

How can I check if my database needs migrations before deploying?

Use the needs_migration boolean property on a MigrationManager instance. If it returns True, the database version is lower than the latest available migration file, and you should call run_migration_up() to bring the schema up to date.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}