Open Notebook API Startup Sequence and Migration Auto-Detection Process

Open Notebook uses a FastAPI lifespan context manager to automatically detect and execute pending SurrealDB schema migrations before the server accepts any requests, ensuring the database is always synchronized with the application code.

When the FastAPI server for Open Notebook (lfnovo/open-notebook) boots, it executes a carefully orchestrated startup sequence that validates the environment, configures security middleware, and runs automatic database migrations. This process ensures that the SurrealDB schema is always up-to-date before any client requests are processed, eliminating manual migration steps and preventing schema mismatches in production.

FastAPI Lifespan and Application Setup

The entry point in api/main.py creates the FastAPI application with a custom lifespan context manager registered at lines 57-61. This pattern ensures that database migrations and security checks run before the server begins accepting traffic, with proper cleanup handled on shutdown.

Environment and Middleware Configuration

The startup begins by loading environment variables via dotenv.load_dotenv() at lines 2-4 of api/main.py. The application then configures middleware in a specific security order: PasswordAuthMiddleware is added at lines 73-86 to ensure authentication runs before CORS processing, followed by CORSMiddleware at lines 88-95. CORS origins are parsed using the _parse_cors_origins helper function (lines 53-64), defaulting to * if not specified.

The Startup Sequence Explained

When the server boots, the lifespan context manager entered at lines 98-102 executes the following sequence:

1. Security Validation: Verifies that OPEN_NOTEBOOK_ENCRYPTION_KEY exists in the environment (lines 107-114), logging a warning if the encryption key is missing.
2. Migration Manager Initialization: Instantiates AsyncMigrationManager() at lines 117-119 to handle schema versioning.
3. Version Detection: Calls get_current_version() in async_migrate.py:71-80 to query the _sbl_migrations table, returning the highest version number or 0 if the table does not exist.
4. Auto-Detection: The needs_migration() method (lines 75-78) compares the current version against the number of bundled migration files in open_notebook/database/migrations/.
5. Migration Execution: If needed, run_migration_up() invokes AsyncMigrationRunner.run_all() (lines 84-89) to execute pending .surrealql scripts sequentially.
6. Version Tracking: After each migration, bump_version() inserts a new row into _sbl_migrations (lines 36-44) to track applied changes.
7. Legacy Migration: Runs migrate_podcast_profiles from open_notebook/podcasts/migration.py (lines 39-45) for backward compatibility.

Once complete, the context yields control to the request handling loop at lines 49-52.

Migration Auto-Detection Mechanism

The automatic migration system relies on version tracking within SurrealDB itself.

Version Storage

The _sbl_migrations table stores the schema history. Each successful migration inserts a row containing the version number and timestamp, created by the bump_version() function in async_migrate.py:36-44.

Detecting Schema Drift

The AsyncMigrationManager loads all .surrealql files from open_notebook/database/migrations/ during initialization. The needs_migration() method returns True when current_version < len(self.up_migrations), triggering the update process.

Sequential Execution

AsyncMigrationRunner.run_all() iterates from the current version to the latest available migration, executing each file and updating the version counter atomically. This ensures that interruptions result in a consistent state, with the version number indicating exactly which migrations succeeded.

Running and Debugging Migrations

To start the API with automatic migrations:

uvicorn api.main:app --host 0.0.0.0 --port 5055

To manually check or run migrations for debugging:

import asyncio
from open_notebook.database.async_migrate import AsyncMigrationManager

async def debug_migrations():
    manager = AsyncMigrationManager()
    current = await manager.get_current_version()
    print(f"Current version: {current}")
    
    if await manager.needs_migration():
        await manager.run_migration_up()
        print("Migrations applied successfully")

asyncio.run(debug_migrations())

To verify applied versions directly in SurrealDB:

SELECT * FROM _sbl_migrations ORDER BY version;

Summary

• Open Notebook implements a lifespan context manager in api/main.py that orchestrates startup tasks before request handling begins.
• The AsyncMigrationManager auto-detects schema drift by comparing the _sbl_migrations table version against bundled .surrealql files in open_notebook/database/migrations/.
• Middleware ordering places PasswordAuthMiddleware before CORSMiddleware to ensure authentication precedes CORS handling.
• Migrations run automatically and sequentially, with each successful update recorded in the _sbl_migrations table.
• A legacy podcast profile migration runs separately after the main database schema updates to maintain backward compatibility.

Frequently Asked Questions

How does Open Notebook detect if migrations are needed?

The AsyncMigrationManager queries the _sbl_migrations table using get_current_version() to determine the highest applied migration number. It compares this value against the count of .surrealql files in the migrations directory. If the file count exceeds the stored version, needs_migration() returns True and triggers the automatic update process.

What happens if the OPEN_NOTEBOOK_ENCRYPTION_KEY is missing during startup?

The lifespan context in api/main.py:107-114 validates the presence of OPEN_NOTEBOOK_ENCRYPTION_KEY. If the variable is missing, the system logs a warning but continues starting the server. This allows development environments to run without the key while alerting production operators to potential security configuration gaps.

Can I run migrations manually without starting the full API?

Yes. You can instantiate AsyncMigrationManager directly in an async Python script and call await manager.run_migration_up() to execute pending migrations. This is useful for CI/CD pipelines, debugging, or maintenance windows where you need to verify database state before deploying application code.

Where are migration files stored and what naming convention do they use?

Migration files are stored in open_notebook/database/migrations/ and use the .surrealql extension with numeric filenames (e.g., 1.surrealql, 2.surrealql). The AsyncMigrationRunner executes these files sequentially based on their numeric order, ensuring dependencies are applied in the correct sequence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}