# Open Notebook API Startup Sequence and Migration Auto-Detection Process

> Discover the Open Notebook API startup sequence. Learn how it auto-detects and runs SurrealDB migrations via FastAPI lifespan for seamless app-to-database synchronization.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-11

---

**Open Notebook uses a FastAPI lifespan context manager to automatically detect and execute pending SurrealDB schema migrations before the server accepts any requests, ensuring the database is always synchronized with the application code.**

When the FastAPI server for Open Notebook (lfnovo/open-notebook) boots, it executes a carefully orchestrated startup sequence that validates the environment, configures security middleware, and runs automatic database migrations. This process ensures that the SurrealDB schema is always up-to-date before any client requests are processed, eliminating manual migration steps and preventing schema mismatches in production.

## FastAPI Lifespan and Application Setup

The entry point in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py) creates the FastAPI application with a custom `lifespan` context manager registered at lines 57-61. This pattern ensures that database migrations and security checks run before the server begins accepting traffic, with proper cleanup handled on shutdown.

### Environment and Middleware Configuration

The startup begins by loading environment variables via `dotenv.load_dotenv()` at lines 2-4 of [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py). The application then configures middleware in a specific security order: `PasswordAuthMiddleware` is added at lines 73-86 to ensure authentication runs before CORS processing, followed by `CORSMiddleware` at lines 88-95. CORS origins are parsed using the `_parse_cors_origins` helper function (lines 53-64), defaulting to `*` if not specified.

## The Startup Sequence Explained

When the server boots, the `lifespan` context manager entered at lines 98-102 executes the following sequence:

1. **Security Validation**: Verifies that `OPEN_NOTEBOOK_ENCRYPTION_KEY` exists in the environment (lines 107-114), logging a warning if the encryption key is missing.
2. **Migration Manager Initialization**: Instantiates `AsyncMigrationManager()` at lines 117-119 to handle schema versioning.
3. **Version Detection**: Calls `get_current_version()` in `async_migrate.py:71-80` to query the `_sbl_migrations` table, returning the highest version number or `0` if the table does not exist.
4. **Auto-Detection**: The `needs_migration()` method (lines 75-78) compares the current version against the number of bundled migration files in `open_notebook/database/migrations/`.
5. **Migration Execution**: If needed, `run_migration_up()` invokes `AsyncMigrationRunner.run_all()` (lines 84-89) to execute pending `.surrealql` scripts sequentially.
6. **Version Tracking**: After each migration, `bump_version()` inserts a new row into `_sbl_migrations` (lines 36-44) to track applied changes.
7. **Legacy Migration**: Runs `migrate_podcast_profiles` from [`open_notebook/podcasts/migration.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/migration.py) (lines 39-45) for backward compatibility.

Once complete, the context yields control to the request handling loop at lines 49-52.

## Migration Auto-Detection Mechanism

The automatic migration system relies on version tracking within SurrealDB itself.

### Version Storage

The `_sbl_migrations` table stores the schema history. Each successful migration inserts a row containing the version number and timestamp, created by the `bump_version()` function in `async_migrate.py:36-44`.

### Detecting Schema Drift

The `AsyncMigrationManager` loads all `.surrealql` files from `open_notebook/database/migrations/` during initialization. The `needs_migration()` method returns `True` when `current_version < len(self.up_migrations)`, triggering the update process.

### Sequential Execution

`AsyncMigrationRunner.run_all()` iterates from the current version to the latest available migration, executing each file and updating the version counter atomically. This ensures that interruptions result in a consistent state, with the version number indicating exactly which migrations succeeded.

## Running and Debugging Migrations

To start the API with automatic migrations:

```bash
uvicorn api.main:app --host 0.0.0.0 --port 5055

```

To manually check or run migrations for debugging:

```python
import asyncio
from open_notebook.database.async_migrate import AsyncMigrationManager

async def debug_migrations():
    manager = AsyncMigrationManager()
    current = await manager.get_current_version()
    print(f"Current version: {current}")
    
    if await manager.needs_migration():
        await manager.run_migration_up()
        print("Migrations applied successfully")

asyncio.run(debug_migrations())

```

To verify applied versions directly in SurrealDB:

```sql
SELECT * FROM _sbl_migrations ORDER BY version;

```

## Summary

- Open Notebook implements a **lifespan context manager** in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py) that orchestrates startup tasks before request handling begins.
- The **`AsyncMigrationManager`** auto-detects schema drift by comparing the `_sbl_migrations` table version against bundled `.surrealql` files in `open_notebook/database/migrations/`.
- **Middleware ordering** places `PasswordAuthMiddleware` before `CORSMiddleware` to ensure authentication precedes CORS handling.
- Migrations run **automatically and sequentially**, with each successful update recorded in the `_sbl_migrations` table.
- A **legacy podcast profile migration** runs separately after the main database schema updates to maintain backward compatibility.

## Frequently Asked Questions

### How does Open Notebook detect if migrations are needed?

The `AsyncMigrationManager` queries the `_sbl_migrations` table using `get_current_version()` to determine the highest applied migration number. It compares this value against the count of `.surrealql` files in the migrations directory. If the file count exceeds the stored version, `needs_migration()` returns `True` and triggers the automatic update process.

### What happens if the OPEN_NOTEBOOK_ENCRYPTION_KEY is missing during startup?

The lifespan context in `api/main.py:107-114` validates the presence of `OPEN_NOTEBOOK_ENCRYPTION_KEY`. If the variable is missing, the system logs a warning but continues starting the server. This allows development environments to run without the key while alerting production operators to potential security configuration gaps.

### Can I run migrations manually without starting the full API?

Yes. You can instantiate `AsyncMigrationManager` directly in an async Python script and call `await manager.run_migration_up()` to execute pending migrations. This is useful for CI/CD pipelines, debugging, or maintenance windows where you need to verify database state before deploying application code.

### Where are migration files stored and what naming convention do they use?

Migration files are stored in `open_notebook/database/migrations/` and use the `.surrealql` extension with numeric filenames (e.g., `1.surrealql`, `2.surrealql`). The `AsyncMigrationRunner` executes these files sequentially based on their numeric order, ensuring dependencies are applied in the correct sequence.