How the Open Notebook API Manages SurrealDB Connection Failures with Startup Retry Logic

The Open Notebook API implements a resilient startup retry loop with exponential back-off that probes SurrealDB using a lightweight query, retrying up to 12 times with configurable delays before failing fast if the database remains unreachable.

When the Open Notebook FastAPI service starts, it must guarantee that SurrealDB is reachable before executing migrations or handling requests. The application implements a robust startup retry mechanism in api/main.py that uses exponential back-off and configurable timeouts to handle transient connection failures during container orchestration or slow database initialization.

Configurable Retry Parameters

The retry behavior is controlled by four constants defined at lines 67-73 in api/main.py:

• DATABASE_STARTUP_RETRY_ATTEMPTS = 12 – Maximum connection attempts.
• DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS = 1 – Starting delay for back-off.
• DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS = 5 – Ceiling for exponential back-off.
• DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS = 5 – Per-probe timeout to prevent indefinite hangs.

These values are read by the _wait_for_database helper function, which orchestrates the startup validation sequence.

The Readiness Probe Implementation

The _wait_for_database function validates connectivity by calling AsyncMigrationManager.ping() from open_notebook/database/async_migrate.py (lines 199-203). This method executes a minimal SurrealQL command through the low-level connection provided by open_notebook/database/repository.py:

# From open_notebook/database/async_migrate.py

async def ping(self):
    await self.db_connection.query("RETURN true;")

Each probe is wrapped in asyncio.wait_for() with a timeout of DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS (5 seconds). This ensures that stalled connections are treated as failures rather than blocking the startup process indefinitely.

Exponential Back-Off Strategy

After each failed attempt, the loop sleeps for an exponentially increasing delay, starting at 1 second and doubling until it reaches the 5-second maximum cap. This prevents aggressive retry storms while accommodating databases that need time to initialize in containerized environments.

The delay sequence follows: 1s → 2s → 4s → 5s (capped) → 5s for remaining attempts.

FastAPI Lifespan Integration

The retry logic executes within the FastAPI lifespan context manager before any routers are mounted. At line 81 in api/main.py, the lifespan function calls await _run_database_migrations(), which internally invokes _wait_for_database():

# Conceptual flow within the lifespan manager

@asynccontextmanager
async def lifespan(app: FastAPI):
    await _wait_for_database()  # Retry loop executes here

    await _run_database_migrations()
    yield
    # Shutdown logic...

Only after the probe succeeds does the API proceed to run database migrations and begin accepting requests. This sequential dependency ensures that the service never starts in a degraded state.

Logging and Fail-Fast Behavior

Each connection failure logs a warning message including the attempt number and specific error details. If all 12 attempts fail, the exception propagates from lines 124-132 in api/main.py, causing the application to abort startup immediately.

This fail-fast approach prevents the API from running with an unavailable database, making deployment issues visible immediately rather than causing runtime errors during request handling.

Customizing Retry Behavior

You can override the default retry constants using environment variables before importing the application:

import os

os.environ["DATABASE_STARTUP_RETRY_ATTEMPTS"] = "20"
os.environ["DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS"] = "2"
os.environ["DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS"] = "10"

# Import and run the app after setting variables

from api.main import app

Summary

• The Open Notebook API uses configurable retry parameters defined in api/main.py to control connection attempts and timing.
• AsyncMigrationManager.ping() in open_notebook/database/async_migrate.py provides a lightweight health check using RETURN true;.
• Exponential back-off with a 5-second cap prevents tight retry loops while allowing slow database startups.
• The FastAPI lifespan context ensures the database is reachable before migrations or request handling begins.
• Fail-fast error handling at startup prevents the API from running with an unreachable SurrealDB instance.

Frequently Asked Questions

How many times does the Open Notebook API retry SurrealDB connections on startup?

The API attempts to connect 12 times by default, as defined by DATABASE_STARTUP_RETRY_ATTEMPTS in api/main.py. You can override this via the DATABASE_STARTUP_RETRY_ATTEMPTS environment variable before starting the service.

What happens if SurrealDB is completely unreachable during startup?

If the database remains unreachable after all retry attempts are exhausted, the API logs a final error and aborts the startup process (lines 124-132 in api/main.py). This prevents the FastAPI application from starting in a degraded state and ensures deployment failures are caught immediately.

How does the API avoid blocking forever on a stuck SurrealDB connection?

Each health check probe uses asyncio.wait_for() with a 5-second timeout (DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS). If the AsyncMigrationManager.ping() method does not complete within this window, it raises an asyncio.TimeoutError, which triggers the next retry iteration rather than hanging indefinitely.

Can I customize the delay between retry attempts?

Yes, the retry delay uses exponential back-off starting at DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS (1 second by default) and doubling each time until it reaches DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS (5 seconds). You can modify these constants via environment variables to adjust the back-off behavior for your infrastructure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}