# How the Open Notebook API Manages SurrealDB Connection Failures with Startup Retry Logic

> Learn how the Open Notebook API uses startup retry logic and exponential back-off to manage SurrealDB connection failures retrying up to 12 times before failing fast.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-07-04

---

**The Open Notebook API implements a resilient startup retry loop with exponential back-off that probes SurrealDB using a lightweight query, retrying up to 12 times with configurable delays before failing fast if the database remains unreachable.**

When the Open Notebook FastAPI service starts, it must guarantee that SurrealDB is reachable before executing migrations or handling requests. The application implements a robust **startup retry mechanism** in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py) that uses exponential back-off and configurable timeouts to handle transient connection failures during container orchestration or slow database initialization.

## Configurable Retry Parameters

The retry behavior is controlled by four constants defined at lines 67-73 in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py):

- `DATABASE_STARTUP_RETRY_ATTEMPTS = 12` – Maximum connection attempts.
- `DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS = 1` – Starting delay for back-off.
- `DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS = 5` – Ceiling for exponential back-off.
- `DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS = 5` – Per-probe timeout to prevent indefinite hangs.

These values are read by the `_wait_for_database` helper function, which orchestrates the startup validation sequence.

## The Readiness Probe Implementation

The `_wait_for_database` function validates connectivity by calling `AsyncMigrationManager.ping()` from [`open_notebook/database/async_migrate.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/async_migrate.py) (lines 199-203). This method executes a minimal SurrealQL command through the low-level connection provided by [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py):

```python

# From open_notebook/database/async_migrate.py

async def ping(self):
    await self.db_connection.query("RETURN true;")

```

Each probe is wrapped in `asyncio.wait_for()` with a timeout of `DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS` (5 seconds). This ensures that stalled connections are treated as failures rather than blocking the startup process indefinitely.

## Exponential Back-Off Strategy

After each failed attempt, the loop sleeps for an exponentially increasing delay, starting at 1 second and doubling until it reaches the 5-second maximum cap. This prevents aggressive retry storms while accommodating databases that need time to initialize in containerized environments.

The delay sequence follows: 1s → 2s → 4s → 5s (capped) → 5s for remaining attempts.

## FastAPI Lifespan Integration

The retry logic executes within the FastAPI lifespan context manager before any routers are mounted. At line 81 in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py), the `lifespan` function calls `await _run_database_migrations()`, which internally invokes `_wait_for_database()`:

```python

# Conceptual flow within the lifespan manager

@asynccontextmanager
async def lifespan(app: FastAPI):
    await _wait_for_database()  # Retry loop executes here

    await _run_database_migrations()
    yield
    # Shutdown logic...

```

Only after the probe succeeds does the API proceed to run database migrations and begin accepting requests. This sequential dependency ensures that the service never starts in a degraded state.

## Logging and Fail-Fast Behavior

Each connection failure logs a warning message including the attempt number and specific error details. If all 12 attempts fail, the exception propagates from lines 124-132 in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py), causing the application to abort startup immediately.

This **fail-fast** approach prevents the API from running with an unavailable database, making deployment issues visible immediately rather than causing runtime errors during request handling.

## Customizing Retry Behavior

You can override the default retry constants using environment variables before importing the application:

```python
import os

os.environ["DATABASE_STARTUP_RETRY_ATTEMPTS"] = "20"
os.environ["DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS"] = "2"
os.environ["DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS"] = "10"

# Import and run the app after setting variables

from api.main import app

```

## Summary

- The Open Notebook API uses **configurable retry parameters** defined in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py) to control connection attempts and timing.
- **AsyncMigrationManager.ping()** in [`open_notebook/database/async_migrate.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/async_migrate.py) provides a lightweight health check using `RETURN true;`.
- **Exponential back-off** with a 5-second cap prevents tight retry loops while allowing slow database startups.
- The **FastAPI lifespan** context ensures the database is reachable before migrations or request handling begins.
- **Fail-fast error handling** at startup prevents the API from running with an unreachable SurrealDB instance.

## Frequently Asked Questions

### How many times does the Open Notebook API retry SurrealDB connections on startup?

The API attempts to connect **12 times by default**, as defined by `DATABASE_STARTUP_RETRY_ATTEMPTS` in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py). You can override this via the `DATABASE_STARTUP_RETRY_ATTEMPTS` environment variable before starting the service.

### What happens if SurrealDB is completely unreachable during startup?

If the database remains unreachable after all retry attempts are exhausted, the API logs a final error and **aborts the startup process** (lines 124-132 in [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py)). This prevents the FastAPI application from starting in a degraded state and ensures deployment failures are caught immediately.

### How does the API avoid blocking forever on a stuck SurrealDB connection?

Each health check probe uses `asyncio.wait_for()` with a **5-second timeout** (`DATABASE_STARTUP_RETRY_PROBE_TIMEOUT_SECONDS`). If the `AsyncMigrationManager.ping()` method does not complete within this window, it raises an `asyncio.TimeoutError`, which triggers the next retry iteration rather than hanging indefinitely.

### Can I customize the delay between retry attempts?

Yes, the retry delay uses **exponential back-off** starting at `DATABASE_STARTUP_RETRY_INITIAL_DELAY_SECONDS` (1 second by default) and doubling each time until it reaches `DATABASE_STARTUP_RETRY_MAX_DELAY_SECONDS` (5 seconds). You can modify these constants via environment variables to adjust the back-off behavior for your infrastructure.