# Database-First Credential Storage Mechanism in Open-Notebook: SurrealDB Implementation

> Explore the database-first credential storage in Open-Notebook using SurrealDB. Learn how it securely stores AI provider authentication data as encrypted records linked to AI models.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-07-04

---

**Open-Notebook stores AI-provider authentication data as first-class records in SurrealDB, encrypting sensitive values at rest and linking credentials to AI models through foreign-key relationships rather than singleton configuration.**

The open-notebook project implements a robust database-first credential storage mechanism that treats authentication data as native database records rather than application configuration. This architecture leverages SurrealDB as the primary datastore, storing each credential as a distinct row in the `credential` table with full encryption support and flexible schema evolution capabilities.

## Core Architecture of the Credential System

### SurrealDB Table Structure

In [`open_notebook/domain/credential.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/credential.py), the `Credential` model extends `ObjectModel` and explicitly defines `table_name = "credential"` (line 37). This design establishes credentials as first-class database citizens rather than singleton configuration objects. Each record represents a single authentication context for an AI provider, with dedicated columns for `api_key`, `base_url`, `endpoint_*`, `project`, and `location`.

### Encryption at Rest

Sensitive values receive automatic encryption before persistence. The model utilizes `encrypt_value` and `decrypt_value` helper functions from [`open_notebook/utils/encryption.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/encryption.py) (referenced in lines 26–27, 34–40, and 62–68 of [`credential.py`](https://github.com/lfnovo/open-notebook/blob/main/credential.py)). This ensures API keys never exist in plaintext within the database, protecting against credential leakage through raw database access.

## Data Model and Flexible Configuration

### Schema Evolution via JSON Config

The credential system supports provider-specific customizations through a flexible `config` JSON bag introduced in migration 15. This field (lines 73–78 in [`credential.py`](https://github.com/lfnovo/open-notebook/blob/main/credential.py)) holds evolving provider options without requiring database schema changes. The JSON structure accommodates new authentication parameters while maintaining strict type safety for core fields like `provider` and `modalities`.

### Field Mirroring Mechanism

To maintain backward compatibility, convenience fields like `num_ctx` automatically synchronize with the JSON config. The `_prepare_save_data` method (lines 44–57) mirrors these fields into the `config` object before saving, while `_mirror_config_to_fields` (lines 86–101) restores them when loading records. This bidirectional synchronization ensures the API can access common fields directly while preserving the flexibility of the underlying JSON structure.

## Relationship with AI Models

Credentials link to AI models through the `credential` foreign-key in the `model` table. The `Credential.get_linked_models` method (lines 15–25 in [`credential.py`](https://github.com/lfnovo/open-notebook/blob/main/credential.py)) enables the system to traverse these relationships efficiently. When deleting or updating credentials, the API automatically cascades changes to associated models or migrates them to alternative credentials, preventing orphaned model configurations.

## Migration from Legacy Singleton Architecture

### The Old ProviderConfig Pattern

Originally, authentication data lived in a singleton `ProviderConfig` class defined in [`open_notebook/domain/provider_config.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/provider_config.py). This legacy structure maintained a dictionary of multiple `ProviderCredential` objects per provider within a single record, violating database-first principles by coupling multiple authentication contexts together.

### Automated Migration Path

The system provides a migration endpoint at `/credentials/migrate-from-provider-config` that invokes `svc_migrate_from_provider_config` (lines 14–18 in [`api/routers/credentials.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/credentials.py)). This tooling automatically extracts each legacy entry into individual `credential` records, preserving the relationship data while transitioning to the normalized database structure. The router also handles cascade deletion logic (lines 23–45 and 71–124) to manage credential lifecycle events safely.

## Implementation Examples

### Creating and Saving Credentials

```python

# Creating a new credential (async)

from open_notebook.domain.credential import Credential
from pydantic import SecretStr

cred = Credential(
    name="Prod‑OpenAI",
    provider="openai",
    modalities=["language", "embedding"],
    api_key=SecretStr("sk-..."),
    base_url="https://api.openai.com/v1",
    num_ctx=8192,
)
await cred.save()          # encrypted API key stored in DB

```

### Loading and Using Credentials

```python

# Loading a credential and using it with Esperanto

cred = await Credential.get("<credential-id>")
config = cred.to_esperanto_config()   # builds the dict for AIFactory

# Example: AIFactory.create_chat(**config)

```

### Deleting with Cascade or Migration

```python

# Deleting a credential and migrating its models

await cred.delete()                     # cascade‑deletes linked models

# Or migrate:

await cred.delete(migrate_to="<other-id>")

```

### Querying by Provider

```python

# Listing all credentials for a provider (used by the API)

creds = await Credential.get_by_provider("anthropic")
for c in creds:
    print(c.name, c.provider, c.api_key)   # api_key is a SecretStr (decrypted)

```

## Summary

- **Dedicated Table Structure**: Credentials reside in the `credential` table as first-class SurrealDB records rather than configuration singletons.
- **Encryption at Rest**: The `encrypt_value` and `decrypt_value` functions in [`open_notebook/utils/encryption.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/encryption.py) secure API keys before persistence.
- **Flexible Schema**: The `config` JSON field supports provider-specific evolution without database migrations, managed by `_prepare_save_data` and `_mirror_config_to_fields`.
- **Model Relationships**: The `credential` foreign-key in the `model` table enables automatic cascading and migration via `get_linked_models`.
- **Migration Support**: The `/credentials/migrate-from-provider-config` endpoint transitions legacy `ProviderConfig` data to individual records without data loss.

## Frequently Asked Questions

### How does open-notebook encrypt API keys in the database?

The system uses `encrypt_value` and `decrypt_value` functions from [`open_notebook/utils/encryption.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/encryption.py) to secure the `api_key` field. These functions are called automatically during save and load operations within the `Credential` model (lines 34–40 and 62–68), ensuring the sensitive value never exists in plaintext within SurrealDB.

### What is the difference between the legacy ProviderConfig and the new credential system?

`ProviderConfig` was a singleton class storing multiple credentials in a single record as a dictionary, while the new database-first credential storage mechanism uses a dedicated `credential` table with individual records per authentication context. This normalization enables better querying, individual credential lifecycle management, and proper foreign-key relationships with AI models.

### Can I migrate existing credentials without losing linked models?

Yes, the migration endpoint `/credentials/migrate-from-provider-config` preserves model relationships by extracting legacy entries into individual `credential` records while maintaining the `credential` foreign-key references in the `model` table. The `svc_migrate_from_provider_config` function handles this transition automatically.

### How does the config JSON field support different AI providers?

The `config` field stores provider-specific parameters as unstructured JSON, allowing the schema to evolve without database migrations. The `_prepare_save_data` method (lines 44–57) synchronizes convenience fields into this JSON before saving, while `_mirror_config_to_fields` (lines 86–101) restores them upon loading, enabling both flexibility and type-safe access.