Database-First Credential Storage Mechanism in Open-Notebook: SurrealDB Implementation

Open-Notebook stores AI-provider authentication data as first-class records in SurrealDB, encrypting sensitive values at rest and linking credentials to AI models through foreign-key relationships rather than singleton configuration.

The open-notebook project implements a robust database-first credential storage mechanism that treats authentication data as native database records rather than application configuration. This architecture leverages SurrealDB as the primary datastore, storing each credential as a distinct row in the credential table with full encryption support and flexible schema evolution capabilities.

Core Architecture of the Credential System

SurrealDB Table Structure

In open_notebook/domain/credential.py, the Credential model extends ObjectModel and explicitly defines table_name = "credential" (line 37). This design establishes credentials as first-class database citizens rather than singleton configuration objects. Each record represents a single authentication context for an AI provider, with dedicated columns for api_key, base_url, endpoint_*, project, and location.

Encryption at Rest

Sensitive values receive automatic encryption before persistence. The model utilizes encrypt_value and decrypt_value helper functions from open_notebook/utils/encryption.py (referenced in lines 26–27, 34–40, and 62–68 of credential.py). This ensures API keys never exist in plaintext within the database, protecting against credential leakage through raw database access.

Data Model and Flexible Configuration

Schema Evolution via JSON Config

The credential system supports provider-specific customizations through a flexible config JSON bag introduced in migration 15. This field (lines 73–78 in credential.py) holds evolving provider options without requiring database schema changes. The JSON structure accommodates new authentication parameters while maintaining strict type safety for core fields like provider and modalities.

Field Mirroring Mechanism

To maintain backward compatibility, convenience fields like num_ctx automatically synchronize with the JSON config. The _prepare_save_data method (lines 44–57) mirrors these fields into the config object before saving, while _mirror_config_to_fields (lines 86–101) restores them when loading records. This bidirectional synchronization ensures the API can access common fields directly while preserving the flexibility of the underlying JSON structure.

Relationship with AI Models

Credentials link to AI models through the credential foreign-key in the model table. The Credential.get_linked_models method (lines 15–25 in credential.py) enables the system to traverse these relationships efficiently. When deleting or updating credentials, the API automatically cascades changes to associated models or migrates them to alternative credentials, preventing orphaned model configurations.

Migration from Legacy Singleton Architecture

The Old ProviderConfig Pattern

Originally, authentication data lived in a singleton ProviderConfig class defined in open_notebook/domain/provider_config.py. This legacy structure maintained a dictionary of multiple ProviderCredential objects per provider within a single record, violating database-first principles by coupling multiple authentication contexts together.

Automated Migration Path

The system provides a migration endpoint at /credentials/migrate-from-provider-config that invokes svc_migrate_from_provider_config (lines 14–18 in api/routers/credentials.py). This tooling automatically extracts each legacy entry into individual credential records, preserving the relationship data while transitioning to the normalized database structure. The router also handles cascade deletion logic (lines 23–45 and 71–124) to manage credential lifecycle events safely.

Implementation Examples

Creating and Saving Credentials

# Creating a new credential (async)

from open_notebook.domain.credential import Credential
from pydantic import SecretStr

cred = Credential(
    name="Prod‑OpenAI",
    provider="openai",
    modalities=["language", "embedding"],
    api_key=SecretStr("sk-..."),
    base_url="https://api.openai.com/v1",
    num_ctx=8192,
)
await cred.save()          # encrypted API key stored in DB

Loading and Using Credentials

# Loading a credential and using it with Esperanto

cred = await Credential.get("<credential-id>")
config = cred.to_esperanto_config()   # builds the dict for AIFactory

# Example: AIFactory.create_chat(**config)

Deleting with Cascade or Migration

# Deleting a credential and migrating its models

await cred.delete()                     # cascade‑deletes linked models

# Or migrate:

await cred.delete(migrate_to="<other-id>")

Querying by Provider

# Listing all credentials for a provider (used by the API)

creds = await Credential.get_by_provider("anthropic")
for c in creds:
    print(c.name, c.provider, c.api_key)   # api_key is a SecretStr (decrypted)

Summary

• Dedicated Table Structure: Credentials reside in the credential table as first-class SurrealDB records rather than configuration singletons.
• Encryption at Rest: The encrypt_value and decrypt_value functions in open_notebook/utils/encryption.py secure API keys before persistence.
• Flexible Schema: The config JSON field supports provider-specific evolution without database migrations, managed by _prepare_save_data and _mirror_config_to_fields.
• Model Relationships: The credential foreign-key in the model table enables automatic cascading and migration via get_linked_models.
• Migration Support: The /credentials/migrate-from-provider-config endpoint transitions legacy ProviderConfig data to individual records without data loss.

Frequently Asked Questions

How does open-notebook encrypt API keys in the database?

The system uses encrypt_value and decrypt_value functions from open_notebook/utils/encryption.py to secure the api_key field. These functions are called automatically during save and load operations within the Credential model (lines 34–40 and 62–68), ensuring the sensitive value never exists in plaintext within SurrealDB.

What is the difference between the legacy ProviderConfig and the new credential system?

ProviderConfig was a singleton class storing multiple credentials in a single record as a dictionary, while the new database-first credential storage mechanism uses a dedicated credential table with individual records per authentication context. This normalization enables better querying, individual credential lifecycle management, and proper foreign-key relationships with AI models.

Can I migrate existing credentials without losing linked models?

Yes, the migration endpoint /credentials/migrate-from-provider-config preserves model relationships by extracting legacy entries into individual credential records while maintaining the credential foreign-key references in the model table. The svc_migrate_from_provider_config function handles this transition automatically.

How does the config JSON field support different AI providers?

The config field stores provider-specific parameters as unstructured JSON, allowing the schema to evolve without database migrations. The _prepare_save_data method (lines 44–57) synchronizes convenience fields into this JSON before saving, while _mirror_config_to_fields (lines 86–101) restores them upon loading, enabling both flexibility and type-safe access.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}