How to Migrate Provider Configurations from Environment Variables to Database in Open Notebook

Migrating provider configurations from environment variables to the database in Open Notebook involves moving API keys from legacy env vars into the ProviderConfig singleton record in SurrealDB, enabling encrypted storage and multi-credential support.

The lfnovo/open-notebook project has transitioned from static environment variable configuration to a dynamic database-driven approach. This migration allows you to manage multiple API keys per provider, set defaults, and encrypt credentials at rest. The new system uses a singleton pattern in SurrealDB to ensure a single source of truth for all AI provider credentials.

Understanding the Provider Configuration Architecture

The Legacy Environment Variable Approach

Previously, Open Notebook relied exclusively on environment variables like OPENAI_API_KEY to authenticate with AI providers. While simple, this approach limited you to one credential per provider and required application restarts to rotate keys. Environment variables also lack built-in encryption and cannot support complex scenarios like different keys for development, staging, and production environments.

The Database-First ProviderConfig Model

The modern architecture centers on the ProviderConfig class in open_notebook/domain/provider_config.py. This singleton record uses record_id = "open_notebook:provider_configs" to guarantee exactly one configuration document exists in SurrealDB. The model stores credentials in a credentials: Dict[str, List[ProviderCredential]] structure, allowing multiple ProviderCredential objects per provider.

Each ProviderCredential (defined around lines 22-44) includes an is_default field to designate which key to use when no explicit ID is specified. The ProviderConfig class provides helper methods including get_default_config, get_config, add_config, delete_config, and set_default_config to manage these credentials programmatically.

Migrating from Environment Variables to SurrealDB

To migrate existing provider configurations, you read the current environment variables and persist them as encrypted database records. The migration is a one-time operation after which the application reads exclusively from SurrealDB.

import os
from pydantic import SecretStr
from open_notebook.domain.provider_config import ProviderConfig, ProviderCredential

async def migrate_env_to_db():
    """Migrate legacy env vars to SurrealDB singleton."""
    config = await ProviderConfig.get_instance()
    
    # Check for OpenAI key in environment

    if os.getenv("OPENAI_API_KEY"):
        openai_cred = ProviderCredential(
            id="env-migrated-1",
            name="Production OpenAI",
            provider="openai",
            is_default=True,
            api_key=SecretStr(os.getenv("OPENAI_API_KEY")),
            base_url=None,
            model="gpt-4o",
        )
        config.add_config("openai", openai_cred)
    
    # Persist to database with automatic encryption

    await config.save()

The save() method in open_notebook/domain/provider_config.py (lines 129-138) handles serialization and calls encrypt_value from open_notebook/utils/encryption.py when OPEN_NOTEBOOK_ENCRYPTION_KEY is configured. This ensures API keys remain encrypted at rest without manual intervention.

Working with ProviderConfig in Your Application

Loading the Singleton Instance

When the application starts, api/main.py loads the configuration via ProviderConfig.get_instance(). This method pulls the latest data from SurrealDB on every call, ensuring all components use current credentials without caching stale values.

from open_notebook.domain.provider_config import ProviderConfig

async def initialize_providers():
    """Load provider configuration on startup."""
    config = await ProviderConfig.get_instance()
    return config

Retrieving Default Credentials

Downstream services like open_notebook/ai/key_provider.py retrieve credentials through the singleton. Use get_default_config(provider) to fetch the active credential for a specific provider, or get_config(provider, config_id) to select a specific key.

config = await ProviderConfig.get_instance()

# Get the default OpenAI credential

openai_cred = config.get_default_config("openai")
if openai_cred:
    api_key = openai_cred.api_key.get_secret_value()
    # Use with your AI client...

Adding and Updating Credentials

You can add new credentials without restarting the application. The add_config method appends to the provider's credential list, while set_default_config updates the is_default flag.

# Create a development credential

dev_cred = ProviderCredential(
    id="dev-1",
    name="Development OpenAI",
    provider="openai",
    is_default=False,
    api_key=SecretStr("sk-dev-key-12345"),
    base_url=None,
    model="gpt-4o-mini",
)

# Add to configuration

config.add_config("openai", dev_cred)

# Promote to default

config.set_default_config("openai", "dev-1")

# Persist changes

await config.save()

Security Features and Encryption

The ProviderConfig model implements transparent encryption through the _prepare_save_data method (lines 15-19). When OPEN_NOTEBOOK_ENCRYPTION_KEY is set in your environment, the system automatically encrypts sensitive values before writing to SurrealDB and decrypts them on retrieval. This eliminates the security risks of storing API keys in plain text environment variables or configuration files.

The encryption layer resides in open_notebook/utils/encryption.py and provides encrypt_value and decrypt_value functions used by the domain model. This approach keeps credentials secure while maintaining the convenience of database-driven configuration.

Summary

• Open Notebook uses a singleton ProviderConfig record in SurrealDB to manage all AI provider credentials, replacing static environment variables.
• The ProviderCredential model supports multiple keys per provider with a default flag for automatic selection.
• Encryption is automatic when OPEN_NOTEBOOK_ENCRYPTION_KEY is configured, securing API keys at rest.
• ProviderConfig.get_instance() loads fresh data from the database on every call, ensuring no stale credentials.
• Migration requires a one-time script to read existing env vars and call add_config followed by save().

Frequently Asked Questions

How do I migrate my existing OpenAI API key from environment variables to the database?

Read the value from OPENAI_API_KEY, create a ProviderCredential object with SecretStr wrapping the key, and pass it to config.add_config("openai", credential). Call await config.save() to persist the encrypted value to SurrealDB. Once saved, you can remove the environment variable from your deployment.

Can I use multiple API keys for the same provider simultaneously?

Yes. The credentials dictionary in ProviderConfig stores a list of ProviderCredential objects per provider. You can maintain separate keys for different environments or projects, then select specific credentials using get_config(provider, config_id) or set different defaults per deployment.

Is the database-stored configuration secure?

Yes. When you configure OPEN_NOTEBOOK_ENCRYPTION_KEY, the save() method automatically encrypts API keys using encrypt_value from open_notebook/utils/encryption.py before writing to SurrealDB. The keys remain encrypted at rest and are only decrypted when retrieved by the application.

Do I need to restart the application after updating credentials?

No. The ProviderConfig.get_instance() method fetches the latest configuration from SurrealDB on every invocation. Services like open_notebook/ai/key_provider.py always receive current credentials, allowing zero-downtime key rotation and updates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}