How Open Notebook Implements Database-First API Key Storage and Encryption

Open Notebook stores AI provider credentials as encrypted SurrealDB records rather than environment variables, using Fernet encryption with a user-supplied key and supporting multiple credentials per provider with graceful decryption error handling.

The open-notebook repository by lfnovo implements a database-first credential management system that moves API keys from global environment variables into encrypted SurrealDB records. This architecture enables per-user or per-project credential isolation while maintaining backward compatibility with legacy providers that still rely on os.environ lookups.

The Credential Domain Model

The Credential class in open_notebook/domain/credential.py defines the schema for storing provider-specific settings. Unlike traditional configurations that use a singleton ProviderConfig, this model treats each credential as an independent database record with the table name credential.

The domain model tracks:

• name: Human-readable identifier
• provider: AI provider slug (e.g., "openai", "anthropic")
• modalities: Supported capabilities (language, embedding, image)
• api_key: Encrypted storage of the secret token
• endpoints: Custom base URLs for provider APIs

Encryption at Rest

Fernet Key Derivation

The system never persists API keys in plaintext. Instead, open_notebook/utils/encryption.py provides encrypt_value and decrypt_value functions that use Fernet symmetric encryption. The encryption key derives from the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable, which accepts any user-supplied string and converts it into a valid Fernet key.

Pre-Save Encryption Hooks

When a credential is saved, the _prepare_save_data method in open_notebook/domain/credential.py automatically encrypts the api_key field before generating the INSERT or UPDATE query. If the API key is null, the system stores null rather than an empty encrypted string.

from open_notebook.domain.credential import Credential
from pydantic import SecretStr

cred = Credential(
    name="My OpenAI Key",
    provider="openai",
    modalities=["language", "embedding"],
    api_key=SecretStr("sk-REALLY-SECRET"),
)
await cred.save()  # Encryption happens automatically before database write

Secure Retrieval and Decryption

SecretStr Handling

During retrieval, Credential.get and Credential.get_all invoke _from_db_row, which detects encrypted strings and decrypts them using decrypt_value. The decrypted result wraps in a Pydantic SecretStr to prevent accidental exposure in logs or stack traces.

Graceful Error Handling for Key Rotation

If decryption fails—such as when the OPEN_NOTEBOOK_ENCRYPTION_KEY changes—the code records a decryption_error attribute on the returned object and substitutes a placeholder UNDECRYPTABLE value. This allows the UI to surface a friendly warning without crashing the application or exposing malformed data.

# Handle decryption error gracefully after key rotation

try:
    cred = await Credential.get("<old-credential-id>")
    if hasattr(cred, 'decryption_error'):
        print("Warning: Stored key cannot be decrypted")
except ValueError as e:
    print("Credential unusable:", e)

Automatic Re-Hydration

The save method remembers the original SecretStr before delegating to the base ObjectModel.save. After the database round-trip, it restores the original SecretStr (or decrypts the returned string if the database echo contains encrypted data), ensuring the in-memory object remains usable for subsequent operations.

Integration with AI Models

Linking Credentials to Models

A Model record references a credential via the credential foreign key. The Credential.get_linked_models method queries the model table for all rows pointing to a specific credential ID, enabling cleanup validation before deletion.

When instantiating a model, credential.to_esperanto_config() produces a dictionary of provider-specific parameters (API key, endpoint, project ID) passed directly to Esperanto's AIFactory methods, bypassing environment variable lookups entirely.

from open_notebook.domain.credential import Credential
from open_notebook.ai.models import Model

cred = await Credential.get("<credential-id>")
config = cred.to_esperanto_config()   # Returns {'api_key': 'sk-REALLY-SECRET', ...}

model = await Model.create(
    name="gpt-4o", 
    provider="openai", 
    credential=cred.id
)

# ModelManager receives config directly, not env vars

Environment Variable Fallback

For legacy providers that still read from os.environ, open_notebook/ai/provision.py contains provision_provider_keys. This helper populates relevant *_API_KEY environment variables from matching credential records before initializing the provider, ensuring seamless operation during the transition to database-first storage.

Migration from Legacy Configuration

Older installations used a single ProviderConfig singleton. The migration scripts in open_notebook/database/migrate.py automatically create Credential records for each existing configuration and update model records to point at the new credential IDs. This preserves continuity while moving to the encrypted, database-first approach described in the migration notes.

# List all credentials for a given provider

creds = await Credential.get_by_provider("anthropic")
for c in creds:
    print(c.name, c.api_key.get_secret_value() if c.api_key else "none")

Summary

• Database-first storage: Credentials live as independent records in SurrealDB rather than global environment variables, supporting multiple keys per provider.
• Fernet encryption: open_notebook/utils/encryption.py encrypts API keys at rest using a user-controlled OPEN_NOTEBOOK_ENCRYPTION_KEY.
• Safe decryption: Credential.get decrypts values into SecretStr objects and handles key rotation errors gracefully with UNDECRYPTABLE placeholders.
• Model integration: The to_esperanto_config() method provides direct configuration to Esperanto's AIFactory, while provision_provider_keys maintains backward compatibility with legacy environment-based providers.
• Automatic migration: Legacy ProviderConfig singletons convert automatically to encrypted Credential records.

Frequently Asked Questions

How is the encryption key configured?

Set the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable to any string before starting the application. The system derives a valid Fernet key from this value. Changing this key after credentials exist will trigger decryption errors until you re-encrypt the records.

Can I store multiple API keys for the same provider?

Yes. Because credentials are database records rather than singleton configuration objects, you can create unlimited Credential records with the same provider slug (e.g., "openai"). Each Model instance references a specific credential ID, enabling per-project or per-user key isolation.

What happens if I lose my encryption key?

Without the original OPEN_NOTEBOOK_ENCRYPTION_KEY, the stored API keys cannot be decrypted. The system returns UNDECRYPTABLE values and sets a decryption_error flag on the credential object, allowing you to identify which records need new API keys generated from the provider dashboard.

How does the system handle providers that only read from environment variables?

The provision_provider_keys helper in open_notebook/ai/provision.py populates environment variables from stored credentials before initializing legacy providers. This allows the database-first architecture to coexist with libraries that expect OPENAI_API_KEY or similar variables in os.environ.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}