How Open Notebook Implements a Fernet-Encrypted, Database-First Credential System

Open Notebook stores AI-provider credentials in SurrealDB using Fernet symmetric encryption, ensuring API keys are encrypted before persistence and only decrypted when loaded into memory as Pydantic SecretStr objects.

The lfnovo/open-notebook repository employs a "database-first" architecture where the credential table serves as the single source of truth for AI provider configurations. This design guarantees that sensitive material never touches the disk in plain text and is only decrypted transiently during runtime operations.

Architecture Overview

The credential system consists of three tightly-coupled components implemented across specific modules:

• open_notebook/domain/credential.py – The domain model handling schema definition, database mapping, and encryption hooks.
• open_notebook/utils/encryption.py – Fernet-based symmetric encryption utilities with lazy key loading.
• open_notebook/database/repository.py – Generic async query wrappers (repo_query, ensure_record_id) that execute SurrealQL.

This tripartite structure separates cryptographic concerns from business logic while maintaining the database-first constraint: credentials are persisted before use, and encryption/decryption occurs at the model layer, not the database layer.

The Credential Domain Model

Located in open_notebook/domain/credential.py, the Credential class inherits from ObjectModel (providing generic CRUD methods) and manages the lifecycle of encrypted API keys.

Encryption on Save with _prepare_save_data

When await cred.save() is invoked, the model intercepts the payload through _prepare_save_data. This method encrypts the api_key (stored internally as Optional[SecretStr]) using encrypt_value before the record is sent to SurrealDB via repo_query. After persistence, the original SecretStr is restored to the instance so callers retain access to the plain key in memory while the database retains only the Fernet token.

from pydantic import SecretStr
from open_notebook.domain.credential import Credential

# Create with SecretStr to prevent accidental logging

cred = Credential(
    name="Prod-OpenAI",
    provider="openai",
    modalities=["language", "embedding"],
    api_key=SecretStr("sk-my-secret-key"),
)

# Persists encrypted data; original SecretStr remains in memory

await cred.save()

Decryption on Load with _from_db_row

Retrieval methods—Credential.get, Credential.get_by_provider, and Credential.get_all—delegate to repo_query and then pass raw database rows to the _from_db_row classmethod. This method decrypts the stored string using decrypt_value and re-wraps the result in SecretStr. If decryption fails (e.g., due to a rotated encryption key), the method populates a decryption_error field on a placeholder credential object, allowing the UI to surface actionable warnings without crashing the application.

# Retrieve and decrypt automatically

loaded = await Credential.get(cred.id)
print(loaded.api_key.get_secret_value())  # Plain text accessible only here

Integration with AI Providers via to_esperanto_config

The model exposes to_esperanto_config() to translate stored fields into the configuration dictionary expected by the Esperanto AI-provider factory. This method bridges the encrypted database record with runtime LLM/embedding model initialization.

config = loaded.to_esperanto_config()

# Returns dict with api_key, endpoint, etc., ready for ModelManager.create_llm()

Fernet Encryption Utilities

The open_notebook/utils/encryption.py module encapsulates all cryptographic operations using the Fernet specification (AES-128-CBC with HMAC-SHA256 authentication).

Key Derivation Strategy

Since Fernet requires a 32-byte URL-safe base64-encoded key, the _ensure_fernet_key function accepts any string from the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable (or its Docker secret variant), hashes it with SHA-256, and encodes the digest to meet Fernet's constraints. This allows operators to use memorable passphrases while maintaining cryptographic strength.

# Key is derived lazily on first encrypt/decrypt call

from open_notebook.utils.encryption import get_fernet

fernet = get_fernet()  # Uses OPEN_NOTEBOOK_ENCRYPTION_KEY from env

Lazy Loading and Token Detection

The get_fernet() function implements lazy loading to prevent startup crashes when the environment variable is missing. The companion function looks_like_fernet_token validates token length and structure, enabling the decryption routine to distinguish between encrypted Fernet tokens and legacy plaintext values.

Graceful Degradation for Legacy Data

The decrypt_value function handles three states:

1. Valid Fernet tokens: Decrypts and returns the UTF-8 string.
2. Malformed tokens: Raises ValueError with explicit instructions to check OPEN_NOTEBOOK_ENCRYPTION_KEY.
3. Legacy plaintext: Returns the original string unchanged, ensuring backward compatibility during migrations.

Database-First Persistence Flow

The end-to-end flow demonstrates how the system maintains its "database-first" guarantee:

1. Instantiation: Create Credential with api_key as SecretStr.
2. Pre-save: _prepare_save_data calls encrypt_value, transforming the plain key into a Fernet token.
3. Persistence: repo_query executes SurrealQL to store the encrypted record in the credential table.
4. Retrieval: Credential.get fetches the row and passes it through _from_db_row, which invokes decrypt_value.
5. Runtime Use: The decrypted SecretStr is passed via to_esperanto_config() to the AI factory.

If the encryption key changes between save and load operations, the retrieval method catches the decryption failure and surfaces it through the decryption_error field rather than exposing stack traces or key material.

Security Design Decisions

Why Fernet? The implementation chooses Fernet over raw AES because it provides authenticated encryption out of the box, preventing both unauthorized reading and tampering of stored credentials.

Key Management: The encryption key is never hard-coded. It must be supplied via environment variables or Docker secrets, and the SHA-256 derivation ensures compatibility with arbitrary passphrase lengths.

Memory Safety: By using Pydantic's SecretStr, the system prevents accidental logging of API keys in stack traces or application logs. The plain text exists only transiently in memory during encryption/decryption operations.

Error Handling: Decryption failures result in descriptive user-facing messages via the decryption_error field, never propagating raw cryptographic exceptions or key material.

Summary

• Open Notebook uses SurrealDB as the source of truth for credentials, encrypting data before it reaches the database.
• The Credential model in open_notebook/domain/credential.py handles transparent encryption via _prepare_save_data and decryption via _from_db_row.
• open_notebook/utils/encryption.py provides Fernet-based utilities with SHA-256 key derivation from the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable.
• Pydantic SecretStr ensures API keys are never accidentally logged, while lazy loading prevents startup failures if encryption keys are missing.
• The system gracefully handles legacy plaintext values and provides clear error messages when decryption fails due to key rotation.

Frequently Asked Questions

How does Open Notebook handle credential encryption key rotation?

If the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable changes after credentials are saved, subsequent calls to Credential.get() will fail to decrypt existing Fernet tokens. The _from_db_row method catches these failures and populates a decryption_error field on the credential object, allowing administrators to identify which records need to be re-saved with the new key through the UI without crashing the application.

Why does the implementation use SHA-256 to derive the Fernet key?

Fernet requires a 32-byte URL-safe base64-encoded key, but memorably passphrases rarely meet this format requirement. The _ensure_fernet_key function in open_notebook/utils/encryption.py hashes the environment variable value with SHA-256 and base64-encodes the digest, producing a valid Fernet key from any arbitrary string while maintaining deterministic derivation (the same passphrase always produces the same key).

What happens if the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable is missing?

The get_fernet() function uses lazy loading: it only attempts to read the environment variable when encryption or decryption is actually attempted. This prevents startup crashes. However, any attempt to save or load credentials will raise an error at that specific point, ensuring the application can boot in environments where credentials are not immediately needed (e.g., read-only viewer modes).

How does the system distinguish between encrypted and legacy plaintext credentials?

The decrypt_value function in open_notebook/utils/encryption.py utilizes looks_like_fernet_token to check if the stored value matches Fernet's structure (URL-safe base64 with specific length and padding). If the token does not look like Fernet, it is returned as-is, ensuring backward compatibility with unencrypted legacy data while new entries are automatically encrypted upon saving.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}