How Open Notebook Handles Credential Encryption and Secure Storage in the Database

Open Notebook uses Fernet symmetric encryption to secure API keys at the field level, storing encrypted tokens in SurrealDB while keeping the encryption key in the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable.

The lfnovo/open-notebook repository implements a transparent encryption layer that protects AI provider credentials at rest. Instead of storing API keys in plaintext, the application encrypts sensitive values before persistence and decrypts them automatically upon retrieval, ensuring that database dumps or logs never expose raw secrets.

Field-Level Encryption Architecture

Open Notebook applies field-level encryption rather than full-database encryption. This design keeps non-sensitive metadata queryable while securing only the confidential API keys.

The Credential Domain Model

The Credential class in open_notebook/domain/credential.py inherits from ObjectModel and defines the schema for provider credentials. The api_key field uses Pydantic’s SecretStr type to prevent accidental logging of cleartext values in memory. When you call save(), the model delegates to _prepare_save_data() (lines 27–44) to transform the instance into a database-ready dictionary.

Encryption Lifecycle

On save, _prepare_save_data() extracts the secret string from the SecretStr, passes it to encrypt_value(), and stores the resulting Fernet token in the database. This ensures the raw key never touches the disk.

On load, the overridden get() method (lines 57–69) and _from_db_row() (lines 78–84) reverse the process. They intercept the database row, extract the encrypted token, decrypt it via decrypt_value(), and wrap the result in a SecretStr before returning the populated Credential instance.

Encryption Implementation Details

The cryptographic routines live in open_notebook/utils/encryption.py and provide a thin wrapper around the Fernet specification.

Fernet Key Derivation from Environment

The system expects a master secret in the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable (or a Docker secret with the same name). The internal _ensure_fernet_key() function derives a valid Fernet key by SHA‑256 hashing the supplied string and base64-encoding the digest. This derived key is then cached and reused for all subsequent operations.

Encrypt and Decrypt Operations

• encrypt_value(value): Accepts a UTF‑8 string, encrypts it with Fernet, and returns a URL‑safe base64 token (e.g., gAAAAAB...).
• decrypt_value(value): Attempts Fernet decryption. If the payload is a valid Fernet token, it returns the plaintext string. If decryption fails because the payload looks like a token but has an invalid signature or wrong key, it raises a clear error. For legacy compatibility, unencrypted plaintext values pass through unchanged instead of crashing the reader.

Flexible Configuration Storage

Beyond the encrypted API key, providers often require non-sensitive settings such as endpoint URLs, project IDs, or context window sizes. These values live in a flexible JSON field called config.

During serialization, _prepare_save_data() (lines 44–57) mirrors top‑level convenience fields like num_ctx into the config bag before writing to SurrealDB. On deserialization, _mirror_config_to_fields() (lines 86–100) flattens the JSON back into model attributes, allowing you to access settings as strongly‑typed properties while keeping the database schema adaptable.

Error Handling and Resilience

If decrypt_value() fails—perhaps because the OPEN_NOTEBOOK_ENCRYPTION_KEY has rotated or the database contains a corrupted token—the system logs a warning and returns a placeholder credential containing an error message. This prevents the entire application from crashing on a single bad record and allows administrators to identify and re‑enter the affected credentials without downtime.

Usage Examples

Creating and persisting a new credential automatically encrypts the API key:

from open_notebook.domain.credential import Credential
from pydantic import SecretStr

cred = Credential(
    name="Prod‑OpenAI",
    provider="openai",
    modalities=["language", "embedding"],
    api_key=SecretStr("sk‑my‑very‑secret‑key"),
    num_ctx=16384,
)
await cred.save()   # encrypts api_key → DB

Loading credentials decrypts them transparently:

from open_notebook.domain.credential import Credential

creds = await Credential.get_by_provider("openai")
for c in creds:
    # c.api_key is a SecretStr containing the clear‑text key

    print(c.name, c.api_key.get_secret_value())

You can also use the encryption utilities directly for custom workflows:

from open_notebook.utils.encryption import encrypt_value, decrypt_value

encrypted = encrypt_value("my‑plain‑token")
print(encrypted)               # e.g., "gAAAAAB..."

plain = decrypt_value(encrypted)
assert plain == "my‑plain‑token"

Summary

• Field-level encryption in open_notebook/domain/credential.py protects API keys via Fernet symmetric encryption before they reach SurrealDB.
• Transparent lifecycle methods (_prepare_save_data and _from_db_row) handle encrypt‑on‑write and decrypt‑on‑read automatically.
• Environment‑derived keys are sourced from OPEN_NOTEBOOK_ENCRYPTION_KEY and hashed via SHA‑256 to produce the Fernet key.
• Flexible configuration allows provider‑specific settings to reside in a JSON config field while remaining accessible as model attributes.
• Graceful degradation ensures that decryption failures return placeholder objects rather than crashing the application.

Frequently Asked Questions

What encryption algorithm does Open Notebook use?

Open Notebook uses Fernet symmetric encryption from the Python cryptography library. Fernet provides authenticated encryption, meaning the ciphertext is protected against tampering and can only be read with the exact key used for encryption. The implementation resides in open_notebook/utils/encryption.py.

Where is the encryption key stored?

The encryption key is stored outside the database in the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable. If you deploy via Docker, you can also use a Docker secret with the same name. The application derives the final Fernet key by SHA‑256 hashing this value, ensuring that even a short passphrase expands to the required 32‑byte key length.

What happens if decryption fails?

If decrypt_value() cannot decrypt a credential—due to a key rotation, corrupted data, or mismatched environment—the system logs a warning and returns a placeholder Credential object containing an error message. This prevents the application from crashing and allows users to identify and re‑configure the affected provider settings.

How does the flexible config field work?

The config field is a JSON column in SurrealDB that stores non‑sensitive provider settings such as num_ctx or custom endpoints. When you save a Credential, _prepare_save_data() copies top‑level fields into this JSON bag. When you load it, _mirror_config_to_fields() restores them to model attributes, giving you type safety while keeping the database schema adaptable to new provider requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}