How Open Notebook Manages Encryption Keys for Storing Sensitive Credentials

Open Notebook encrypts provider API keys at rest using Fernet (AES-128-CBC with HMAC-SHA256) by deriving the encryption key from the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable, ensuring sensitive credentials are never stored in plaintext.

The lfnovo/open-notebook repository implements a robust field-level encryption scheme to protect sensitive provider credentials stored in SurrealDB. This system uses a lazy-loaded Fernet key derived from a configurable environment variable, encrypting API keys before persistence and decrypting them transparently upon retrieval. Understanding how encryption keys are managed for storing sensitive credentials is essential for operators deploying this notebook-based AI interface.

Encryption Architecture Overview

The encryption system relies on two primary components working together: a low-level cryptographic utility module and a domain model that orchestrates automatic encryption for credential records.

The Encryption Utilities Layer

Located in open_notebook/utils/encryption.py, this module handles all cryptographic operations. It provides the encrypt_value and decrypt_value functions that wrap the Fernet implementation from the cryptography library.

The key management flow follows this sequence:

1. get_secret_from_env reads either the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable or the file path specified in OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE (for Docker secrets).
2. _ensure_fernet_key hashes the raw secret using SHA-256 and encodes the digest to a URL-safe Base64 string, creating a valid Fernet key.
3. get_fernet returns a cached Fernet instance using lazy loading, allowing the application to start even if the key is not yet configured.

The Credential Model Integration

The open_notebook/domain/credential.py file implements the domain model that persists provider credentials. When a Credential instance is saved, the _prepare_save_data method extracts the SecretStr value from the api_key field and calls encrypt_value to generate ciphertext before storing it in SurrealDB.

During retrieval, methods like get, get_all, and _from_db_row invoke decrypt_value to transform the stored ciphertext back into clear-text SecretStr objects. If decryption fails due to a missing or incorrect key, the system creates an "UNDECRYPTABLE" placeholder and attaches an error message, preventing application crashes.

How the Encryption Key Lifecycle Works

Configuration and Key Derivation

Operators configure the encryption key by setting either the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable or providing a file path via OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE. The first call to any encryption function triggers lazy loading: the raw value is hashed with SHA-256 and encoded to create the Fernet key material.

# Configure via environment variable

export OPEN_NOTEBOOK_ENCRYPTION_KEY="my-super-secret-passphrase"

# Or use Docker secrets

export OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE="/run/secrets/notebook_key"

This derivation process ensures that the actual passphrase never sits in memory as the encryption key; only the derived hash is used for cryptographic operations.

Encryption During Persistence

When a Credential instance is saved, the plaintext API key is automatically encrypted. The system stores the resulting ciphertext—a URL-safe Base64 Fernet token—in the api_key column of SurrealDB.

from open_notebook.domain.credential import Credential
from pydantic import SecretStr
import asyncio

async def create_credential():
    cred = Credential(
        name="Prod OpenAI",
        provider="openai",
        modalities=["language", "embedding"],
        api_key=SecretStr("sk-prod-openai-xyz"),
    )
    await cred.save()  # Automatically encrypts before DB write

asyncio.run(create_credential())

Decryption on Retrieval

Upon loading a credential, the decryption process happens transparently. The system retrieves the ciphertext from SurrealDB and decrypts it using the derived Fernet key.

import asyncio
from open_notebook.domain.credential import Credential

async def load():
    cred = await Credential.get("credential-id-123")
    # Returns SecretStr with decrypted plaintext

    print(cred.api_key.get_secret_value())

asyncio.run(load())

If the encryption key is missing or incorrect, decrypt_value raises a ValueError with a clear message prompting the operator to verify the OPEN_NOTEBOOK_ENCRYPTION_KEY configuration.

Key Rotation Strategy

To rotate encryption keys, operators update the environment variable and re-save existing credentials (or run a migration script that re-encrypts values with the new key). The code remains agnostic to the specific secret value—only the derived Fernet key matters—making rotation straightforward without requiring application code changes.

Implementation Examples

Direct Encryption and Decryption

You can use the encryption utilities directly for custom implementations:

from open_notebook.utils.encryption import encrypt_value, decrypt_value

plain = "sk-abc123"
cipher = encrypt_value(plain)      # Returns Fernet token

recovered = decrypt_value(cipher)  # Returns original string

assert recovered == plain

Handling Missing or Invalid Keys

When the encryption key is incorrect or unset, the system provides clear error handling:

from open_notebook.utils.encryption import decrypt_value

try:
    decrypt_value(some_ciphertext)
except ValueError as exc:
    # Error indicates: "data appears to be encrypted but key is incorrect"

    raise RuntimeError("Set a valid OPEN_NOTEBOOK_ENCRYPTION_KEY") from exc

Integration with AI Providers

The open_notebook/ai/key_provider.py module retrieves decrypted API keys from Credential records and falls back to environment variables when necessary, ensuring AI providers receive valid authentication tokens without exposing encrypted storage details.

Summary

• Encryption keys are managed through a single environment variable (OPEN_NOTEBOOK_ENCRYPTION_KEY) or Docker secret file, deriving a Fernet key via SHA-256 hashing.
• Field-level encryption in open_notebook/domain/credential.py ensures API keys are encrypted before reaching SurrealDB and decrypted automatically upon retrieval.
• Lazy loading allows the application to start without the key present, failing gracefully only when encryption/decryption is actually attempted.
• Key rotation requires updating the environment variable and re-encrypting existing records, with no changes needed to the application code.
• Graceful degradation prevents application crashes when decryption fails, using "UNDECRYPTABLE" placeholders instead of breaking the entire record.

Frequently Asked Questions

What happens if I forget to set the OPEN_NOTEBOOK_ENCRYPTION_KEY?

If the environment variable is missing, the application will start successfully due to lazy loading, but any attempt to encrypt or decrypt credentials will fail with a clear error message. The decrypt_value function in open_notebook/utils/encryption.py raises a ValueError prompting you to configure the correct key.

Can I use different encryption keys for different credentials?

The current implementation uses a global encryption key derived from a single environment variable. All credentials in the database are encrypted with the same key. To use different keys, you would need to modify the Credential model in open_notebook/domain/credential.py to support key identifiers and multiple key sources.

How does Open Notebook handle key rotation without data loss?

Key rotation requires updating the OPEN_NOTEBOOK_ENCRYPTION_KEY environment variable and then re-saving all credentials to trigger re-encryption with the new key. The system stores only the ciphertext, so as long as you maintain access to the old key during the transition, you can decrypt with the old key and re-encrypt with the new one without losing data.

Is the encryption key ever stored in the database?

No. The encryption key is derived at runtime from the environment variable and exists only in memory as a Fernet instance. The open_notebook/utils/encryption.py module never persists the key to disk or database, and the original passphrase is hashed immediately upon retrieval, ensuring the database contains only encrypted ciphertext with no embedded key material.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}