How to Configure Database-Stored Credentials for AI Providers Instead of Environment Variables

Open Notebook stores AI provider authentication in SurrealDB's credential table, automatically decrypting and loading keys at runtime via provision_provider_keys() in open_notebook/ai/key_provider.py, which overrides environment variables while maintaining backward compatibility for legacy setups.

The lfnovo/open-notebook project eliminates the need to hardcode sensitive API keys in environment variables by implementing a secure, database-first credential management system. This architecture allows you to rotate keys, manage multiple provider configurations, and deploy across environments without restarting services or exposing secrets in configuration files.

Why Store Credentials in SurrealDB?

Storing credentials in the database rather than environment variables provides several operational advantages. You can update API keys at runtime without container restarts, support multiple credential sets for different use cases, and encrypt secrets at rest using the application's encryption layer.

The system implements a database-first configuration pattern. When provision_provider_keys() executes, it queries the credential table and populates environment variables like OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT dynamically. If a credential exists in the database, it overrides any pre-existing environment variable. If no database credential exists, the system falls back to traditional environment variables, ensuring backward compatibility with legacy deployments.

Core Architecture Components

The Credential Model

The Credential class in open_notebook/domain/credential.py defines the database schema and encryption logic. It stores the provider name, encrypted API key, and a flexible config object (added in Migration 15) for provider-specific settings.

The _prepare_save_data method automatically encrypts the api_key field before persistence, while decrypt_value handles decryption during retrieval. The model supports provider-specific fields like base_url, endpoint, and project, which map to their respective environment variable names.

The Key Provider Module

The open_notebook/ai/key_provider.py module contains the runtime loading logic. The provision_provider_keys() function orchestrates credential retrieval, while _get_default_credential() executes the SurrealQL query SELECT * FROM credential WHERE provider = $provider via Credential.get_by_provider().

Once retrieved, the module decrypts the secret and injects it into os.environ using provider-specific mappings defined in PROVIDER_CONFIG. This happens automatically before model instantiation.

The Credentials API

The api/routers/credentials.py router exposes REST endpoints for CRUD operations. You can create, update, and delete credentials without direct database access, enabling secure credential rotation through API calls.

Step-by-Step Configuration

1. Create a Credential Record

You can create credentials via the REST API or directly in Python. The API endpoint accepts a JSON payload containing the provider name, modalities, API key, and optional configuration parameters.

curl -X POST http://localhost:5055/credentials \
  -H "Content-Type: application/json" \
  -d '{
        "name": "OpenAI Production",
        "provider": "openai",
        "modalities": ["language", "embedding"],
        "api_key": "sk-REDACTED",
        "config": {"num_ctx": 16384}
      }'

2. Load Credentials at Runtime

Before instantiating AI models, call provision_provider_keys() to populate environment variables from the database:

from open_notebook.ai.key_provider import provision_provider_keys
from esperanto import AIFactory

# Load DB-stored OpenAI credentials into env vars

await provision_provider_keys("openai")

# Create model; it will use the database credential

model = AIFactory.create_language(model_name="gpt-4", provider="openai")

3. Configure Provider-Specific Settings

For providers requiring additional configuration beyond the API key, use the config object or dedicated fields like base_url. This example configures a self-hosted Ollama instance:

from open_notebook.domain.credential import Credential

await Credential(
    name="Local Ollama",
    provider="ollama",
    api_key="dummy",            # Ollama may not require a key

    base_url="http://localhost:11434"
).save()

await provision_provider_keys("ollama")

# OLLAMA_API_BASE is now set to http://localhost:11434

How Runtime Loading Works

When you call provision_provider_keys("<provider>"), the following sequence executes:

1. Query Execution: The system calls _get_default_credential(), which executes Credential.get_by_provider() to fetch the first matching credential from SurrealDB.

2. Decryption: The encrypted api_key is decrypted using the application's encryption utilities.

3. Environment Injection: The decrypted key and additional configuration fields are injected into os.environ under the provider-specific variable names (e.g., OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT).

4. Model Provisioning: When AIFactory.create_language() or similar methods run, they access these environment variables, now populated with database-stored values rather than system-level configuration.

This process occurs automatically in open_notebook/ai/models.py when models are provisioned, ensuring seamless integration with the rest of the application.

Accessing Credentials Directly

For custom logic or administrative functions, you can query credentials directly:

from open_notebook.domain.credential import Credential

# Get all stored OpenAI credentials

openai_creds = await Credential.get_by_provider("openai")
for cred in openai_creds:
    print(cred.name, cred.api_key.get_secret_value())

Summary

• Database-First Approach: SurrealDB stores encrypted credentials in the credential table, overriding environment variables at runtime while maintaining fallback support.
• Automatic Loading: The provision_provider_keys() function in open_notebook/ai/key_provider.py handles decryption and environment variable injection automatically.
• Secure Storage: The Credential model in open_notebook/domain/credential.py encrypts API keys via _prepare_save_data before persistence.
• Flexible Configuration: The config object (Migration 15) and dedicated fields like base_url support provider-specific settings for OpenAI, Azure, Ollama, and other providers.
• API Management: The api/routers/credentials.py router provides RESTful endpoints for credential lifecycle management without service restarts.

Frequently Asked Questions

Does storing credentials in the database replace environment variables completely?

No, it complements them. The system uses a database-first approach where credentials stored in SurrealDB override environment variables, but if no database credential exists for a provider, the application falls back to standard environment variables. This ensures backward compatibility while enabling runtime configuration.

How are API keys encrypted in the database?

The Credential model in open_notebook/domain/credential.py uses the _prepare_save_data method to encrypt the api_key field before saving. When retrieved via _get_default_credential() or get_by_provider(), the decrypt_value method decrypts the secret before injecting it into environment variables.

Can I configure multiple credentials for the same provider?

Yes, the Credential.get_by_provider() method returns all credentials for a given provider, and you can select specific ones by name or ID. However, provision_provider_keys() currently loads the first available credential by default. For custom logic, retrieve credentials directly and manage selection in your application code.

What provider-specific fields are supported beyond the API key?

The Credential model supports fields like base_url, endpoint, and project, which map to environment variables such as OLLAMA_API_BASE and AZURE_OPENAI_ENDPOINT. Additional provider-specific settings can be stored in the flexible config JSON object introduced in Migration 15, allowing extensibility for new provider requirements without schema changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}