How the Open Notebook connection_tester Validates AI Provider Credentials

The Open Notebook connection_tester validates AI provider credentials through a two-stage process that checks provider-specific HTTP endpoints for Azure and Ollama, or executes minimal Esperanto model calls for standard providers like OpenAI and Anthropic, returning a standardized success boolean and message.

The connection_tester module in the lfnovo/open-notebook repository provides a critical safeguard, ensuring API keys and endpoints are functional before they're used in production workflows. This validation utility prevents downstream failures by testing credentials against actual provider infrastructure using targeted HTTP requests or lightweight model invocations.

Two-Stage Validation Architecture

The validation system operates in two distinct stages depending on the provider type, as implemented in open_notebook/ai/connection_tester.py.

Provider-Specific Endpoint Checks

For providers exposing lightweight HTTP endpoints, the tester makes direct GET requests to verify connectivity and credential validity.

Azure OpenAI: The _test_azure_connection function (lines 40-94) contacts {endpoint}/openai/models?api-version=... and parses the returned model list. A 200 response with non-empty payload confirms valid credentials, while HTTP 401/403 errors map to specific "Invalid API key" or "Insufficient permissions" messages.

Ollama: The _test_ollama_connection function (lines 97-130) queries {base_url}/api/tags to retrieve available models, reporting the count of discovered models upon success.

OpenAI-Compatible Servers: The _test_openai_compatible_connection function (lines 132-168) validates against {base_url}/models endpoints for generic OpenAI-compatible providers.

These functions return a tuple (bool success, str message) that propagates to the API client.

Standard Provider Checks via Esperanto

For standard providers including OpenAI, Anthropic, Google, Groq, Mistral, DeepSeek, and X-AI, the tester uses the Esperanto abstraction layer with minimal model calls.

The system selects inexpensive test models from the TEST_MODELS dictionary (lines 18-37) and creates instances via AIFactory.create_* methods:

• Language models: Executes ainvoke("Hi")
• Embedding models: Executes aembed(["test"])
• Text-to-speech models: Validates construction only (no API call required)

Central Orchestration and Error Handling

The test_credential function in api/credentials_service.py (lines 84-106) serves as the central router. It extracts the stored Credential record, builds Esperanto configuration via credential.to_esperanto_config(), and routes to either provider-specific helpers or the generic Esperanto path.

Error normalization occurs in _normalize_error_message (lines 236-254), which translates provider-specific exceptions into standardized user-facing messages.

The function returns a uniform JSON payload:

{
  "provider": "openai",
  "success": true,
  "message": "Connection successful"
}

Practical Implementation Examples

Testing via FastAPI Endpoint

import httpx

# Assume the FastAPI server is running at http://localhost:5055

async def check_credential(credential_id: str):
    async with httpx.AsyncClient() as client:
        resp = await client.get(
            f"http://localhost:5055/credentials/{credential_id}/test"
        )
        return resp.json()

# Usage

result = await check_credential("credential:123")
print(result)   # → {'provider': 'openai', 'success': True, 'message': 'Connection successful'}

Direct Python Validation

from open_notebook.ai.connection_tester import TEST_MODELS
from esperanto.factory import AIFactory
from open_notebook.ai.key_provider import provision_provider_keys

async def validate_openai():
    # Provision any missing environment variables

    await provision_provider_keys("openai")
    # Use the minimal test model defined in TEST_MODELS

    model_name, _ = TEST_MODELS["openai"]
    model = AIFactory.create_language(
        model_name=model_name,
        provider="openai",
        config={},          # Empty config ⇒ falls back to env vars

    )
    await model.to_langchain().ainvoke("Hello")
    print("OpenAI credentials are valid")

# Run in an async context

await validate_openai()

Manual Azure Connection Testing

from open_notebook.ai.connection_tester import _test_azure_connection

async def azure_check():
    success, msg = await _test_azure_connection(
        endpoint="https://my-azure-openai.openai.azure.com",
        api_key="MY_AZURE_KEY",
        api_version="2024-10-21"
    )
    print(success, msg)

# → True "Connected. 5 models: gpt-35-turbo, gpt-35-turbo-16k, ..."

Key Source Files

• open_notebook/ai/connection_tester.py: Implements low-level credential checks for all providers
• api/credentials_service.py: High-level API routing and unified result formatting
• open_notebook/ai/models.py: Provides ModelManager and Model entities linking credentials to Esperanto instances
• tests/test_credentials_api.py: Test suite confirming credential-testing endpoint behavior

Summary

• The connection_tester validates AI provider credentials through provider-specific HTTP endpoint checks or minimal Esperanto model invocations.
• Azure, Ollama, and OpenAI-compatible servers receive direct HTTP validation against their model listing endpoints.
• Standard providers like OpenAI and Anthropic are validated via lightweight ainvoke() or aembed() calls using inexpensive test models.
• The test_credential function in api/credentials_service.py orchestrates all validation paths and returns standardized JSON responses.
• Error messages are normalized through _normalize_error_message to provide consistent user feedback across different provider implementations.

Frequently Asked Questions

What providers use the HTTP endpoint validation method?

Azure OpenAI, Ollama, and generic OpenAI-compatible servers receive direct HTTP validation through the _test_azure_connection, _test_ollama_connection, and _test_openai_compatible_connection functions respectively. These providers expose lightweight model listing endpoints that confirm credential validity without consuming inference credits.

How does the tester validate credentials without incurring high costs?

The system uses the TEST_MODELS dictionary (lines 18-37) to select the cheapest available model for each provider. For language models, it sends a single "Hi" prompt via ainvoke(). For embedding models, it calls aembed(["test"]). Text-to-speech models only require construction validation, avoiding any API consumption entirely.

Where is the credential validation logic centralized?

The test_credential function in api/credentials_service.py (lines 84-106) serves as the central orchestration point. It extracts credential records, builds Esperanto configurations, and routes requests to either provider-specific helpers or the generic Esperanto validation path, ensuring consistent behavior across all provider types.

What format does the connection tester return?

All validation paths return a standardized JSON payload containing three fields: provider (string identifier), success (boolean), and message (string description). This uniform structure allows the frontend to handle credential validation results predictably regardless of the underlying AI provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}