How connection_tester Validates AI Provider Credentials with Minimal API Calls

The ConnectionTester avoids expensive model generations by using lightweight "list-models" probes and minimal-payload chat requests to validate credentials without consuming tokens.

The open_notebook/ai/connection_tester.py module in the lfnovo/open-notebook repository implements a credential-validation strategy designed to verify AI provider connectivity without triggering costly API calls. By targeting metadata endpoints and using tiny test payloads, the system confirms that keys are functional while keeping network traffic and token usage near zero.

Provider-Specific Model Listing Probes

The primary validation strategy relies on provider-specific endpoints that return model catalogs rather than generating completions. This approach validates authentication headers and network reachability without invoking inference engines.

Azure OpenAI Validation

For Azure OpenAI resources, the tester sends a GET request to the {endpoint}/openai/models endpoint with the appropriate api-version query parameter. A successful 200 response containing the model list confirms valid credentials.

In open_notebook/ai/connection_tester.py, the _test_azure_connection function implements this logic around lines 40-55, constructing the request shown at line 66:

# Example: Validate Azure OpenAI credentials

from open_notebook.ai.connection_tester import _test_azure_connection

async def check_azure():
    ok, msg = await _test_azure_connection(
        endpoint="https://my-azure.openai.azure.com",
        api_key="sk-…",
        api_version="2024-10-21"
    )
    print(ok, msg)  # -> True  Connected. 3 models: gpt-35-turbo, …

Ollama Local Server Checks

For Ollama instances, the tester calls the local /api/tags endpoint, which lists installed models without requiring authentication. A 200 response indicates the server is reachable and responsive.

See the implementation in open_notebook/ai/connection_tester.py at lines 97-104.

OpenAI-Compatible Services

Generic OpenAI-compatible providers are validated via GET {base_url}/models with an optional Authorization: Bearer {api_key} header. The response structure mirrors Azure's model listing format.

The _test_openai_compatible_connection function handles this flow at lines 32-42:

# Example: Validate any OpenAI-compatible service

from open_notebook.ai.connection_tester import _test_openai_compatible_connection

async def check_compatible():
    ok, msg = await _test_openai_compatible_connection(
        base_url="https://api.myprovider.com/v1",
        api_key="my‑key"
    )
    print(ok, msg)  # -> True  Connected. 5 models available: …

Minimal-Payload Fallback Testing

When a provider does not expose a lightweight model-listing endpoint, the system falls back to test_individual_model, defined at lines 56-66. This function sends a minimal chat request containing only the string "Hi!" to the cheapest available model defined in TEST_MODELS (typically gpt-3.5-turbo for OpenAI).

The call to esp_model.achat_complete at line 77 executes this tiny payload, costing virtually nothing while confirming the API key can invoke actual inference:

# Example: End-to-end model validation with minimal payload

from open_notebook.ai.connection_tester import test_individual_model
from open_notebook.ai.models import ModelManager

async def validate_model():
    manager = ModelManager()
    # Assumes a Model record with id="openai:gpt-3.5-turbo"

    model = await manager.get_model("openai:gpt-3.5-turbo")
    ok, msg = await test_individual_model(model)
    print(ok, msg)  # -> True  Response: Hello! …

Audio-Based Validation for Speech-to-Text

For providers exposing only speech-to-text (STT) endpoints, the tester employs _get_test_audio (lines 23-33) to generate a short pre-bundled audio clip or a silent WAV fallback. Uploading this test file to the transcription endpoint and receiving any valid response confirms the credentials work for audio processing without requiring expensive generation calls.

Unified Error Normalization

All low-level httpx exceptions are caught and translated into human-readable messages by _normalize_error_message at lines 36-53. This function maps technical exceptions to clear status indicators like Invalid API key, Rate limited, or Connection error, ensuring callers receive actionable feedback without exposing raw stack traces or sensitive request details.

Summary

• List-models probes are the primary validation method, targeting Azure's /openai/models, Ollama's /api/tags, and generic /models endpoints to verify credentials without token consumption.
• Minimal chat fallback uses a single "Hi!" message via test_individual_model when list endpoints are unavailable, keeping costs negligible.
• Audio testing validates STT-only providers using silent or pre-bundled WAV files.
• Error normalization converts httpx exceptions into clear, user-friendly messages via _normalize_error_message.

Frequently Asked Questions

Why not just send a completion request to validate credentials?

Sending a full completion request consumes tokens and incurs latency. The connection_tester prioritizes metadata endpoints that return model catalogs without invoking inference engines, minimizing both cost and response time while still confirming authentication headers are valid.

Which providers support list-models validation versus minimal chat fallback?

Azure OpenAI, Ollama, and OpenAI-compatible services support the list-models probe approach. Providers without standardized model listing endpoints trigger the fallback to test_individual_model, which uses the minimal "Hi!" payload with the cheapest configured model from TEST_MODELS.

How does the tester handle rate limiting or network errors?

The _normalize_error_message function (lines 36-53) intercepts all httpx exceptions and translates them into specific messages like Rate limited or Connection error. This allows the UI to display actionable feedback without exposing raw HTTP status codes or exception traces to end users.

Can I customize the test model used for credential validation?

The fallback mechanism uses models defined in the TEST_MODELS configuration, typically selecting the cheapest available option (e.g., gpt-3.5-turbo). To customize validation behavior, modify the model selection logic within test_individual_model or adjust the TEST_MODELS mapping in the configuration to prioritize your preferred low-cost model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}