# How Model Configuration Testing and Validation Works in pi-web: A Technical Deep Dive > Discover how pi-web validates model configurations using two API endpoints. Learn about runtime loading tests and upstream discovery verification for provider settings. - Repository: [Alex Yang/pi-web](https://github.com/agegr/pi-web) - Tags: deep-dive - Published: 2026-08-15 --- **pi-web validates model configurations through two dedicated API endpoints that perform runtime loading tests and upstream discovery verification before saving any provider settings.** The pi-web project provides a robust validation pipeline for AI model configurations. Rather than trusting user-provided provider settings blindly, the codebase implements two specialized routes that create temporary runtime environments and make live requests to upstream APIs. This article explains exactly how these validation mechanisms work, with direct references to the source code implementation. ## Overview of the Validation Architecture The validation system centers on two distinct operations: - **Testing** confirms that a single model configuration can be loaded, authenticated, and queried successfully - **Discovery** verifies that a provider's base URL and credentials can fetch and parse a complete model list Both routes operate as isolated Next.js API handlers under `app/api/models-config/`, ensuring that broken configurations never reach the persistent [`models.json`](https://github.com/agegr/pi-web/blob/main/models.json) store. ## Runtime Configuration Testing (`/api/models-config/test`) The [test route](https://github.com/agegr/pi-web/blob/main/app/api/models-config/test/route.ts) performs end-to-end validation by spinning up a temporary ModelRuntime environment and executing a live completion call. ### Creating the Isolated Test Environment When a POST request arrives, the handler extracts `providerName`, `provider` configuration, and `model` definition from the JSON body. It then generates a unique temporary directory using Node.js `mkdtempSync` and writes a minimal [`models.json`](https://github.com/agegr/pi-web/blob/main/models.json) file containing only the supplied configuration: ```typescript const modelsJsonContent = { [providerName]: { ...provider, models: [model], }, }; fs.writeFileSync(modelsJsonPath, JSON.stringify(modelsJsonContent, null, 2)); ``` This isolated file structure ensures that the test cannot interfere with existing configurations while still exercising the exact same loading logic used in production. ### Loading and Authenticating the Runtime With the temporary [`models.json`](https://github.com/agegr/pi-web/blob/main/models.json) in place, the handler initializes `ModelRuntime.create({ modelsPath })`. This call triggers pi-web's standard validation pipeline: schema verification, provider normalization, and model registry construction. Any loading failure returns immediately with diagnostic details. Authentication resolution follows via `modelRuntime.getAuth(providerName)`, which extracts API keys from environment variables or explicit configuration. The handler specifically validates that an `apiKey` exists before proceeding, preventing wasted requests: ```typescript const auth = await modelRuntime.getAuth(providerName); if (!auth?.apiKey) { return NextResponse.json({ error: 'API key missing' }, { status: 400 }); } ``` ### Executing the Live Completion Test The critical validation step sends an actual request to the upstream provider. The handler calls `completeSimple` with a minimal prompt—*"Reply with OK only."*—and strict timing constraints: - **Timeout:** 20 seconds via `AbortSignal.timeout(20000)` - **Retries:** Disabled (`maxRetries: 0`) to isolate genuine connection issues from transient failures ```typescript const response = await completeSimple({ modelRuntime, providerName, modelId: model.id, messages: [{ role: 'user', content: 'Reply with OK only.' }], timeoutSignal: AbortSignal.timeout(20000), fetchOptions: { maxRetries: 0 }, }); ``` Success requires the call to complete without `AbortError` and return valid content. The handler captures latency, HTTP status, and truncates the response text to 300 characters for the JSON response. Finally, it removes the temporary directory to prevent filesystem accumulation: ```typescript rmSync(tempDir, { recursive: true, force: true }); ``` A successful test proves three capabilities simultaneously: configuration parsing, authentication resolution, and live API communication. ## Provider Discovery Validation (`/api/models-config/discover`) The [discover route](https://github.com/agegr/pi-web/blob/main/app/api/models-config/discover/route.ts) takes a different approach—it validates that a provider's publicly accessible model list endpoint works correctly. ### Building the Target URL and Headers The handler accepts `providerName` and `provider` (including `baseUrl` and optional `api` type) from the POST body. It constructs the full model list URL through `buildModelsListUrl`, which maps provider types to their respective endpoint patterns: | API Type | Endpoint Pattern | |----------|-----------------| | `openai-completions` | `{baseUrl}/models` | | `anthropic-messages` | `{baseUrl}/v1/models` | | `google-generative` | `{baseUrl}/v1beta/models` | | `ollama` | `{baseUrl}/api/tags` | Header construction via `buildHeaders` injects provider-specific authentication: ```typescript const headers = buildHeaders({ apiKey: provider.apiKey, api: provider.api, providerName, }); ``` This automatically sets `x-api-key`, `anthropic-version`, `Authorization: Bearer`, or other required headers based on the provider type. ### Fetching and Parsing the Model List The handler performs a direct HTTP GET with the same 20-second timeout used in testing. Non-OK responses propagate their status and truncated body (500 characters max) as JSON errors. Valid JSON responses proceed to `parseDiscoveredModels`, which normalizes heterogeneous provider schemas into pi-web's internal model format. Empty model lists trigger explicit failure—distinguishing between "endpoint reachable but broken" and "endpoint working with no models available." Success returns the normalized model array plus the resolved endpoint URL for UI display: ```json { "models": [ { "id": "claude-3-5-sonnet-20240620", "contextWindow": 200000 }, { "id": "claude-3-opus-20240229", "contextWindow": 200000 } ], "endpoint": "https://api.anthropic.com/v1/models" } ``` ## Practical Usage Examples ### Testing a Model Configuration ```bash curl -X POST https://your-instance.com/api/models-config/test \ -H "Content-Type: application/json" \ -d '{ "providerName": "openai", "provider": { "apiKey": "sk-...", "baseUrl": "https://api.openai.com/v1" }, "model": { "id": "gpt-4o-mini", "contextWindow": 128000, "description": "OpenAI GPT-4o mini" } }' ``` **Successful response:** ```json { "ok": true, "latencyMs": 847, "status": 200, "responseText": "OK" } ``` ### Discovering Available Models ```bash curl -X POST https://your-instance.com/api/models-config/discover \ -H "Content-Type: application/json" \ -d '{ "providerName": "anthropic", "provider": { "apiKey": "sk-ant-...", "baseUrl": "https://api.anthropic.com", "api": "anthropic-messages" } }' ``` **Successful response:** ```json { "models": [ { "id": "claude-3-5-sonnet-20240620", "contextWindow": 200000, "maxOutputTokens": 8192, "description": "Claude 3.5 Sonnet" } ], "endpoint": "https://api.anthropic.com/v1/models" } ``` ## Security and Access Controls Both validation routes are protected by middleware defined in [request-security.ts](https://github.com/agegr/pi-web/blob/main/lib/request-security.ts). This layer ensures that only authenticated administrators can trigger expensive upstream API calls and temporary filesystem operations, preventing abuse of the validation endpoints as open proxies. The configuration storage itself—handled by [models-config/route.ts](https://github.com/agegr/pi-web/blob/main/app/api/models-config/route.ts)—remains separate from validation, ensuring that only tested, working configurations persist to the user's [`models.json`](https://github.com/agegr/pi-web/blob/main/models.json) file. ## Summary - **Temporary runtime creation** in the test route isolates validation from production configurations while using identical loading logic - **Live completion calls** with strict timeouts and disabled retries provide deterministic connectivity verification - **Provider-specific URL building and header injection** in discovery handles heterogeneous API conventions automatically - **Consistent 20-second timeouts** across both routes prevent hung requests from degrading user experience - **Automatic cleanup** of temporary directories and response truncation maintain resource hygiene ## Frequently Asked Questions ### What happens if the API key is missing during testing? The test route returns HTTP 400 with `{"error": "API key missing"}` before attempting any network request. This validation occurs immediately after `modelRuntime.getAuth` resolves. ### Can discovery work without specifying the api field? Yes. The [discover route](https://github.com/agegr/pi-web/blob/main/app/api/models-config/discover/route.ts) infers sensible defaults from the `providerName` when `api` is omitted, though explicit configuration ensures correct endpoint selection for non-standard deployments. ### Why does the test use "Reply with OK only" as the prompt? This minimal prompt minimizes token costs and latency while still requiring valid model processing. Any functional model must parse the request and generate coherent output; "OK" serves as unambiguous success confirmation. ### How does pi-web prevent validation endpoint abuse? The [request-security.ts](https://github.com/agegr/pi-web/blob/main/lib/request-security.ts) middleware layer enforces authentication requirements. Additionally, both routes impose strict timeouts and input size limits, and the test route's temporary directory creation is bounded by system temp space.