How to Configure Custom Endpoint Models for LLM Providers with Per-Model Overrides in Craft Agents OSS

Craft Agents OSS lets you configure custom LLM endpoints (such as Ollama, vLLM, or DashScope) and override capabilities like image support on a per-model basis by defining an LlmConnection with authType set to api_key_with_endpoint and using the setModelSupportsImages utility to toggle specific model flags.

Craft Agents OSS supports connecting to arbitrary OpenAI- or Anthropic-compatible endpoints, enabling integration with self-hosted models or third-party providers beyond standard APIs. When you configure a custom endpoint, you must explicitly describe the available models and can optionally specify per-model overrides for capabilities such as image processing. This article explains how to configure custom endpoint models for LLM providers with per-model overrides using the configuration schema and runtime utilities in the craft-ai-agents/craft-agents-oss repository.

Configuring the LlmConnection Schema

The configuration for custom endpoints resides in the LlmConnection interface defined in packages/shared/src/config/llm-connections.ts. To point Craft Agents at a custom endpoint, you must set the authType field to api_key_with_endpoint, which triggers the endpoint-specific validation logic defined in authTypeRequiresEndpoint.

A complete LlmConnection object for a custom endpoint requires four critical fields:

• baseUrl: The URL of your custom endpoint (e.g., http://localhost:11434/v1 for Ollama)
• customEndpoint: An object specifying the protocol (openai-completions or anthropic-messages) and a default supportsImages boolean
• models: An array of model identifiers that can be either simple strings or objects containing per-model overrides
• authType: Must be set to api_key_with_endpoint as defined in the authTypeRequiresEndpoint helper

Implementing Per-Model Overrides

When you define models in the models array, each entry can be a plain string (like "gemma") or an object with an id and capability flags. This design allows you to override connection-level defaults for specific models.

The supportsImages Flag Hierarchy

The system resolves image support capabilities through a specific precedence chain implemented in packages/server-core/src/sessions/runtime-config.ts (lines 30-34):

1. Per-model override: If a model object explicitly sets supportsImages to a boolean, that value takes precedence
2. Connection-level default: If the model has no override, the system falls back to customEndpoint.supportsImages
3. Driver consumption: The agent driver in packages/shared/src/agent/backend/internal/drivers/pi.ts (line 241) receives the final resolved flag

This hierarchy enables mixed endpoints where one model supports vision capabilities while another remains text-only, despite sharing the same base URL.

Using setModelSupportsImages

The setModelSupportsImages function in packages/shared/src/config/llm-connections.ts provides a pure functional way to toggle image support for specific models without mutating the original configuration:

export function setModelSupportsImages(
  connection: LlmConnection,
  modelId: string,
  enabled: boolean,
): LlmConnection

This helper automatically promotes string entries in the models array to objects when you apply overrides, ensuring type safety while maintaining backward compatibility with existing configurations.

Runtime Configuration Flow

When a session initializes, packages/server-core/src/sessions/runtime-config.ts builds the runtime model list by merging connection defaults with per-model overrides. The logic checks typeof model.supportsImages === 'boolean' to determine if an override exists before falling back to the connection-level default.

The final configuration propagates to the agent drivers (such as pi.ts or anthropic.ts), which use the resolved flag to determine whether to serialize image data in API requests.

Configuration Examples

JSON Schema for Custom Endpoints

Here is a complete configuration for an Ollama local instance with mixed model capabilities:

{
  "slug": "ollama-local",
  "name": "Ollama (Local)",
  "providerType": "pi_compat",
  "authType": "api_key_with_endpoint",
  "baseUrl": "http://localhost:11434/v1",
  "customEndpoint": {
    "api": "openai-completions",
    "supportsImages": true
  },
  "models": [
    "gemma",
    { "id": "llama2", "supportsImages": false }
  ],
  "defaultModel": "gemma",
  "createdAt": 1720150800000
}

In this example, gemma inherits the true default from customEndpoint.supportsImages, while llama2 explicitly overrides this to disable image support.

Programmatic Toggle

To update capabilities programmatically:

import { setModelSupportsImages } from '@/shared/config/llm-connections';
import { getLlmConnection, saveLlmConnection } from '@/shared/config/storage';

const conn = await getLlmConnection('ollama-local');
const updated = setModelSupportsImages(conn, 'llama2', true);
await saveLlmConnection(updated);

After persistence, the modelSupportsImages utility (defined at line 49 in packages/shared/src/config/llm-connections.ts) will return true for llama2 and true for gemma (inheriting the default).

RPC Update Method

The UI communicates changes via RPC to packages/server-core/src/handlers/rpc/llm-connections.ts:

{
  "method": "LLM_Connection:update",
  "params": {
    "slug": "ollama-local",
    "modelId": "llama2",
    "supportsImages": true
  }
}

The server-side handler applies the update through setModelSupportsImages and broadcasts the change to running agent sessions, ensuring immediate capability updates without restarts.

Key Implementation Files

File	Purpose
packages/shared/src/config/llm-connections.ts	Defines LlmConnection, CustomEndpointConfig, and helper functions including setModelSupportsImages and modelSupportsImages
packages/shared/src/config/__tests__/set-model-supports-images.test.ts	Test suite validating the promotion of string entries to objects and override precedence
packages/server-core/src/sessions/runtime-config.ts	Merges connection defaults with per-model overrides at session startup
packages/shared/src/agent/backend/internal/drivers/pi.ts	Consumes the final supportsImages flag when constructing API requests
packages/server-core/src/handlers/rpc/llm-connections.ts	RPC entry point for UI-driven configuration updates

Summary

• Configure custom endpoints by setting authType to api_key_with_endpoint and providing a baseUrl and customEndpoint configuration in packages/shared/src/config/llm-connections.ts.
• Define models in the models array as either strings (inheriting defaults) or objects with per-model flags like supportsImages.
• Use setModelSupportsImages() to programmatically toggle capabilities; this pure function returns a new LlmConnection while promoting string entries to objects automatically.
• The runtime resolves capabilities in packages/server-core/src/sessions/runtime-config.ts, giving per-model overrides precedence over connection-level defaults.
• Changes persist through saveLlmConnection() and propagate to running agents via the RPC handler in packages/server-core/src/handlers/rpc/llm-connections.ts.

Frequently Asked Questions

How do I add image support to only one model on a custom endpoint?

Define the model as an object with supportsImages: true in the models array while keeping the connection-level customEndpoint.supportsImages set to false (or omitting it). According to the runtime logic in packages/server-core/src/sessions/runtime-config.ts, the per-model boolean takes precedence over the connection default.

Can I use the same custom endpoint configuration for both OpenAI-compatible and Anthropic-compatible servers?

Yes. Set the customEndpoint.api field to either openai-completions or anthropic-messages depending on your endpoint's protocol. The LlmConnection type in packages/shared/src/config/llm-connections.ts supports both protocols, and the agent driver selection depends on this field.

What happens if I change a model's supportsImages flag while agents are running?

The RPC handler in packages/server-core/src/handlers/rpc/llm-connections.ts applies the update via setModelSupportsImages and broadcasts the configuration change to active sessions. The runtime configuration updates immediately, so subsequent agent invocations use the new capability flag without requiring a server restart.

How do I check if a specific model supports images in my code?

Use the modelSupportsImages utility exported from packages/shared/src/config/llm-connections.ts (line 49). This function accepts an LlmConnection and a modelId. It returns the resolved boolean by checking for per-model overrides first, then falling back to the connection-level default if no override exists.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}