# How to Configure Custom Endpoint Models for LLM Providers with Per-Model Overrides in Craft Agents OSS

> Learn to configure custom LLM endpoints in Craft Agents OSS with per-model overrides for features like image support. Define LlmConnections and use setModelSupportsImages utility.

- Repository: [Craft Ai Agents/craft-agents-oss](https://github.com/craft-ai-agents/craft-agents-oss)
- Tags: how-to-guide
- Published: 2026-07-03

---

**Craft Agents OSS lets you configure custom LLM endpoints (such as Ollama, vLLM, or DashScope) and override capabilities like image support on a per-model basis by defining an `LlmConnection` with `authType` set to `api_key_with_endpoint` and using the `setModelSupportsImages` utility to toggle specific model flags.**

Craft Agents OSS supports connecting to arbitrary OpenAI- or Anthropic-compatible endpoints, enabling integration with self-hosted models or third-party providers beyond standard APIs. When you configure a custom endpoint, you must explicitly describe the available models and can optionally specify per-model overrides for capabilities such as image processing. This article explains how to configure custom endpoint models for LLM providers with per-model overrides using the configuration schema and runtime utilities in the `craft-ai-agents/craft-agents-oss` repository.

## Configuring the LlmConnection Schema

The configuration for custom endpoints resides in the `LlmConnection` interface defined in [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts). To point Craft Agents at a custom endpoint, you must set the `authType` field to **`api_key_with_endpoint`**, which triggers the endpoint-specific validation logic defined in `authTypeRequiresEndpoint`.

A complete `LlmConnection` object for a custom endpoint requires four critical fields:

- **`baseUrl`**: The URL of your custom endpoint (e.g., `http://localhost:11434/v1` for Ollama)
- **`customEndpoint`**: An object specifying the protocol (`openai-completions` or `anthropic-messages`) and a default `supportsImages` boolean
- **`models`**: An array of model identifiers that can be either simple strings or objects containing per-model overrides
- **`authType`**: Must be set to `api_key_with_endpoint` as defined in the `authTypeRequiresEndpoint` helper

## Implementing Per-Model Overrides

When you define models in the `models` array, each entry can be a plain string (like `"gemma"`) or an object with an `id` and capability flags. This design allows you to override connection-level defaults for specific models.

### The supportsImages Flag Hierarchy

The system resolves image support capabilities through a specific precedence chain implemented in [`packages/server-core/src/sessions/runtime-config.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/sessions/runtime-config.ts) (lines 30-34):

1. **Per-model override**: If a model object explicitly sets `supportsImages` to a boolean, that value takes precedence
2. **Connection-level default**: If the model has no override, the system falls back to `customEndpoint.supportsImages`
3. **Driver consumption**: The agent driver in [`packages/shared/src/agent/backend/internal/drivers/pi.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/agent/backend/internal/drivers/pi.ts) (line 241) receives the final resolved flag

This hierarchy enables mixed endpoints where one model supports vision capabilities while another remains text-only, despite sharing the same base URL.

### Using setModelSupportsImages

The `setModelSupportsImages` function in [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts) provides a pure functional way to toggle image support for specific models without mutating the original configuration:

```typescript
export function setModelSupportsImages(
  connection: LlmConnection,
  modelId: string,
  enabled: boolean,
): LlmConnection

```

This helper automatically promotes string entries in the `models` array to objects when you apply overrides, ensuring type safety while maintaining backward compatibility with existing configurations.

## Runtime Configuration Flow

When a session initializes, [`packages/server-core/src/sessions/runtime-config.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/sessions/runtime-config.ts) builds the runtime model list by merging connection defaults with per-model overrides. The logic checks `typeof model.supportsImages === 'boolean'` to determine if an override exists before falling back to the connection-level default.

The final configuration propagates to the agent drivers (such as [`pi.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/pi.ts) or [`anthropic.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/anthropic.ts)), which use the resolved flag to determine whether to serialize image data in API requests.

## Configuration Examples

### JSON Schema for Custom Endpoints

Here is a complete configuration for an Ollama local instance with mixed model capabilities:

```json
{
  "slug": "ollama-local",
  "name": "Ollama (Local)",
  "providerType": "pi_compat",
  "authType": "api_key_with_endpoint",
  "baseUrl": "http://localhost:11434/v1",
  "customEndpoint": {
    "api": "openai-completions",
    "supportsImages": true
  },
  "models": [
    "gemma",
    { "id": "llama2", "supportsImages": false }
  ],
  "defaultModel": "gemma",
  "createdAt": 1720150800000
}

```

In this example, `gemma` inherits the `true` default from `customEndpoint.supportsImages`, while `llama2` explicitly overrides this to disable image support.

### Programmatic Toggle

To update capabilities programmatically:

```typescript
import { setModelSupportsImages } from '@/shared/config/llm-connections';
import { getLlmConnection, saveLlmConnection } from '@/shared/config/storage';

const conn = await getLlmConnection('ollama-local');
const updated = setModelSupportsImages(conn, 'llama2', true);
await saveLlmConnection(updated);

```

After persistence, the `modelSupportsImages` utility (defined at line 49 in [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts)) will return `true` for `llama2` and `true` for `gemma` (inheriting the default).

### RPC Update Method

The UI communicates changes via RPC to [`packages/server-core/src/handlers/rpc/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/handlers/rpc/llm-connections.ts):

```json
{
  "method": "LLM_Connection:update",
  "params": {
    "slug": "ollama-local",
    "modelId": "llama2",
    "supportsImages": true
  }
}

```

The server-side handler applies the update through `setModelSupportsImages` and broadcasts the change to running agent sessions, ensuring immediate capability updates without restarts.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts) | Defines `LlmConnection`, `CustomEndpointConfig`, and helper functions including `setModelSupportsImages` and `modelSupportsImages` |
| [`packages/shared/src/config/__tests__/set-model-supports-images.test.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/__tests__/set-model-supports-images.test.ts) | Test suite validating the promotion of string entries to objects and override precedence |
| [`packages/server-core/src/sessions/runtime-config.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/sessions/runtime-config.ts) | Merges connection defaults with per-model overrides at session startup |
| [`packages/shared/src/agent/backend/internal/drivers/pi.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/agent/backend/internal/drivers/pi.ts) | Consumes the final `supportsImages` flag when constructing API requests |
| [`packages/server-core/src/handlers/rpc/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/handlers/rpc/llm-connections.ts) | RPC entry point for UI-driven configuration updates |

## Summary

- Configure custom endpoints by setting `authType` to `api_key_with_endpoint` and providing a `baseUrl` and `customEndpoint` configuration in [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts).
- Define models in the `models` array as either strings (inheriting defaults) or objects with per-model flags like `supportsImages`.
- Use `setModelSupportsImages()` to programmatically toggle capabilities; this pure function returns a new `LlmConnection` while promoting string entries to objects automatically.
- The runtime resolves capabilities in [`packages/server-core/src/sessions/runtime-config.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/sessions/runtime-config.ts), giving per-model overrides precedence over connection-level defaults.
- Changes persist through `saveLlmConnection()` and propagate to running agents via the RPC handler in [`packages/server-core/src/handlers/rpc/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/handlers/rpc/llm-connections.ts).

## Frequently Asked Questions

### How do I add image support to only one model on a custom endpoint?

Define the model as an object with `supportsImages: true` in the `models` array while keeping the connection-level `customEndpoint.supportsImages` set to `false` (or omitting it). According to the runtime logic in [`packages/server-core/src/sessions/runtime-config.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/sessions/runtime-config.ts), the per-model boolean takes precedence over the connection default.

### Can I use the same custom endpoint configuration for both OpenAI-compatible and Anthropic-compatible servers?

Yes. Set the `customEndpoint.api` field to either `openai-completions` or `anthropic-messages` depending on your endpoint's protocol. The `LlmConnection` type in [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts) supports both protocols, and the agent driver selection depends on this field.

### What happens if I change a model's supportsImages flag while agents are running?

The RPC handler in [`packages/server-core/src/handlers/rpc/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/server-core/src/handlers/rpc/llm-connections.ts) applies the update via `setModelSupportsImages` and broadcasts the configuration change to active sessions. The runtime configuration updates immediately, so subsequent agent invocations use the new capability flag without requiring a server restart.

### How do I check if a specific model supports images in my code?

Use the `modelSupportsImages` utility exported from [`packages/shared/src/config/llm-connections.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/config/llm-connections.ts) (line 49). This function accepts an `LlmConnection` and a `modelId`. It returns the resolved boolean by checking for per-model overrides first, then falling back to the connection-level default if no override exists.