How to Add Custom LLM Model Adapters to Harvey AI: Step-by-Step Guide

Harvey AI uses a pluggable ModelAdapter abstraction layer in harness/adapters/base.py that lets you integrate any LLM provider by subclassing the base class and implementing four abstract methods.

Harvey AI ships with built-in support for Anthropic, OpenAI, Google, Mistral, Fireworks, and Baseten. When you need to connect a proprietary or niche LLM service, the adapter pattern in the harveyai/harvey-labs repository provides a clean, documented interface without modifying core agent logic.

Understanding the ModelAdapter Architecture

The adapter system lives in harness/adapters/. The base class (base.py) defines a stable contract that the agent loop, evaluation scripts, and tooling layer all depend on. Each provider implementation isolates API-specific code in its own module.

Key design benefits of this approach:

• Separation of concerns: Harvey's core logic works with canonical message formats; adapters handle provider-specific translations.
• Testability: Each adapter can be unit-tested independently against recorded API responses.
• Extensibility: New providers require no changes to harness/agent_loop.py or tool definitions.

The Four Methods Every Custom Adapter Must Implement

Your subclass must implement these abstract methods from ModelAdapter:

chat(messages, tools) → ModelResponse

This is the primary inference method. It accepts:

• messages: A list of standardized message dicts (system, user, assistant roles).
• tools: Canonical JSON-Schema tool definitions from harness/tools.py.

It returns a ModelResponse containing the assistant message, any tool calls, extracted text, and token usage counts.

make_tool_result_messages(results) → list[dict]

Transforms (tool_call_id, result) tuples into provider-specific result messages. Different providers handle tool results differently—Anthropic requires a single batched user message, while OpenAI and Google accept one message per result.

make_system_message(content) → dict

Returns a system-role message formatted for the provider's native schema.

make_user_message(content) → dict

Returns a user-role message formatted for the provider's native schema.

Step-by-Step: Creating a Custom LLM Adapter

Step 1: Create the Adapter Module

Create a new file in harness/adapters/ following the naming convention of existing adapters:

touch harness/adapters/foo.py

Step 2: Subclass ModelAdapter and Implement Required Methods

Below is a minimal, fully-functional template based on patterns from harness/adapters/openai.py and harness/adapters/anthropic.py:

# File: harness/adapters/foo.py

"""FooAI adapter – a template for adding custom LLM providers."""

import json
from harness.adapters.base import ModelAdapter, ModelResponse, ToolCall


class FooAIAdapter(ModelAdapter):
    """Adapter for FooAI's language-model API."""

    def __init__(
        self,
        model: str,
        temperature: float = 0.0,
        max_tokens: int = 4096,
        reasoning_effort: str | None = None,
    ):
        super().__init__(model, temperature, reasoning_effort)
        self.max_tokens = max_tokens
        # Initialize your provider's SDK client here

        self.client = self._initialize_client()
        self._system_prompt: str | None = None

    def chat(self, messages: list[dict], tools: list[dict]) -> ModelResponse:
        """Execute chat completion with tool support."""
        # Extract system prompt from message history

        system = ""
        conversation_messages = []
        for msg in messages:
            if msg["role"] == "system":
                system = msg["content"]
            else:
                conversation_messages.append(msg)

        # Convert canonical tool definitions to provider format

        provider_tools = [self._translate_tool(t) for t in tools]

        # Execute provider-specific API call

        raw_response = self._call_provider_api(
            system=system,
            messages=conversation_messages,
            tools=provider_tools,
        )

        # Parse provider response into canonical structures

        tool_calls: list[ToolCall] = []
        text_parts: list[str] = []

        for item in raw_response["output"]:
            if item["type"] == "tool_call":
                tool_calls.append(
                    ToolCall(
                        id=item["call_id"],
                        name=item["name"],
                        arguments=json.dumps(item["arguments"]),
                    )
                )
            elif item["type"] == "text":
                text_parts.append(item["content"])

        # Construct message for conversation history

        assistant_message = {
            "role": "assistant",
            "content": raw_response["output"],
        }

        return ModelResponse(
            message=assistant_message,
            tool_calls=tool_calls,
            text="\n".join(text_parts),
            input_tokens=raw_response.get("usage", {}).get("input_tokens", 0),
            output_tokens=raw_response.get("usage", {}).get("output_tokens", 0),
        )

    def make_tool_result_messages(
        self, results: list[tuple[str, str]]
    ) -> list[dict]:
        """Format tool execution results for provider's expected structure."""
        messages = []
        for tool_call_id, result in results:
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tool_call_id,
                    "content": str(result),
                }
            )
        return messages

    def make_system_message(self, content: str) -> dict:
        """Create system message in canonical format."""
        self._system_prompt = content
        return {"role": "system", "content": content}

    def make_user_message(self, content: str) -> dict:
        """Create user message in canonical format."""
        return {"role": "user", "content": content}

    def _translate_tool(self, tool: dict) -> dict:
        """Convert canonical tool definition to provider-specific schema."""
        return {
            "type": "function",
            "function": {
                "name": tool["name"],
                "description": tool["description"],
                "parameters": tool["parameters"],
            },
        }

    def _initialize_client(self):
        """Initialize provider SDK client. Override with actual implementation."""
        raise NotImplementedError("Configure your provider's client initialization")

    def _call_provider_api(self, system: str, messages: list, tools: list) -> dict:
        """Execute actual API call to provider. Override with implementation."""
        raise NotImplementedError("Implement provider-specific API call")

Step 3: Handle Tool Definition Translation

The canonical tool format used throughout Harvey AI follows JSON-Schema conventions defined in harness/tools.py. Your adapter's _translate_tool method maps this to your provider's expected structure.

Reference implementations to study:

• harness/adapters/openai.py: Maps to OpenAI's function-calling schema with strict mode support.
• harness/adapters/anthropic.py: Maps to Anthropic's tool use format with tool_choice handling.

Step 4: Register Your Adapter

Harvey AI discovers adapters through the model_provider configuration field. The harness imports the module matching your provider name and instantiates the corresponding adapter class.

Two registration patterns are supported:

Automatic discovery (recommended): Name your file foo.py and class FooAIAdapter. Set model_provider: "foo" in your config.

Explicit registration: Add an import mapping in harness/__init__.py or your configuration loader:

from harness.adapters.foo import FooAIAdapter

ADAPTER_REGISTRY = {
    "openai": OpenAIAdapter,
    "anthropic": AnthropicAdapter,
    "foo": FooAIAdapter,  # your custom adapter

}

Step 5: Test Your Adapter

Create tests in tests/test_adapters.py following patterns from existing adapter tests:

# tests/test_adapters.py additions

import pytest
from harness.adapters.foo import FooAIAdapter
from harness.adapters.base import ModelResponse, ToolCall


def test_foo_adapter_instantiation():
    """Verify adapter can be created with standard parameters."""
    adapter = FooAIAdapter(model="foo-large-v1", temperature=0.2)
    assert adapter.model == "foo-large-v1"
    assert adapter.temperature == 0.2


def test_foo_adapter_make_messages():
    """Verify system and user message formatting."""
    adapter = FooAIAdapter(model="foo-large-v1")
    
    system_msg = adapter.make_system_message("Be helpful.")
    assert system_msg["role"] == "system"
    assert system_msg["content"] == "Be helpful."
    
    user_msg = adapter.make_user_message("Hello!")
    assert user_msg["role"] == "user"


def test_foo_adapter_tool_results():
    """Verify tool result message formatting matches provider expectations."""
    adapter = FooAIAdapter(model="foo-large-v1")
    results = [("call_123", "result data"), ("call_456", "more data")]
    
    messages = adapter.make_tool_result_messages(results)
    assert len(messages) == 2
    assert all(m["role"] == "tool" for m in messages)
    assert messages[0]["tool_call_id"] == "call_123"

Using Your Custom LLM Adapter

Once implemented, use your adapter identically to built-in providers:

from harness.adapters.foo import FooAIAdapter
from harness.tools import TOOL_DEFINITIONS

# Initialize adapter

adapter = FooAIAdapter(
    model="foo-large-v1",
    temperature=0.2,
    max_tokens=4096,
)

# Build conversation

messages = [
    adapter.make_system_message("You are a legal research assistant."),
    adapter.make_user_message("Draft a contract clause for IP assignment."),
]

# Execute with tool support

response = adapter.chat(messages, TOOL_DEFINITIONS)

print(f"Response: {response.text}")
print(f"Tokens used: {response.input_tokens} in, {response.output_tokens} out")

if response.tool_calls:
    for call in response.tool_calls:
        print(f"Tool requested: {call.name}({call.arguments})")

Key Source Files for Reference

File	Purpose
harness/adapters/base.py	Abstract ModelAdapter, ModelResponse, and ToolCall dataclasses
harness/adapters/openai.py	OpenAI Responses API implementation with streaming support
harness/adapters/anthropic.py	Anthropic Messages API with adaptive-thinking and batch tool results
harness/agent_loop.py	Adapter instantiation and conversation orchestration
tests/test_adapters.py	Contract tests for all adapter implementations

Summary

Adding custom LLM model adapters to Harvey AI requires implementing a four-method contract defined in harness/adapters/base.py:

• chat() handles inference and response parsing.
• make_tool_result_messages() formats tool execution feedback.
• make_system_message() and make_user_message() create role-formatted messages.

The adapter pattern isolates provider-specific code, maintains testability, and requires no changes to Harvey's core agent logic. Study harness/adapters/openai.py and harness/adapters/anthropic.py for production reference implementations.

Frequently Asked Questions

What happens if my LLM provider doesn't support tool calling?

Implement chat() to return an empty tool_calls list and raise an appropriate error if tools are passed. Your adapter can still function for non-tool use cases, or you can implement tool-calling via prompt-based techniques. The harness/adapters/base.py interface does not require functional tool support.

Can I add streaming support to my custom adapter?

Yes. The base ModelAdapter can be extended with streaming methods. Study event-streaming patterns in harness/adapters/openai.py for the stream=True parameter handling. Ensure your ModelResponse accumulation logic handles partial chunks correctly.

How do I handle provider-specific authentication in my adapter?

Initialize credentials in your adapter's __init__ method or via environment variables. Follow the pattern in harness/adapters/anthropic.py, which retrieves API keys through os.environ.get("ANTHROPIC_API_KEY") with clear error messages on missing configuration.

Does Harvey AI support multiple custom adapters simultaneously?

Yes. The configuration system selects adapters per-conversation or per-deployment. Register multiple custom providers following the naming conventions, then specify the desired model_provider in your runtime configuration or environment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}