How to Add a New LLM Provider to pi-ai: A Complete Implementation Guide

Adding a new LLM provider to pi-ai requires extending the type definitions in packages/ai/src/types.ts, implementing a streaming function in src/providers/<provider>.ts, registering the API in register-builtins.ts, and adding model discovery logic to scripts/generate-models.ts.

The pi-ai package in the badlogic/pi-mono repository provides a unified, type-safe interface for streaming LLM responses across multiple providers. Whether you need to integrate a proprietary API or a self-hosted model, adding a new LLM provider to pi-ai follows a consistent architectural pattern that maps providers to APIs and models through a thin abstraction layer.

Understanding the pi-ai Provider Architecture

Pi-ai is built around a thin, type-safe layer that maps providers → APIs → models. The architecture consists of four main touchpoints:

• Core type definitions (packages/ai/src/types.ts) – Union types for APIs, providers, and streaming options
• Provider implementation (src/providers/<provider>.ts) – The streaming logic that translates pi-ai's generic interface to provider-specific API calls
• API registry (src/providers/register-builtins.ts) – Global registration of the provider and its environment variable detection
• Model discovery (scripts/generate-models.ts) – Automated fetching of the provider's model catalog

Step-by-Step Implementation Guide

Extend Core Type Definitions

Begin by extending the union types in packages/ai/src/types.ts to include your new provider and its API identifier:

// packages/ai/src/types.ts
export type KnownApi = 
  | "openai-api"
  | "anthropic-api"
  | "amazon-bedrock-api"
  | "myprovider-api";  // Add your API here

export type KnownProvider = 
  | "openai"
  | "anthropic"
  | "amazon-bedrock"
  | "myprovider";  // Add your provider here

export interface MyProviderOptions extends StreamOptions {
  region?: string;          // Provider-specific flags
  toolChoice?: "auto" | "none";
}

Implement the Provider Logic

Create a new file at packages/ai/src/providers/myprovider.ts that implements the stream<Provider>() function. Use the existing amazon-bedrock.ts implementation as a reference for structure:

// packages/ai/src/providers/myprovider.ts
import {
  AssistantMessageEventStream,
} from "../utils/event-stream.js";
import {
  Model,
  Context,
  StreamFunction,
  AssistantMessage,
  TextContent,
  ToolCall,
} from "../types.js";

export interface MyProviderOptions {
  region?: string;
  toolChoice?: "auto" | "none";
}

export const streamMyProvider: StreamFunction<"myprovider-api", MyProviderOptions> = (
  model: Model<"myprovider-api">,
  context: Context,
  opts: MyProviderOptions = {}
) => {
  const stream = new AssistantMessageEventStream();

  (async () => {
    const output: AssistantMessage = {
      role: "assistant",
      content: [],
      api: "myprovider-api",
      provider: model.provider,
      model: model.id,
      usage: { 
        input: 0, 
        output: 0, 
        cacheRead: 0, 
        cacheWrite: 0, 
        totalTokens: 0,
        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } 
      },
      stopReason: "stop",
      timestamp: Date.now(),
    };

    // Build request payload using helper functions
    const payload = {
      model: model.id,
      messages: context.messages,
      temperature: opts.temperature,
      max_tokens: opts.maxTokens,
      // Provider-specific parameters
      region: opts.region,
      tool_choice: opts.toolChoice,
    };
    opts.onPayload?.(payload);

    // Issue HTTP request
    const resp = await fetch("https://api.myprovider.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${opts.apiKey ?? process.env.MYPROVIDER_API_KEY}`,
        ...opts.headers,
      },
      body: JSON.stringify(payload),
      signal: opts.signal,
    });

    // Parse streaming response
    const reader = resp.body!.getReader();
    const decoder = new TextDecoder();
    let buffer = "";
    
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split("\n");
      buffer = lines.pop() ?? "";
      
      for (const line of lines) {
        if (!line) continue;
        const event = JSON.parse(line);
        
        // Translate provider events to pi-ai streaming events
        if (event.type === "text") {
          const block: TextContent = { type: "text", text: event.content };
          output.content.push(block);
          stream.push({ 
            type: "text_delta", 
            delta: event.content, 
            contentIndex: output.content.length - 1 
          });
        } else if (event.type === "tool_call") {
          const call: ToolCall = {
            type: "toolCall",
            id: event.id,
            name: event.name,
            arguments: event.arguments,
          };
          output.content.push(call);
          stream.push({ type: "toolcall_end", toolCall: call });
        }
      }
    }

    stream.push({ type: "done", reason: output.stopReason, message: output });
    stream.end();
  })().catch(err => {
    stream.push({ 
      type: "error", 
      reason: "error", 
      error: { ...output, errorMessage: err.message } 
    });
    stream.end();
  });

  return stream;
};

Register the API Provider

Register your implementation in packages/ai/src/providers/register-builtins.ts to make it available globally:

// packages/ai/src/providers/register-builtins.ts
import { streamMyProvider } from "./myprovider.js";

registerApiProvider({
  api: "myprovider-api",
  stream: streamMyProvider,
  // Optional: provide a simplified streaming interface
  streamSimple: streamSimpleMyProvider,
});

Add Environment Variable Detection

If your provider requires an API key, add detection logic in packages/ai/src/env-api-keys.ts:

// packages/ai/src/env-api-keys.ts
export function getApiKey(provider: KnownProvider): string | undefined {
  // ... existing providers ...
  
  if (provider === "myprovider") {
    return process.env.MYPROVIDER_API_KEY;
  }
  
  return undefined;
}

Generate Model Metadata

Add a discovery function in packages/ai/scripts/generate-models.ts to fetch your provider's model catalog:

// packages/ai/scripts/generate-models.ts
async function fetchMyProviderModels(): Promise<Model<any>[]> {
  const res = await fetch("https://api.myprovider.com/v1/models");
  const data = await res.json();
  
  return data.models
    .filter(m => m.tool_call)  // Keep only tool-capable models
    .map(m => ({
      id: m.id,
      name: m.name,
      api: "myprovider-api",
      provider: "myprovider",
      baseUrl: "https://api.myprovider.com/v1",
      reasoning: m.reasoning ?? false,
      input: m.modalities?.includes("image") ? ["text", "image"] : ["text"],
      cost: {
        input: m.pricing?.input ?? 0,
        output: m.pricing?.output ?? 0,
        cacheRead: m.pricing?.cacheRead ?? 0,
        cacheWrite: m.pricing?.cacheWrite ?? 0,
      },
      contextWindow: m.limit?.context ?? 4096,
      maxTokens: m.limit?.output ?? 4096,
    }));
}

Optional Integration Points

Coding Agent Defaults

If you want the coding agent to support your provider out of the box, add a default model ID in packages/coding-agent/src/core/model-resolver.ts.

Documentation and Changelog

Update the Supported Providers table in packages/ai/README.md with authentication requirements and supported features. Add an entry under ## [Unreleased] in packages/ai/CHANGELOG.md describing the new provider integration.

Testing Your New Provider

Before submitting your implementation, ensure comprehensive test coverage. Copy an existing provider's test suite (such as packages/ai/test/stream.test.ts) and modify it to test your provider:

• Basic streaming: Verify text generation and event streaming
• Tool usage: Test function calling and tool result handling
• Token accounting: Validate input/output token counting and cost calculation
• Abort handling: Ensure AbortSignal properly cancels requests
• Edge cases: Empty payloads, context overflow, and image handling (if applicable)

Run the test suite with npm run check and targeted vitest runs to verify your implementation doesn't break existing providers.

Summary

• Type definitions: Extend KnownApi and KnownProvider unions in packages/ai/src/types.ts to include your provider
• Implementation: Create packages/ai/src/providers/<provider>.ts implementing the StreamFunction type with proper event streaming
• Registration: Call registerApiProvider() in packages/ai/src/providers/register-builtins.ts to expose your provider globally
• Authentication: Add environment variable detection in packages/ai/src/env-api-keys.ts if your provider requires API keys
• Model discovery: Implement a fetch function in packages/ai/scripts/generate-models.ts to pull available models and pricing data
• Testing: Copy existing provider tests and verify streaming, tool calls, token counting, and abort handling before submitting

Frequently Asked Questions

Do I need to modify the coding agent to add a new LLM provider to pi-ai?

No, modifying the coding agent is optional. The core provider integration only requires changes to the packages/ai module. However, if you want the coding agent to use your provider by default, you can add a sensible default model ID in packages/coding-agent/src/core/model-resolver.ts so that the agent automatically selects your provider when no specific model is requested.

How does pi-ai handle authentication for custom providers?

Pi-ai uses a centralized environment variable detection system located in packages/ai/src/env-api-keys.ts. When adding a new provider, you implement a check for your specific provider identifier that returns the appropriate environment variable (such as process.env.MYPROVIDER_API_KEY). The streaming function can then access this key via opts.apiKey or fall back to the environment detection, ensuring secure credential handling without hardcoding secrets.

Can I add a provider that doesn't support streaming responses?

Yes, though pi-ai is optimized for streaming. If your provider only supports batch responses, you can still implement the StreamFunction interface by buffering the complete response and then emitting it as a single text_delta event followed by done. However, for the best integration with pi-ai's event-driven architecture, you should implement true streaming if the provider supports it via Server-Sent Events (SSE) or similar mechanisms, parsing chunks as they arrive and pushing events to the AssistantMessageEventStream.

What testing is required before submitting a new provider implementation?

You should copy an existing provider's test suite (such as packages/ai/test/stream.test.ts) and adapt it for your provider. Essential test coverage includes: basic streaming functionality to verify text generation, tool usage testing for function calling capabilities, token accounting validation to ensure accurate cost calculation, abort handling to verify AbortSignal cancellation works correctly, and edge cases like empty payloads or context overflow. Run npm run check and targeted vitest runs to ensure your changes don't break existing providers before submitting your pull request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}