How Vane Integrates with Ollama for Local LLMs: A Technical Deep Dive

Vane integrates with Ollama by implementing a provider-based architecture that registers OllamaProvider as a first-class model source, discovers available models via the /api/tags endpoint, and uses the official ollama npm client to handle chat completion, streaming, and embeddings through the OllamaLLM and OllamaEmbedding classes.

Vane is an open-source AI platform that treats local Large Language Models as first-class citizens. By integrating with Ollama, Vane enables users to run chat completion and embedding models entirely on their local machines without external API keys or cloud dependencies.

Provider Registration and Configuration

Vane discovers the Ollama integration through the OllamaProvider class located in src/lib/models/providers/ollama/index.ts. This class extends BaseModelProvider and is registered via the server-side registry function getConfiguredModelProviderById. The provider exposes configuration UI fields for the base URL, defaulting to http://localhost:11434, and parses connection settings to instantiate the underlying client.

Discovering Local Models

When Vane initializes, it queries the Ollama server’s /api/tags endpoint (lines 34‑55 in src/lib/models/providers/ollama/index.ts) to retrieve a list of locally installed models. The response is normalized into Vane’s internal ModelList shape, which categorizes models into embedding and chat types based on their capabilities. If the Ollama server is unreachable, the getDefaultModels method catches connection errors (lines 56‑64) and surfaces user-friendly messages instead of crashing.

Chat Completion and Streaming

The OllamaLLM class in src/lib/models/providers/ollama/ollamaLLM.ts handles all chat functionality using the ollama npm package (^0.6.3).

Synchronous Generation

The generateText method (lines 70‑105) constructs a request payload containing the model name, converted message format, optional function-calling tools, and generation parameters such as top_p and temperature. For known reasoning models, it explicitly disables Ollama’s built-in "thinking" feature to ensure consistent output.

Streaming Responses

For real-time interactions, streamText (lines 124‑180) opens a streaming connection by setting stream: true in the request. It yields partial contentChunk objects as they arrive from the local server and fabricates deterministic IDs when Ollama does not return them, enabling seamless tool-call handling during partial generations.

Structured Output

The generateObject method (lines 84‑119) leverages Ollama’s format argument to request JSON output conforming to a Zod schema. Vane repairs potentially malformed JSON using the @toolsycc/json-repair library before parsing, ensuring robust structured data extraction from local models.

Embedding Generation

Vane implements local embeddings through the OllamaEmbedding class in src/lib/models/providers/ollama/ollamaEmbedding.ts. Both embedText and embedChunks methods (lines 21‑38) instantiate an Ollama client with the configured host and call ollamaClient.embed with the selected model name. This returns a matrix of floating-point vectors suitable for semantic search and retrieval-augmented generation workflows without leaving the local environment.

Error Handling and Resilience

Connection failures are handled gracefully within the provider initialization. The getDefaultModels function wraps the Ollama API call in a try-catch block (lines 56‑64 in src/lib/models/providers/ollama/index.ts), returning an empty model list and descriptive error messages when the service is unavailable. This allows Vane to start successfully even when the Ollama daemon is offline, deferring model availability checks until the service recovers.

Practical Implementation Examples

Registering the Ollama Provider

Configure Vane to recognize your local Ollama instance by registering the provider with your base URL:

// src/lib/config/serverRegistry.ts
import OllamaProvider from '@/lib/models/providers/ollama';

registerProvider(
  new OllamaProvider('ollama', 'Ollama', {
    baseURL: process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434',
  })
);

Loading a Chat Model and Sending Prompts

Retrieve the provider and load a specific model like phi3 for single-turn conversations:

import { getModelProvider } from '@/lib/config/serverRegistry';

async function chatWithOllama(prompt: string) {
  const provider = getModelProvider('ollama');
  const llm = await provider.loadChatModel('phi3');

  const response = await llm.generateText({
    messages: [{ role: 'user', content: prompt }],
  });

  console.log('Ollama reply:', response.content);
}

Behind the scenes, loadChatModel creates an OllamaLLM instance (lines 91‑95 in index.ts).

Streaming Real-Time Responses

Process partial tokens as they generate for a responsive interface:

async function streamChat(prompt: string) {
  const provider = getModelProvider('ollama');
  const llm = await provider.loadChatModel('phi3');

  const stream = llm.streamText({
    messages: [{ role: 'user', content: prompt }],
  });

  for await (const chunk of stream) {
    process.stdout.write(chunk.contentChunk);
    if (chunk.done) console.log('\n--- finished ---');
  }
}

Generating Local Embeddings

Create vector representations of documents using a local embedding model:

async function embedDocuments(docs: string[]) {
  const provider = getModelProvider('ollama');
  const embedder = await provider.loadEmbeddingModel('nomic-embed-text');

  const vectors = await embedder.embedText(docs);
  console.log('Embeddings:', vectors);
}

OllamaEmbedding.embedText forwards texts to Ollama’s /api/embed endpoint (lines 21‑27).

Summary

• Vane registers Ollama via OllamaProvider extending BaseModelProvider in src/lib/models/providers/ollama/index.ts, enabling configuration through environment variables or UI fields.
• Model discovery queries the /api/tags endpoint to populate available chat and embedding models dynamically.
• Chat functionality in ollamaLLM.ts supports synchronous generation, real-time streaming, and JSON-structured output with schema validation.
• Embeddings are handled by ollamaEmbedding.ts, wrapping the local /api/embed endpoint for vector generation.
• The integration relies on the official ollama npm package (^0.6.3) and defaults to http://localhost:11434 for zero-configuration local deployment.

Frequently Asked Questions

How does Vane discover which models are installed in my local Ollama instance?

Vane queries the Ollama server's /api/tags endpoint through the getDefaultModels method in src/lib/models/providers/ollama/index.ts (lines 34‑55). This returns a list of locally available models that Vane normalizes into its internal ModelList format, categorizing them as either chat or embedding models based on their capabilities.

Can I use Ollama for both chat and embeddings in Vane?

Yes. Vane implements separate classes for each use case: OllamaLLM handles chat completion and streaming in src/lib/models/providers/ollama/ollamaLLM.ts, while OllamaEmbedding manages vector generation in src/lib/models/providers/ollama/ollamaEmbedding.ts. Both instantiate the ollama npm client with your configured base URL and support model-specific parameters.

What happens if my Ollama server is not running when Vane starts?

If the Ollama server is unreachable, the getDefaultModels method catches connection errors (lines 56‑64 in src/lib/models/providers/ollama/index.ts) and returns an empty model list with a user-friendly error message rather than crashing. The UI will indicate that no models are available until the Ollama service is started.

Does Vane support streaming responses from Ollama models?

Yes. The OllamaLLM.streamText method (lines 124‑180 in ollamaLLM.ts) opens a persistent connection to Ollama's streaming endpoint by setting stream: true in the request payload. It yields partial content chunks in real-time and handles tool-call fragments, providing a responsive user experience for local inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}