# How Vane Integrates with Ollama for Local LLMs: A Technical Deep Dive

> Discover how Vane integrates with Ollama for local LLMs. This technical deep dive explains the provider architecture, model discovery, and official npm client usage for seamless local AI.

- Repository: [Kushagra Srivastava/Vane](https://github.com/ItzCrazyKns/Vane)
- Tags: deep-dive
- Published: 2026-03-11

---

**Vane integrates with Ollama by implementing a provider-based architecture that registers `OllamaProvider` as a first-class model source, discovers available models via the `/api/tags` endpoint, and uses the official `ollama` npm client to handle chat completion, streaming, and embeddings through the `OllamaLLM` and `OllamaEmbedding` classes.**

Vane is an open-source AI platform that treats local Large Language Models as first-class citizens. By integrating with Ollama, Vane enables users to run chat completion and embedding models entirely on their local machines without external API keys or cloud dependencies.

## Provider Registration and Configuration

Vane discovers the Ollama integration through the `OllamaProvider` class located in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts). This class extends `BaseModelProvider` and is registered via the server-side registry function `getConfiguredModelProviderById`. The provider exposes configuration UI fields for the base URL, defaulting to `http://localhost:11434`, and parses connection settings to instantiate the underlying client.

## Discovering Local Models

When Vane initializes, it queries the Ollama server’s `/api/tags` endpoint (lines 34‑55 in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts)) to retrieve a list of locally installed models. The response is normalized into Vane’s internal `ModelList` shape, which categorizes models into `embedding` and `chat` types based on their capabilities. If the Ollama server is unreachable, the `getDefaultModels` method catches connection errors (lines 56‑64) and surfaces user-friendly messages instead of crashing.

## Chat Completion and Streaming

The `OllamaLLM` class in [`src/lib/models/providers/ollama/ollamaLLM.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/ollamaLLM.ts) handles all chat functionality using the `ollama` npm package (`^0.6.3`).

### Synchronous Generation

The `generateText` method (lines 70‑105) constructs a request payload containing the model name, converted message format, optional function-calling tools, and generation parameters such as `top_p` and `temperature`. For known reasoning models, it explicitly disables Ollama’s built-in "thinking" feature to ensure consistent output.

### Streaming Responses

For real-time interactions, `streamText` (lines 124‑180) opens a streaming connection by setting `stream: true` in the request. It yields partial `contentChunk` objects as they arrive from the local server and fabricates deterministic IDs when Ollama does not return them, enabling seamless tool-call handling during partial generations.

### Structured Output

The `generateObject` method (lines 84‑119) leverages Ollama’s `format` argument to request JSON output conforming to a Zod schema. Vane repairs potentially malformed JSON using the `@toolsycc/json-repair` library before parsing, ensuring robust structured data extraction from local models.

## Embedding Generation

Vane implements local embeddings through the `OllamaEmbedding` class in [`src/lib/models/providers/ollama/ollamaEmbedding.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/ollamaEmbedding.ts). Both `embedText` and `embedChunks` methods (lines 21‑38) instantiate an `Ollama` client with the configured host and call `ollamaClient.embed` with the selected model name. This returns a matrix of floating-point vectors suitable for semantic search and retrieval-augmented generation workflows without leaving the local environment.

## Error Handling and Resilience

Connection failures are handled gracefully within the provider initialization. The `getDefaultModels` function wraps the Ollama API call in a try-catch block (lines 56‑64 in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts)), returning an empty model list and descriptive error messages when the service is unavailable. This allows Vane to start successfully even when the Ollama daemon is offline, deferring model availability checks until the service recovers.

## Practical Implementation Examples

### Registering the Ollama Provider

Configure Vane to recognize your local Ollama instance by registering the provider with your base URL:

```typescript
// src/lib/config/serverRegistry.ts
import OllamaProvider from '@/lib/models/providers/ollama';

registerProvider(
  new OllamaProvider('ollama', 'Ollama', {
    baseURL: process.env.OLLAMA_BASE_URL ?? 'http://localhost:11434',
  })
);

```

### Loading a Chat Model and Sending Prompts

Retrieve the provider and load a specific model like `phi3` for single-turn conversations:

```typescript
import { getModelProvider } from '@/lib/config/serverRegistry';

async function chatWithOllama(prompt: string) {
  const provider = getModelProvider('ollama');
  const llm = await provider.loadChatModel('phi3');

  const response = await llm.generateText({
    messages: [{ role: 'user', content: prompt }],
  });

  console.log('Ollama reply:', response.content);
}

```

*Behind the scenes, `loadChatModel` creates an `OllamaLLM` instance (lines 91‑95 in [`index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/index.ts)).*

### Streaming Real-Time Responses

Process partial tokens as they generate for a responsive interface:

```typescript
async function streamChat(prompt: string) {
  const provider = getModelProvider('ollama');
  const llm = await provider.loadChatModel('phi3');

  const stream = llm.streamText({
    messages: [{ role: 'user', content: prompt }],
  });

  for await (const chunk of stream) {
    process.stdout.write(chunk.contentChunk);
    if (chunk.done) console.log('\n--- finished ---');
  }
}

```

### Generating Local Embeddings

Create vector representations of documents using a local embedding model:

```typescript
async function embedDocuments(docs: string[]) {
  const provider = getModelProvider('ollama');
  const embedder = await provider.loadEmbeddingModel('nomic-embed-text');

  const vectors = await embedder.embedText(docs);
  console.log('Embeddings:', vectors);
}

```

*`OllamaEmbedding.embedText` forwards texts to Ollama’s `/api/embed` endpoint (lines 21‑27).*

## Summary

- Vane registers Ollama via `OllamaProvider` extending `BaseModelProvider` in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts), enabling configuration through environment variables or UI fields.
- Model discovery queries the `/api/tags` endpoint to populate available chat and embedding models dynamically.
- Chat functionality in [`ollamaLLM.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/ollamaLLM.ts) supports synchronous generation, real-time streaming, and JSON-structured output with schema validation.
- Embeddings are handled by [`ollamaEmbedding.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/ollamaEmbedding.ts), wrapping the local `/api/embed` endpoint for vector generation.
- The integration relies on the official `ollama` npm package (`^0.6.3`) and defaults to `http://localhost:11434` for zero-configuration local deployment.

## Frequently Asked Questions

### How does Vane discover which models are installed in my local Ollama instance?

Vane queries the Ollama server's `/api/tags` endpoint through the `getDefaultModels` method in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts) (lines 34‑55). This returns a list of locally available models that Vane normalizes into its internal `ModelList` format, categorizing them as either chat or embedding models based on their capabilities.

### Can I use Ollama for both chat and embeddings in Vane?

Yes. Vane implements separate classes for each use case: `OllamaLLM` handles chat completion and streaming in [`src/lib/models/providers/ollama/ollamaLLM.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/ollamaLLM.ts), while `OllamaEmbedding` manages vector generation in [`src/lib/models/providers/ollama/ollamaEmbedding.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/ollamaEmbedding.ts). Both instantiate the `ollama` npm client with your configured base URL and support model-specific parameters.

### What happens if my Ollama server is not running when Vane starts?

If the Ollama server is unreachable, the `getDefaultModels` method catches connection errors (lines 56‑64 in [`src/lib/models/providers/ollama/index.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/src/lib/models/providers/ollama/index.ts)) and returns an empty model list with a user-friendly error message rather than crashing. The UI will indicate that no models are available until the Ollama service is started.

### Does Vane support streaming responses from Ollama models?

Yes. The `OllamaLLM.streamText` method (lines 124‑180 in [`ollamaLLM.ts`](https://github.com/ItzCrazyKns/Vane/blob/main/ollamaLLM.ts)) opens a persistent connection to Ollama's streaming endpoint by setting `stream: true` in the request payload. It yields partial content chunks in real-time and handles tool-call fragments, providing a responsive user experience for local inference.