# How to Set Up a Local Ollama Server for AI Agents: Complete Setup Guide

> Set up a local Ollama server for AI agents easily. Follow this complete guide to install Ollama, pull models, and connect your agent code to the local endpoint for powerful AI integration.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-08-22

---

**You can set up a local Ollama server for AI agents by installing Ollama on your platform, launching the daemon with `ollama serve`, pulling the `qwen3:0.6b` model, and configuring your agent code to point to the OpenAI-compatible endpoint at `http://127.0.0.1:11434`.**

The **bojieli/ai-agent-book** repository uses Ollama as its default local LLM backend for agent demonstrations including log sanitization, tool-calling, and active-tool discovery. This guide covers the exact installation steps and configuration parameters found in the source code, ensuring your local AI agents can run entirely offline without external API dependencies.

## Installation and Platform Setup

Ollama supports macOS, Windows, and Linux environments. The repository provides specific installation commands for each platform in the local LLM serving documentation.

### macOS Installation

For macOS users, the recommended approach uses Homebrew. According to the log-sanitization chapter, you can install via:

```bash
brew install ollama

```

This command appears in [`chapter3/log-sanitization/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/log-sanitization/README.md) and represents the quickest method to get the Ollama binary on macOS systems.

### Windows and Linux Installation

Windows users should download the installer directly from the official Ollama website, as referenced in [`chapter2/local_llm_serving/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/local_llm_serving/README.md). Linux users can run the official install script provided by Ollama. Both methods establish the same HTTP API foundation that the AI Agent Book expects.

## Starting the Ollama Daemon and Pulling Models

Once installed, you must start the Ollama daemon to expose the local API endpoint on port **11434**.

### Launch the Server

Run the following command to start the background service:

```bash
ollama serve

```

This launches the local server on the default port **11434**. The daemon must remain running to accept `POST` requests to `/v1/chat/completions` from your AI agents.

### Download the Default Model

The repository defaults to **qwen3:0.6b** for its demonstrations because it is fast, free, and sufficient for agent workflows. Pull this model using:

```bash
ollama pull qwen3:0.6b

```

As documented in [`chapter3/log-sanitization/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/log-sanitization/README.md), this model is explicitly referenced in the configuration files for log sanitization and other agent experiments.

## Verification and Agent Configuration

Before running agent code, verify the server responds correctly and configure your application to route LLM calls through Ollama.

### Verify Server Connectivity

Test that the server is reachable and the model is loaded:

```bash
curl -s http://127.0.0.1:11434/api/tags | jq .models[0].name

```

A successful response returns `qwen3:0.6b`, confirming the endpoint is active and the model is available.

### Configure the Agent Provider

In [`chapter3/log-sanitization/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/log-sanitization/config.py), the repository defines Ollama-specific settings that route all LLM calls to your local server:

```python
import os

PROVIDER = "ollama"  # Tells the framework to use Ollama backend

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://127.0.0.1:11434")
MODEL = "qwen3:0.6b"

```

The code uses the `OPENAI_BASE_URL` convention internally, allowing standard OpenAI client libraries to communicate with Ollama's compatible interface. Setting `PROVIDER = "ollama"` switches the agent framework from cloud APIs to your local instance.

## Running AI Agents with the Local Server

With the server running and configuration set, execute agent experiments using the local model. For the log-sanitization demonstration:

```bash
python -m chapter3.log_sanitization.main --mode llm

```

This command invokes the agent pipeline (reasoning → tool calls → output) powered entirely by your local Ollama server rather than external APIs.

### Native Client Implementation

The repository includes [`ollama_native.py`](https://github.com/bojieli/ai-agent-book/blob/main/ollama_native.py), which implements the direct HTTP client used by all experiments. This file contains the `chat()` and `chat_stream()` functions that format requests and parse responses from the Ollama API endpoint.

## Advanced: Structured Outputs

Ollama supports JSON-structured outputs for deterministic agent processing. To enable this, add the `response_format` parameter to your API calls:

```python
payload = {
    "model": "qwen3:0.6b",
    "messages": messages,
    "stream": False,
    "response_format": {"type": "json_object"}
}

```

This feature is particularly useful for downstream processing tasks like PII redaction or tool argument parsing, allowing agents to receive predictable, machine-readable responses.

## Summary

- **Install Ollama** using platform-specific commands (`brew install ollama` for macOS, official installers for Windows/Linux) as documented in [`chapter2/local_llm_serving/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/local_llm_serving/README.md).
- **Start the daemon** with `ollama serve` to expose the API on port **11434**; verify connectivity via `curl` to `/api/tags`.
- **Pull the default model** `qwen3:0.6b` using `ollama pull`, matching the configuration in [`chapter3/log-sanitization/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/log-sanitization/config.py).
- **Configure agents** by setting `PROVIDER = "ollama"` and `OLLAMA_HOST` to point to your local server endpoint.
- **Run experiments** using module commands like `python -m chapter3.log_sanitization.main --mode llm` to execute AI agent workflows offline.

## Frequently Asked Questions

### What port does the local Ollama server use by default?

The default port is **11434**. When you run `ollama serve`, the daemon binds to `http://127.0.0.1:11434`, and this is the endpoint you configure in `OLLAMA_HOST` to route AI agent requests locally.

### Can I use a different model than qwen3:0.6b?

Yes. While the AI Agent Book defaults to `qwen3:0.6b` for its demonstrations, you can replace it with any Ollama-supported model such as `llama3.1:8b`. Simply run `ollama pull <model-name>` and update the `MODEL` variable in your [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py) file to match your selection.

### How does the agent code switch between cloud and local LLM providers?

The framework checks the `PROVIDER` variable defined in configuration files like [`chapter3/log-sanitization/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/log-sanitization/config.py). When set to `"ollama"`, the code routes requests through the Ollama client implementation in [`ollama_native.py`](https://github.com/bojieli/ai-agent-book/blob/main/ollama_native.py) using the `OPENAI_BASE_URL` pattern; otherwise, it uses standard cloud API endpoints.

### Does Ollama support streaming responses for AI agents?

Yes. The [`ollama_native.py`](https://github.com/bojieli/ai-agent-book/blob/main/ollama_native.py) file implements both `chat()` and `chat_stream()` functions, allowing agents to receive real-time token streams or complete responses depending on the `stream` parameter in the request payload.