# How to Run Llama3 Chinese Locally with Ollama: Complete Setup Guide

> Learn to run Llama3 Chinese locally with Ollama. Install Ollama, pull the model, and get an OpenAI compatible API for offline chat. Complete setup guide.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Install Ollama, pull the `shareai/llama3.1-dpo-zh` model, and access an OpenAI-compatible API at `http://localhost:11434/v1/chat/completions` to chat with Llama3 Chinese entirely offline.**

The `crazyboym/llama3-chinese-chat` repository provides a streamlined path to run Llama3 Chinese locally using **Ollama**, a lightweight wrapper around [`llama.cpp`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/llama.cpp) that exposes a full HTTP API. This approach eliminates the need for heavy PyTorch dependencies while delivering fast inference on both CPU and GPU. Whether you need a local Chinese chatbot for privacy or offline development, this guide shows you exactly how to run Llama3 Chinese locally with Ollama using the official deployment configuration.

## Prerequisites

Before starting, ensure you have a system capable of running 8B parameter models. The `shareai/llama3.1-dpo-zh` GGUF file requires approximately 8GB of disk space and sufficient RAM or VRAM for the quantized weights.

You will also need **Ollama** installed. The repository documentation in [`deploy/ollama/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/ollama/README.md) references the standard Ollama installation script for macOS and Linux.

## Step-by-Step Deployment

### Install Ollama

Download and install Ollama using the official installer. For Linux and macOS, run:

```bash
curl -fsSL https://ollama.com/install.sh | sh

```

This installs the Ollama binary which bundles [`llama.cpp`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/llama.cpp) and manages model files automatically.

### Pull the Llama3 Chinese Model

Execute the following command to download and cache the quantized model:

```bash
ollama run shareai/llama3.1-dpo-zh

```

The first execution downloads the GGUF file (approximately 8GB) to `~/.ollama/models` and immediately starts an interactive REPL. This command is documented in [`deploy/ollama/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/ollama/README.md) as the primary deployment method for the Chinese fine-tuned weights.

### Test the Local API with cURL

Ollama automatically starts an HTTP server on `http://localhost:11434`. Verify the installation by sending a chat completion request:

```bash
curl http://localhost:11434/v1/chat/completions -d '{
  "model": "shareai/llama3.1-dpo-zh",
  "messages": [{"role": "user", "content": "讲个笑话?"}],
  "stream": false
}'

```

The API returns a JSON response following the OpenAI schema, with the generated Chinese content available at `choices[0].message.content`.

### Integrate with Python

For programmatic access, use standard HTTP clients or the OpenAI Python SDK. Here is a minimal example using `requests`:

```python
import requests
import json

url = "http://localhost:11434/v1/chat/completions"
payload = {
    "model": "shareai/llama3.1-dpo-zh",
    "messages": [{"role": "user", "content": "介绍一下北京故宫"}],
    "temperature": 0.7,
    "stream": False,
}
headers = {"Content-Type": "application/json"}

response = requests.post(url, data=json.dumps(payload), headers=headers)
print(response.json()["choices"][0]["message"]["content"])

```

To use the official `openai` library, set `api_base="http://localhost:11434/v1"` and `api_key="ollama"` to route requests through your local instance.

## Architecture Overview

Understanding the stack helps troubleshoot performance issues. The deployment consists of three layers as outlined in the repository documentation:

- **GGUF Weights**: The model file stored in `~/.ollama/models` contains quantized 8B parameters. The repository references `shareai/llama3.1-8b-instruct-dpo-zh` as the source checkpoint.
- **Ollama Runtime**: A thin CLI that wraps [`llama.cpp`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/llama.cpp), handling tokenization, batching, and hardware detection (CUDA/ROCm/CPU fallback).
- **OpenAI-Compatible API**: Exposes endpoints at `/v1/chat/completions`, enabling drop-in replacement for OpenAI in existing tools like LangChain or LM Studio.

For comparison, the repository also provides [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) which demonstrates loading the same GGUF directly with `transformers`, and [`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py) for a browser-based UI alternative.

## Summary

Running Llama3 Chinese locally requires minimal configuration when using Ollama. Key steps include:

- Installing Ollama via the official shell script
- Pulling the `shareai/llama3.1-dpo-zh` model with a single command
- Accessing the OpenAI-compatible API at `http://localhost:11434/v1/chat/completions`
- Using standard HTTP clients or Python SDKs to interact with the model
- Leveraging optional UI components in `deploy/streamlit/` for non-technical users

## Frequently Asked Questions

### What hardware is required to run Llama3 Chinese locally?

You need approximately 8GB of disk space for the model file and sufficient RAM or VRAM to hold the quantized weights. Ollama automatically selects GPU acceleration when available, falling back to CPU inference if necessary.

### Can I use the official OpenAI Python SDK with Ollama?

Yes. Configure the client with `api_base="http://localhost:11434/v1"` and any placeholder API key. The endpoint accepts identical parameters to OpenAI's chat completions, including `temperature`, `top_p`, and `stream`.

### Where does Ollama store the downloaded model weights?

Ollama caches GGUF files in `~/.ollama/models` on Linux and macOS, and `%USERPROFILE%\.ollama\models` on Windows. The `shareai/llama3.1-dpo-zh` model persists here after the initial pull.

### How do I enable streaming responses?

Set `"stream": true` in your API request payload. The server returns Server-Sent Events (SSE) with token-by-token updates, reducing time-to-first-token for long generation tasks.