How to Run Llama3 Chinese Locally with Ollama: Complete Setup Guide

Install Ollama, pull the shareai/llama3.1-dpo-zh model, and access an OpenAI-compatible API at http://localhost:11434/v1/chat/completions to chat with Llama3 Chinese entirely offline.

The crazyboym/llama3-chinese-chat repository provides a streamlined path to run Llama3 Chinese locally using Ollama, a lightweight wrapper around llama.cpp that exposes a full HTTP API. This approach eliminates the need for heavy PyTorch dependencies while delivering fast inference on both CPU and GPU. Whether you need a local Chinese chatbot for privacy or offline development, this guide shows you exactly how to run Llama3 Chinese locally with Ollama using the official deployment configuration.

Prerequisites

Before starting, ensure you have a system capable of running 8B parameter models. The shareai/llama3.1-dpo-zh GGUF file requires approximately 8GB of disk space and sufficient RAM or VRAM for the quantized weights.

You will also need Ollama installed. The repository documentation in deploy/ollama/README.md references the standard Ollama installation script for macOS and Linux.

Step-by-Step Deployment

Install Ollama

Download and install Ollama using the official installer. For Linux and macOS, run:

curl -fsSL https://ollama.com/install.sh | sh

This installs the Ollama binary which bundles llama.cpp and manages model files automatically.

Pull the Llama3 Chinese Model

Execute the following command to download and cache the quantized model:

ollama run shareai/llama3.1-dpo-zh

The first execution downloads the GGUF file (approximately 8GB) to ~/.ollama/models and immediately starts an interactive REPL. This command is documented in deploy/ollama/README.md as the primary deployment method for the Chinese fine-tuned weights.

Test the Local API with cURL

Ollama automatically starts an HTTP server on http://localhost:11434. Verify the installation by sending a chat completion request:

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "shareai/llama3.1-dpo-zh",
  "messages": [{"role": "user", "content": "讲个笑话?"}],
  "stream": false
}'

The API returns a JSON response following the OpenAI schema, with the generated Chinese content available at choices[0].message.content.

Integrate with Python

For programmatic access, use standard HTTP clients or the OpenAI Python SDK. Here is a minimal example using requests:

import requests
import json

url = "http://localhost:11434/v1/chat/completions"
payload = {
    "model": "shareai/llama3.1-dpo-zh",
    "messages": [{"role": "user", "content": "介绍一下北京故宫"}],
    "temperature": 0.7,
    "stream": False,
}
headers = {"Content-Type": "application/json"}

response = requests.post(url, data=json.dumps(payload), headers=headers)
print(response.json()["choices"][0]["message"]["content"])

To use the official openai library, set api_base="http://localhost:11434/v1" and api_key="ollama" to route requests through your local instance.

Architecture Overview

Understanding the stack helps troubleshoot performance issues. The deployment consists of three layers as outlined in the repository documentation:

• GGUF Weights: The model file stored in ~/.ollama/models contains quantized 8B parameters. The repository references shareai/llama3.1-8b-instruct-dpo-zh as the source checkpoint.
• Ollama Runtime: A thin CLI that wraps llama.cpp, handling tokenization, batching, and hardware detection (CUDA/ROCm/CPU fallback).
• OpenAI-Compatible API: Exposes endpoints at /v1/chat/completions, enabling drop-in replacement for OpenAI in existing tools like LangChain or LM Studio.

For comparison, the repository also provides deploy/python/chat_demo.py which demonstrates loading the same GGUF directly with transformers, and deploy/streamlit/web_llama3_chat.py for a browser-based UI alternative.

Summary

Running Llama3 Chinese locally requires minimal configuration when using Ollama. Key steps include:

• Installing Ollama via the official shell script
• Pulling the shareai/llama3.1-dpo-zh model with a single command
• Accessing the OpenAI-compatible API at http://localhost:11434/v1/chat/completions
• Using standard HTTP clients or Python SDKs to interact with the model
• Leveraging optional UI components in deploy/streamlit/ for non-technical users

Frequently Asked Questions

What hardware is required to run Llama3 Chinese locally?

You need approximately 8GB of disk space for the model file and sufficient RAM or VRAM to hold the quantized weights. Ollama automatically selects GPU acceleration when available, falling back to CPU inference if necessary.

Can I use the official OpenAI Python SDK with Ollama?

Yes. Configure the client with api_base="http://localhost:11434/v1" and any placeholder API key. The endpoint accepts identical parameters to OpenAI's chat completions, including temperature, top_p, and stream.

Where does Ollama store the downloaded model weights?

Ollama caches GGUF files in ~/.ollama/models on Linux and macOS, and %USERPROFILE%\.ollama\models on Windows. The shareai/llama3.1-dpo-zh model persists here after the initial pull.

How do I enable streaming responses?

Set "stream": true in your API request payload. The server returns Server-Sent Events (SSE) with token-by-token updates, reducing time-to-first-token for long generation tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}