# Deploying Llama3 Chinese with vLLM: Complete Setup Guide for OpenAI-Compatible APIs

> Deploy Llama3 Chinese models with vLLM and expose OpenAI compatible APIs. Follow our complete setup guide for efficient deployment using the crazyboym/llama3-chinese-chat repository.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Deploy Llama3 Chinese models using vLLM to expose OpenAI-compatible `/v1/chat/completions` endpoints, or use the alternative Streamlit and Flask interfaces provided in the `crazyboym/llama3-chinese-chat` repository.**

The `crazyboym/llama3-chinese-chat` repository provides a production-ready pipeline for fine-tuned Llama 3 models specialized for Chinese dialogue. Whether you need high-throughput API serving or interactive prototyping tools, the project supports **vLLM** deployment alongside Streamlit web UIs and lightweight Python servers. This guide covers the complete architecture—from model loading and prompt templating to launching scalable inference endpoints.

## Model Architecture and Loading Pipeline

The repository implements a three-layer architecture defined in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md) that handles everything from quantization to dialogue management.

### Model and Tokenizer Loader

The `load_model()` function handles FP16 or 4-bit quantized loading, optional LoRA adapters, and automatic device placement. Located in the main [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md), this utility supports `BitsAndBytesConfig` for reduced memory footprint.

```python
def load_model(model_name_or_path, load_in_4bit=False, adapter_name_or_path=None):
    # Quantization config for 4-bit mode

    quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
    ) if load_in_4bit else None
    
    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        load_in_4bit=load_in_4bit,
        trust_remote_code=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map='auto',
        quantization_config=quantization_config,
    )
    
    # Wrap with LoRA if adapter provided

    if adapter_name_or_path:
        model = PeftModel.from_pretrained(model, adapter_name_or_path)
    return model

```

### Chat Prompt Template

The system uses a native **Llama 3 token format** registered via `register_template()` in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md). This ensures compatibility with the model's special tokens including `<|begin_of_text|>`, `<|eot_id|>`, and `<|end_of_text|>`.

```python
register_template(
    template_name='llama3',
    system_format='<|begin_of_text|><<SYS>>\n{content}\n<</SYS>>\n\n',
    user_format='user\n\n{content}<|eot_id|>',
    assistant_format='assistant\n\n{content}<|end_of_text|>\n',
    system="You are a helpful, excellent and smart assistant. ...",
    stop_word='<|end_of_text|>'
)

```

### Inference Loop

The `main()` function in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md) demonstrates the generation workflow, maintaining rolling dialogue history and applying the registered template.

```python
while True:
    query = input('# User：')

    input_ids = build_prompt(tokenizer, template, query, copy.deepcopy(history), system=None).to(model.device)
    outputs = model.generate(
        input_ids,
        max_new_tokens=500,
        do_sample=True,
        top_p=0.9,
        temperature=0.6,
        repetition_penalty=1.1,
        eos_token_id=stop_token_id,
    )
    response = tokenizer.decode(outputs.tolist()[0][len(input_ids[0]):]).strip()
    history.append({"role": "user", "content": query})
    history.append({"role": "assistant", "content": response})

```

## Deploying Llama3 Chinese with vLLM

The primary production deployment method uses **vLLM** to serve an OpenAI-compatible API server. This approach provides high-throughput inference with continuous batching and paged attention.

### Launching the vLLM Server

Reference the instructions in [`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md) to start the server. Use `--dtype float16` for compatibility with the Chinese fine-tuned weights, and optionally extend context length with `--max-model-len`.

```bash
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model /path/to/llama3-chinese-instruct-dpo-8b \
    --dtype float16 \
    --max-model-len 32768 \
    --host 0.0.0.0 \
    --port 8000

```

### Calling the API Endpoint

Once running, the server exposes standard OpenAI endpoints. Send requests to `http://localhost:8000/v1/chat/completions` using the same JSON schema as OpenAI's API.

```bash
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "llama3-chinese",
      "messages": [{"role":"user","content":"请用中文解释量子纠缠"}],
      "max_tokens": 300,
      "temperature": 0.7
    }'

```

## Alternative Deployment Interfaces

Beyond vLLM, the repository offers lightweight alternatives for development and testing.

### Streamlit Web UI

For interactive debugging, use the Streamlit interface defined in [`deploy/web_streamlit_for_instruct_v2.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_instruct_v2.py). This variant supports the instruct-DPO model and automatically applies the system prompt.

```bash
pip install -U streamlit transformers==4.40.1
streamlit run deploy/web_streamlit_for_instruct_v2.py /path/to/llama3-chinese-instruct-dpo-8b --theme.base="dark"

```

### Python CLI Demo

The pure Python demo in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) demonstrates loading, prompting, and streaming without external server dependencies.

```bash
python deploy/python/chat_demo.py

```

### Flask HTTP Server

For integration testing, the minimal Flask-style server in [`deploy/API/easy_server_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/API/easy_server_demo.py) exposes REST endpoints at `http://127.0.0.1:5000`.

```bash
python deploy/API/easy_server_demo.py

```

Example client usage:

```python
import requests
resp = requests.post(
    "http://127.0.0.1:5000/chat",
    json={"prompt": "用中文写一段古诗"}
)
print(resp.json()["response"])

```

## Configuration and Optimization

### 4-Bit Quantization

Enable **4-bit quantization** by passing `load_in_4bit=True` to `load_model()`, or use the `--quantization awq` flag with vLLM. This reduces VRAM requirements to approximately 8 GB for 8B parameter models.

### LoRA Adapter Support

Deploy fine-tuned checkpoints without merging weights by specifying `adapter_name_or_path` in the loader. The `PeftModel` wrapper automatically applies adapter layers during inference.

### Context Length Extension

To support contexts beyond 8K tokens, modify `max_position_embeddings` in the model's [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) and adjust `rope_theta` as documented in the README section "上下文长度简单扩张法（32K、96K）". When using vLLM, pass `--max-model-len 32768` to allocate sufficient KV cache.

## Summary

- **vLLM deployment** provides the highest throughput via OpenAI-compatible APIs using the instructions in [`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md).
- **Model loading** in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md) supports FP16, 4-bit quantization, and dynamic LoRA adapter injection via `load_model()`.
- **Prompt formatting** requires the native Llama 3 template with special tokens registered through `register_template()`.
- **Alternative interfaces** include Streamlit ([`deploy/web_streamlit_for_instruct_v2.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_instruct_v2.py)), Flask ([`deploy/API/easy_server_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/API/easy_server_demo.py)), and pure Python ([`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)).
- **Hardware optimization** is achievable through 4-bit quantization and context length adjustments in the model configuration.

## Frequently Asked Questions

### How do I reduce VRAM usage when deploying Llama3 Chinese with vLLM?

Use 4-bit quantization by passing `load_in_4bit=True` to the `load_model()` function in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md), which implements `BitsAndBytesConfig` with NF4 quantization. For vLLM specifically, use quantized checkpoints (AWQ or GPTQ) and set `--max-model-len` to limit KV cache allocation, reducing VRAM requirements to approximately 8 GB for 8B models.

### What prompt format does the Llama3 Chinese model expect?

The model expects the native Llama 3 chat format registered via `register_template()` in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md), using special tokens including `<|begin_of_text|>` for system prompts, `<|eot_id|>` to terminate user turns, and `<|end_of_text|>` as the stop word for assistant completions. The template automatically wraps content with `user` and `assistant` role identifiers.

### Can I use custom LoRA adapters with the vLLM deployment?

When using the native Python pipeline, pass the adapter path to `load_model()` via `adapter_name_or_path`, which wraps the base model with `PeftModel`. For vLLM deployments, merge the LoRA weights into the base checkpoint before serving, or use vLLM's experimental LoRA support by specifying the adapter path in the API request parameters.

### How do I extend the context length beyond the default 8K tokens?

Modify `max_position_embeddings` in the model's [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) and adjust the `rope_theta` parameter according to the "上下文长度简单扩张法" section. When launching vLLM, add `--max-model-len 32768` (or 96000) to allocate sufficient KV cache memory for the extended context window.