Deploying Llama3 Chinese with vLLM: Complete Setup Guide for OpenAI-Compatible APIs

Deploy Llama3 Chinese models using vLLM to expose OpenAI-compatible /v1/chat/completions endpoints, or use the alternative Streamlit and Flask interfaces provided in the crazyboym/llama3-chinese-chat repository.

The crazyboym/llama3-chinese-chat repository provides a production-ready pipeline for fine-tuned Llama 3 models specialized for Chinese dialogue. Whether you need high-throughput API serving or interactive prototyping tools, the project supports vLLM deployment alongside Streamlit web UIs and lightweight Python servers. This guide covers the complete architecture—from model loading and prompt templating to launching scalable inference endpoints.

Model Architecture and Loading Pipeline

The repository implements a three-layer architecture defined in README.md that handles everything from quantization to dialogue management.

Model and Tokenizer Loader

The load_model() function handles FP16 or 4-bit quantized loading, optional LoRA adapters, and automatic device placement. Located in the main README.md, this utility supports BitsAndBytesConfig for reduced memory footprint.

def load_model(model_name_or_path, load_in_4bit=False, adapter_name_or_path=None):
    # Quantization config for 4-bit mode

    quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
    ) if load_in_4bit else None
    
    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        load_in_4bit=load_in_4bit,
        trust_remote_code=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map='auto',
        quantization_config=quantization_config,
    )
    
    # Wrap with LoRA if adapter provided

    if adapter_name_or_path:
        model = PeftModel.from_pretrained(model, adapter_name_or_path)
    return model

Chat Prompt Template

The system uses a native Llama 3 token format registered via register_template() in README.md. This ensures compatibility with the model's special tokens including <|begin_of_text|>, <|eot_id|>, and <|end_of_text|>.

register_template(
    template_name='llama3',
    system_format='<|begin_of_text|><<SYS>>\n{content}\n<</SYS>>\n\n',
    user_format='user\n\n{content}<|eot_id|>',
    assistant_format='assistant\n\n{content}<|end_of_text|>\n',
    system="You are a helpful, excellent and smart assistant. ...",
    stop_word='<|end_of_text|>'
)

Inference Loop

The main() function in README.md demonstrates the generation workflow, maintaining rolling dialogue history and applying the registered template.

while True:
    query = input('# User：')

    input_ids = build_prompt(tokenizer, template, query, copy.deepcopy(history), system=None).to(model.device)
    outputs = model.generate(
        input_ids,
        max_new_tokens=500,
        do_sample=True,
        top_p=0.9,
        temperature=0.6,
        repetition_penalty=1.1,
        eos_token_id=stop_token_id,
    )
    response = tokenizer.decode(outputs.tolist()[0][len(input_ids[0]):]).strip()
    history.append({"role": "user", "content": query})
    history.append({"role": "assistant", "content": response})

Deploying Llama3 Chinese with vLLM

The primary production deployment method uses vLLM to serve an OpenAI-compatible API server. This approach provides high-throughput inference with continuous batching and paged attention.

Launching the vLLM Server

Reference the instructions in deploy/vLLM/README.md to start the server. Use --dtype float16 for compatibility with the Chinese fine-tuned weights, and optionally extend context length with --max-model-len.

pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model /path/to/llama3-chinese-instruct-dpo-8b \
    --dtype float16 \
    --max-model-len 32768 \
    --host 0.0.0.0 \
    --port 8000

Calling the API Endpoint

Once running, the server exposes standard OpenAI endpoints. Send requests to http://localhost:8000/v1/chat/completions using the same JSON schema as OpenAI's API.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "llama3-chinese",
      "messages": [{"role":"user","content":"请用中文解释量子纠缠"}],
      "max_tokens": 300,
      "temperature": 0.7
    }'

Alternative Deployment Interfaces

Beyond vLLM, the repository offers lightweight alternatives for development and testing.

Streamlit Web UI

For interactive debugging, use the Streamlit interface defined in deploy/web_streamlit_for_instruct_v2.py. This variant supports the instruct-DPO model and automatically applies the system prompt.

pip install -U streamlit transformers==4.40.1
streamlit run deploy/web_streamlit_for_instruct_v2.py /path/to/llama3-chinese-instruct-dpo-8b --theme.base="dark"

Python CLI Demo

The pure Python demo in deploy/python/chat_demo.py demonstrates loading, prompting, and streaming without external server dependencies.

python deploy/python/chat_demo.py

Flask HTTP Server

For integration testing, the minimal Flask-style server in deploy/API/easy_server_demo.py exposes REST endpoints at http://127.0.0.1:5000.

python deploy/API/easy_server_demo.py

Example client usage:

import requests
resp = requests.post(
    "http://127.0.0.1:5000/chat",
    json={"prompt": "用中文写一段古诗"}
)
print(resp.json()["response"])

Configuration and Optimization

4-Bit Quantization

Enable 4-bit quantization by passing load_in_4bit=True to load_model(), or use the --quantization awq flag with vLLM. This reduces VRAM requirements to approximately 8 GB for 8B parameter models.

LoRA Adapter Support

Deploy fine-tuned checkpoints without merging weights by specifying adapter_name_or_path in the loader. The PeftModel wrapper automatically applies adapter layers during inference.

Context Length Extension

To support contexts beyond 8K tokens, modify max_position_embeddings in the model's config.json and adjust rope_theta as documented in the README section "上下文长度简单扩张法（32K、96K）". When using vLLM, pass --max-model-len 32768 to allocate sufficient KV cache.

Summary

• vLLM deployment provides the highest throughput via OpenAI-compatible APIs using the instructions in deploy/vLLM/README.md.
• Model loading in README.md supports FP16, 4-bit quantization, and dynamic LoRA adapter injection via load_model().
• Prompt formatting requires the native Llama 3 template with special tokens registered through register_template().
• Alternative interfaces include Streamlit (deploy/web_streamlit_for_instruct_v2.py), Flask (deploy/API/easy_server_demo.py), and pure Python (deploy/python/chat_demo.py).
• Hardware optimization is achievable through 4-bit quantization and context length adjustments in the model configuration.

Frequently Asked Questions

How do I reduce VRAM usage when deploying Llama3 Chinese with vLLM?

Use 4-bit quantization by passing load_in_4bit=True to the load_model() function in README.md, which implements BitsAndBytesConfig with NF4 quantization. For vLLM specifically, use quantized checkpoints (AWQ or GPTQ) and set --max-model-len to limit KV cache allocation, reducing VRAM requirements to approximately 8 GB for 8B models.

What prompt format does the Llama3 Chinese model expect?

The model expects the native Llama 3 chat format registered via register_template() in README.md, using special tokens including <|begin_of_text|> for system prompts, <|eot_id|> to terminate user turns, and <|end_of_text|> as the stop word for assistant completions. The template automatically wraps content with user and assistant role identifiers.

Can I use custom LoRA adapters with the vLLM deployment?

When using the native Python pipeline, pass the adapter path to load_model() via adapter_name_or_path, which wraps the base model with PeftModel. For vLLM deployments, merge the LoRA weights into the base checkpoint before serving, or use vLLM's experimental LoRA support by specifying the adapter path in the API request parameters.

How do I extend the context length beyond the default 8K tokens?

Modify max_position_embeddings in the model's config.json and adjust the rope_theta parameter according to the "上下文长度简单扩张法" section. When launching vLLM, add --max-model-len 32768 (or 96000) to allocate sufficient KV cache memory for the extended context window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}