# Using QLoRA for Memory-Efficient Llama 3 Chinese Training

> Train Llama 3 Chinese models efficiently with QLoRA. This technique drastically cuts GPU memory needs, enabling fine-tuning on consumer hardware with 4-bit quantization.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: tutorial
- Published: 2026-02-28

---

**QLoRA enables 4-bit quantization of Llama 3 base weights while training only low-rank adapters, reducing GPU memory requirements from over 10GB to approximately 4-6GB for 8B models and allowing fine-tuning on consumer hardware.**

The `crazyboym/llama3-chinese-chat` repository provides a complete implementation lifecycle for adapting Meta's Llama 3 to Chinese linguistic contexts through Quantized Low-Rank Adaptation (QLoRA). This approach preserves the base model's multilingual capabilities while minimizing computational costs through parameter-efficient fine-tuning techniques.

## Memory Efficiency and Training Costs

QLoRA addresses the primary bottleneck in large language model fine-tuning: GPU memory consumption. According to the training cost table in [`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md) (lines 536-537), quantizing Llama 3 to 4-bit precision reduces memory usage from roughly 10GB to 4-6GB for 8B parameter models, while 8-bit quantization maintains approximately 10GB requirements.

The methodology freezes the original Llama 3 weights in 4-bit (Normal Float 4) or 8-bit precision using **bits-and-bytes** quantization, then injects trainable low-rank matrices (typically rank 128 with alpha 256) into the attention layers. Only these adapter parameters update during training, enabling gradient computation on hardware with limited VRAM while maintaining model quality comparable to full fine-tuning.

## Repository Architecture for QLoRA Workflows

The repository organizes QLoRA functionality across several key locations:

- **[`train/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/train/README.md)** and subdirectories ([`train/CPT/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/train/CPT/README.md), [`train/SFT/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/train/SFT/README.md)): Document the `--bits 4` and `--bits 8` flags for quantized training configurations
- **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)**: Contains the `load_model()` function (lines 43-71) that handles PEFT adapter loading
- **[`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md)**: Documents the `--qlora-adapter-name-or-path` server flag (line 195)
- **[`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py)**: Implements graphical interface adapter loading (lines 34-58)
- **`tools/`**: Provides data conversion utilities for preparing Chinese datasets in ShareGPT/Firefly formats required by LoRA trainers

## Training QLoRA Adapters

To initiate QLoRA fine-tuning on Chinese dialogue data, configure your training YAML to use 4-bit quantization and execute via the XTuner framework:

```bash
xtuner train \
  llama3-8b.yaml \
  --bits 4 \
  --lora_r 128 \
  --lora_alpha 256 \
  --output_dir ./qlora_adapter

```

This process generates a QLoRA adapter directory containing [`adapter_config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/adapter_config.json), `adapter_model.bin`, and associated training states. The base Llama 3 weights remain unmodified and stored in quantized format, while only the low-rank adaptation matrices (typically <1% of total parameters) train on your Chinese corpus.

## Deploying QLoRA Adapters

The repository supports three deployment patterns for QLoRA-fine-tuned models, all utilizing the unified adapter-loading interface.

### Python Inference with PEFT

For custom inference pipelines, use the `load_model` function from [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py). This implementation automatically detects and loads QLoRA adapters via the `PeftModel` class:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

def load_model(model_name_or_path, load_in_4bit=False, adapter_name_or_path=None):
    quantization_config = (BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        llm_int8_threshold=6.0,
        llm_int8_has_fp16_weight=False,
    ) if load_in_4bit else None)

    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        load_in_4bit=load_in_4bit,
        trust_remote_code=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map="auto",
        quantization_config=quantization_config,
    )

    if adapter_name_or_path is not None:
        model = PeftModel.from_pretrained(model, adapter_name_or_path)

    return model

```

Pass your trained adapter path to the `adapter_name_or_path` parameter to merge the QLoRA weights with the quantized base model.

### vLLM Server Deployment

For production serving, launch the vLLM server with explicit QLoRA adapter support:

```bash
python -m vllm.entrypoint.api_server \
  --model shareAI/llama3-Chinese-chat-8b \
  --qlora-adapter-name-or-path ./qlora_adapter \
  --max-model-len 16384 \
  --port 8000

```

The `--qlora-adapter-name-or-path` flag, documented at line 195 of [`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md), instructs vLLM to load the PEFT adapter alongside the quantized base weights, enabling high-throughput inference with minimal memory overhead.

### Streamlit Interactive Interface

For demonstration purposes, the Streamlit implementation in [`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py) (lines 34-58) accepts adapter paths through the same `load_model` interface:

```python
model, tokenizer = load_model(
    model_name_or_path="shareAI/llama3-Chinese-chat-8b",
    adapter_name_or_path="./qlora_adapter",
    load_in_4bit=False
)

```

This allows researchers to interactively test Chinese conversational capabilities without writing additional inference code.

## End-to-End Workflow Example

Complete the following steps to fine-tune and serve a Chinese QLoRA adapter:

1. Convert raw Chinese dialogue data to ShareGPT format using [`tools/convert_raw_data_for_firefly.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/convert_raw_data_for_firefly.py)
2. Execute quantized training with `xtuner train --bits 4`
3. Deploy via vLLM: `python -m vllm.entrypoint.api_server --qlora-adapter-name-or-path ./qlora_adapter`
4. Query the model:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3-8b","messages":[{"role":"user","content":"请用古诗的形式介绍北京"}]}'

```

## Summary

- **QLoRA reduces memory requirements** from ~10GB to ~4-6GB for Llama 3 8B models through 4-bit quantization and adapter-based fine-tuning
- **The `crazyboym/llama3-chinese-chat` repository** provides unified adapter loading across Python scripts, vLLM servers, and Streamlit interfaces
- **Training utilizes standard tools** like XTuner with `--bits 4` flags to generate PEFT-compatible adapter directories
- **Deployment requires no code changes** between standard LoRA and QLoRA adapters; simply specify the adapter path via `adapter_name_or_path` or `--qlora-adapter-name-or-path`

## Frequently Asked Questions

### How much GPU memory is required for QLoRA training of Llama 3?

QLoRA training of Llama 3 8B models requires approximately 4-6GB of GPU memory when using 4-bit (NF4) quantization, or roughly 10GB when using 8-bit quantization, according to the cost tables in the repository README. This enables fine-tuning on consumer-grade GPUs like the RTX 3090 or RTX 4090 that would otherwise be insufficient for full 16-bit fine-tuning.

### What files are generated during QLoRA training?

QLoRA training produces a standard PEFT adapter directory containing [`adapter_config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/adapter_config.json) (specifying rank, alpha, and target modules), `adapter_model.bin` (the trained low-rank matrices), and optionally `adapter_model.safetensors`. The quantized base model weights remain separate and are loaded dynamically during inference via the `BitsAndBytesConfig` quantization parameters.

### Can QLoRA adapters be merged back into the base model?

Yes, QLoRA adapters can be merged into the base model using the `PeftModel.merge_and_unload()` method, creating a standalone model with fused weights. However, the `crazyboym/llama3-chinese-chat` deployment scripts typically keep adapters separate to maintain flexibility, allowing the same base model to serve multiple specialized Chinese adapters simultaneously through the `--qlora-adapter-name-or-path` mechanism.

### Does QLoRA impact Chinese language generation quality compared to full fine-tuning?

When properly configured with ranks between 64-256 and appropriate alpha values (typically 2x the rank), QLoRA achieves comparable performance to full fine-tuning on Chinese NLP benchmarks while preserving the base model's English capabilities. The repository's training guides recommend rank 128 with alpha 256 as a balanced starting point for Chinese conversational fine-tuning.