# Loading and Using LoRA Adapters with PeftModel in Llama-3 Chinese Chat

> Learn to load and use LoRA adapters with PeftModel in Llama-3 Chinese Chat. Discover efficient 4-bit inference and unified model loading.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**The `crazyboym/llama3-chinese-chat` repository implements a unified `load_model` function that combines `AutoModelForCausalLM` base model loading with optional LoRA adapter injection via `PeftModel.from_pretrained`, supporting efficient 4-bit inference through `BitsAndBytesConfig`.**

The `crazyboym/llama3-chinese-chat` repository demonstrates a production-ready pattern for loading and using LoRA adapters with PeftModel, enabling developers to augment Meta's Llama-3 models with Chinese language capabilities without modifying base weights. This architecture separates model acquisition, quantization, and adapter injection into a single cached resource that powers multiple UI front-ends and CLI demos.

## Core Implementation in `load_model`

The repository's loading logic is centralized in the `load_model` function within [[`deploy/web_streamlit_for_v1.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py)](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py#L71-L95) (lines 71-95). This function handles both full-precision and quantized model initialization while conditionally applying PEFT adapters.

### Base Model Initialization with 4-Bit Quantization

When `load_in_4bit=True`, the function creates a `BitsAndBytesConfig` and passes it to `AutoModelForCausalLM.from_pretrained` alongside critical efficiency parameters:

- `trust_remote_code=True` – enables execution of model-specific code from the Hugging Face Hub
- `low_cpu_mem_usage=True` – reduces RAM overhead during loading
- `device_map='auto'` – automatically distributes layers across available GPUs

This configuration allows the base Llama-3 model to load in 4-bit precision, significantly reducing VRAM requirements while maintaining inference quality.

### Conditional Adapter Injection via PeftModel

If `adapter_name_or_path` is provided (not `None`), the function wraps the base model with `PeftModel.from_pretrained` at lines 94-95. This call loads the LoRA weights from `adapter_model.bin` and injects low-rank matrices into the transformer layers without altering the original base parameters. The implementation keeps adapter logic isolated from generation code, allowing the same downstream functions to work regardless of whether an adapter is active.

### Resource Caching with Streamlit

The `load_model` function is decorated with `@st.cache_resource` to prevent reloading the large language model on every UI interaction. This caching strategy persists both the base model and any injected LoRA adapters across Streamlit sessions, eliminating redundant GPU memory allocation.

## Tokenizer Consistency Strategy

The tokenizer is always instantiated from the **base** model directory using `AutoTokenizer.from_pretrained(model_name_or_path)`, regardless of whether a LoRA adapter is loaded. This design choice ensures that tokenization remains consistent between training (performed on the base model vocabulary) and inference, preventing token mismatches that could degrade Chinese language generation quality when adapters are applied.

## Unified API Across Deployment Interfaces

The `load_model` signature—`model_name_or_path, adapter_name_or_path=None, load_in_4bit=False`—is reused identically across every entry point in the repository:

- [[`deploy/web_streamlit_for_instruct_v2.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_instruct_v2.py)](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_instruct_v2.py#L71-L95) – instruct-tuned demo interface
- [[`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py)](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py#L34-L57) – general chat UI
- [[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py#L44-L70) – pure Python CLI demo

This standardization guarantees that LoRA-loading behavior remains identical whether users interact via Streamlit web interfaces or command-line scripts.

## Merging LoRA Checkpoints Before Loading

For advanced use cases requiring combined adaptations, the repository includes [[`tools/merge_weight.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/merge_weight.py)](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/merge_weight.py), which demonstrates weighted averaging of two separate `adapter_model.bin` checkpoints. This preprocessing step creates a merged adapter file that can be passed to `load_model` as a single `adapter_name_or_path`, effectively combining multiple fine-tunes before PEFT injection.

## Practical Implementation Examples

The following patterns demonstrate the three primary loading scenarios supported by the repository.

**Base model without adapter (full-precision):**

```python
model, tokenizer = load_model(
    model_name_or_path="meta-llama/Meta-Llama-3-8B-Instruct",
    adapter_name_or_path=None,
    load_in_4bit=False,
)

```

**Base model with LoRA adapter (4-bit quantized):**

```python
model, tokenizer = load_model(
    model_name_or_path="meta-llama/Meta-Llama-3-8B-Instruct",
    adapter_name_or_path="path/to/lora_adapter",
    load_in_4bit=True,  # Enables BitsAndBytes 4-bit mode

)

```

**Merged adapter checkpoint:**

```python

# First run tools/merge_weight.py to create adapter_model_merged.bin

model, tokenizer = load_model(
    model_name_or_path="meta-llama/Meta-Llama-3-8B-Instruct",
    adapter_name_or_path="./adapter_model_merged.bin",
)

```

## Summary

- The `load_model` function in [`deploy/web_streamlit_for_v1.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py) (lines 71-95) centralizes all model acquisition and adapter injection logic
- **PeftModel.from_pretrained** loads LoRA weights at lines 94-95 only when `adapter_name_or_path` is specified, leaving base parameters frozen
- **4-bit quantization** via `BitsAndBytesConfig` reduces memory footprint without requiring separate loading paths for adapted versus non-adapted models
- The tokenizer is always loaded from the base model path to ensure vocabulary consistency across adapter configurations
- The `@st.cache_resource` decorator eliminates redundant model reloading in Streamlit applications
- **[`tools/merge_weight.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/merge_weight.py)** supports preprocessing multiple LoRA checkpoints into a single adapter file before PEFT loading

## Frequently Asked Questions

### What is the difference between loading with and without an adapter?

When `adapter_name_or_path=None`, the function returns the base `AutoModelForCausalLM` model directly. When a path is provided, the function wraps the base model with `PeftModel.from_pretrained`, injecting trainable low-rank matrices that modify the forward pass without changing the original weights.

### Why is the tokenizer loaded from the base model rather than the adapter path?

LoRA adapters modify attention and feed-forward layer weights but do not alter the vocabulary or token embeddings. Loading `AutoTokenizer.from_pretrained(model_name_or_path)` ensures that tokenization matches the base model's training vocabulary, preventing out-of-vocabulary errors that would occur if the tokenizer were loaded from adapter-specific metadata.

### How does 4-bit quantization interact with LoRA adapter loading?

The `BitsAndBytesConfig` is applied during the initial `AutoModelForCausalLM.from_pretrained` call, quantizing the base weights before `PeftModel` wraps the model. The LoRA adapters remain in full precision while the base weights use 4-bit representation, enabling efficient inference with minimal VRAM overhead.

### Can multiple LoRA adapters be active simultaneously?

The standard `load_model` implementation accepts a single `adapter_name_or_path`. To combine multiple adaptations, use the [`tools/merge_weight.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/merge_weight.py) script to perform weighted averaging of separate `adapter_model.bin` files into a single checkpoint. This merged checkpoint can then be loaded as a unified adapter via PEFT.