How to Merge Llama3 Adapter Weights: Complete Guide to Blending PEFT Adapters

Merge Llama3 adapter weights by using the tools/merge_weight.py script from the crazyboym/llama3-chinese-chat repository, which performs linear interpolation between two adapter_model.bin checkpoints using a configurable alpha weight (default 0.7) to produce a single blended adapter.

When fine-tuning Llama 3 with PEFT methods like LoRA, each training stage generates an adapter_model.bin file containing only the delta weights. The crazyboym/llama3-chinese-chat repository provides a lightweight utility to combine multiple checkpoints without retraining, enabling multi-domain or progressive fine-tuning workflows.

How Adapter Merging Works in Llama3

The merge utility implements the same principle used by the RWKV "Merge LoRA" technique: linear interpolation of matching tensor keys. Because adapter weights are low-rank matrices that add small, task-specific biases to the base model, a convex combination of two adapters produces a model that behaves like a blend of both tasks.

The Weighted Sum Algorithm

Located in tools/merge_weight.py, the script executes the following operations:

1. Loads two adapter checkpoints (w_a and w_b) in standard PEFT format
2. Iterates over every tensor key in the first checkpoint
3. Computes a weighted sum for matching keys:

w_c[k] = w_a[k] * α + w_b[k] * (1 - α)  # α defaults to 0.7

If a key exists only in the first checkpoint, it is preserved unchanged. The resulting state dictionary is saved as adapter_model_merged.bin, compatible with standard PEFT loading routines.

Step-by-Step Guide to Merge Llama3 Adapter Weights

Prerequisites

Clone the repository and ensure you have two adapter checkpoints ready:

git clone https://github.com/crazyboym/llama3-chinese-chat.git
cd llama3-chinese-chat

# Verify your checkpoints exist (example paths)

ls ./checkpoint-700/adapter_model.bin
ls ./checkpoint-800/adapter_model.bin

Running the Merge Script

Execute the utility from the repository root:

python tools/merge_weight.py

By default, this creates ./adapter_model_merged.bin using a 70/30 weighting favoring the first checkpoint.

Adjusting the Interpolation Weight (α)

To customize the blending ratio, edit line 10 in tools/merge_weight.py:

# Default: 70% from checkpoint A, 30% from checkpoint B

w_c[k] = w_a[k] * 0.7 + w_b[k] * 0.3

# Example: 60/40 split for balanced domains

w_c[k] = w_a[k] * 0.6 + w_b[k] * 0.4

The alpha value determines how much influence each adapter retains in the final model.

Loading the Merged Adapter for Inference

After generating adapter_model_merged.bin, load it using the standard PEFT workflow demonstrated in the repository's README.md and deployment scripts like deploy/web_streamlit_for_v1.py:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_path = "shareAI/llama3-Chinese-chat-8b"
adapter_dir = "./merged_adapter"  # Directory containing adapter_model_merged.bin

# Load base model

model = AutoModelForCausalLM.from_pretrained(
    base_path,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Attach merged adapter

model = PeftModel.from_pretrained(model, adapter_dir)
tokenizer = AutoTokenizer.from_pretrained(base_path, trust_remote_code=True)

Note: PEFT expects a directory path rather than a single file. Place adapter_model_merged.bin inside a folder (e.g., merged_adapter/) before loading.

Choosing the Right Weighting Strategy

Select your alpha value based on the relationship between your checkpoints:

• Progressive refinement (later checkpoints improve earlier ones): Use α = 0.7 (default) to retain stability from the earlier checkpoint while incorporating refinements.
• Domain adaptation (small domain data after large pre-training): Use α = 0.4 to give more weight to the domain-specific adapter.
• Equal importance (multilingual vs. domain expertise): Use α = 0.5 for balanced representation of both tasks.

Summary

• File location: The merge utility is implemented in tools/merge_weight.py within the crazyboym/llama3-chinese-chat repository.
• Core mechanism: Linear interpolation combines matching tensor keys from two adapter_model.bin files using the formula w_c = w_a * α + w_b * (1 - α).
• Default behavior: Alpha is set to 0.7, favoring the first checkpoint.
• Output: Produces adapter_model_merged.bin, loadable via PeftModel.from_pretrained.
• Use cases: Enable multi-domain training, progressive fine-tuning, or task blending without retraining from scratch.

Frequently Asked Questions

What is the mathematical principle behind merging Llama3 adapter weights?

Adapter weights are low-rank additive matrices rather than multiplicative transformations. Because they represent small task-specific deltas to the base model, linear interpolation (convex combination) creates a valid adapter that blends the behavioral characteristics of both source checkpoints. This is implemented in tools/merge_weight.py as a weighted sum of matching tensor keys.

Where is the merge utility located in the llama3-chinese-chat repository?

The merge script is located at tools/merge_weight.py in the repository root. This utility is referenced by the project's documentation and is designed to process standard PEFT adapter_model.bin files produced by Hugging Face transformers training loops.

Can I merge more than two adapter checkpoints?

The current tools/merge_weight.py script handles exactly two checkpoints. To merge three or more adapters, run the script iteratively—merge the first two, then merge the result with the third checkpoint, adjusting the alpha value at each stage to control the relative influence of each training stage.

How do I load the merged adapter_model_merged.bin file?

PEFT requires a directory path for PeftModel.from_pretrained(). Create a folder (e.g., merged_adapter/), place adapter_model_merged.bin inside it, then pass the folder path to PeftModel.from_pretrained(model, "./merged_adapter"). This pattern is demonstrated in the repository's Streamlit deployment files including deploy/web_streamlit_for_v1.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}