How LlamaFactory Supports Multiple LLM Architectures: Auto Detection and Unified Training

LlamaFactory supports multiple LLM architectures by inspecting model configuration files in src/llamafactory/model/loader.py to dynamically select the appropriate Hugging Face AutoModel class—whether for causal language modeling, vision-language, encoder-decoder, or audio-text generation—enabling a single unified training pipeline.

The open-source framework hiyouga/LlamaFactory eliminates manual architecture handling by routing any Transformers-compatible checkpoint through standardized loading and adapter logic. This approach to LlamaFactory multiple LLM architectures support allows researchers to fine-tune decoder-only models like LLaMA, multimodal vision-language models like LLaVA, and encoder-decoder models like T5 using identical Python APIs.

Dynamic Architecture Detection via AutoModel Mapping

At the heart of LlamaFactory's architecture flexibility lies the conditional class selection logic in src/llamafactory/model/loader.py (lines 66-74). The loader inspects the type of the model's configuration object against the internal _model_mapping dictionaries of various Hugging Face Auto classes:

• AutoModelForCausalLM for decoder-only transformer architectures (e.g., LLaMA, Falcon, Mistral).
• AutoModelForImageTextToText for vision-language models accepting image and text inputs (e.g., LLaVA, Qwen-VL).
• AutoModelForSeq2SeqLM for encoder-decoder architectures (e.g., T5, BART, Flan-T5).
• AutoModelForTextToWaveform for audio-text and speech models (e.g., Qwen-Audio, Whisper).

# src/llamafactory/model/loader.py (excerpt)

if type(config) in AutoModelForImageTextToText._model_mapping.keys():
    load_class = AutoModelForImageTextToText
elif type(config) in AutoModelForSeq2SeqLM._model_mapping.keys():
    load_class = AutoModelForSeq2SeqLM
elif type(config) in AutoModelForTextToWaveform._model_mapping.keys():
    load_class = AutoModelForTextToWaveform
else:
    load_class = AutoModelForCausalLM

This dynamic dispatch ensures that the correct model class is instantiated without user intervention, regardless of whether the checkpoint is a standard text-only LLM or a complex multimodal architecture.

Unified Config, Tokenizer, and Processor Loading

Before model instantiation, load_config and load_tokenizer in src/llamafactory/model/loader.py (lines 26-30) utilize the transformers Auto APIs to retrieve architecture-specific components:

config = AutoConfig.from_pretrained(model_args.model_name_or_path, **init_kwargs)
tokenizer = AutoTokenizer.from_pretrained(...)
processor = AutoProcessor.from_pretrained(...)  # optional, may be None

The optional processor object handles multimodal inputs (images or audio) when present, returning None for text-only models. After loading, the framework applies unified patches via src/llamafactory/model/patcher.py, which may inject value heads for RLHF or apply Liger kernels for optimized training, regardless of the underlying architecture.

Architecture-Agnostic Fine-Tuning with PEFT

All fine-tuning strategies are implemented in src/llamafactory/model/adapter.py, which operates on the generic PreTrainedModel interface. Because the adapter layer manipulates parameters by name and leverages the PEFT library's generic wrappers, it supports any architecture loaded by the framework:

• Full/Freeze fine-tuning: Directly manipulates requires_grad flags on selected parameters.
• LoRA/OFT/PiSSA: Creates LoraConfig or OFTConfig objects (lines 51-62) and wraps the model with get_peft_model, applying low-rank adapters to attention layers irrespective of model type.

# src/llamafactory/model/adapter.py (excerpt)

if finetuning_args.finetuning_type == "lora":
    peft_config = LoraConfig(task_type=TaskType.CAUSAL_LM, **peft_kwargs)
elif finetuning_args.finetuning_type == "oft":
    peft_config = OFTConfig(task_type=TaskType.CAUSAL_LM, **peft_kwargs)

model = get_peft_model(model, peft_config)

For quantized models, special handling in adapter.py ensures only LoRA and OFT adapters are attached, bypassing full-parameter updates that are incompatible with quantized weights.

Extensible Backend Plugins

LlamaFactory further extends multiple LLM architectures support through conditional backend plugins wired into the loading pipeline. When flags like use_kt (KTransformers) or use_unsloth are enabled in ModelArguments, the loader applies monkey-patches or swaps in optimized implementations without changing the core training logic. These plugins hook into the same Auto class detection system, allowing advanced inference engines or memory-efficient implementations to handle any supported architecture.

Practical Loading Examples

Decoder-Only LLM (LLaMA-3) for Full Fine-Tuning

from llamafactory.model.loader import load_tokenizer, load_model
from llamafactory.hparams import ModelArguments, FinetuningArguments

model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B-Instruct",
    trust_remote_code=False,
)
finetune_args = FinetuningArguments(finetuning_type="full", stage="sft")

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
)

The loader automatically selects AutoModelForCausalLM based on the LLaMA configuration.

Multimodal Vision-Language Model (LLaVA) for LoRA

model_args = ModelArguments(model_name_or_path="liuhaodong/llava-v1.5-13b")
finetune_args = FinetuningArguments(finetuning_type="lora", lora_target=["q_proj", "v_proj"])

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
)

Here, load_model detects the LLaVA architecture and instantiates AutoModelForImageTextToText, while adapter.py applies LoRA adapters uniformly.

Encoder-Decoder Model (FLAN-T5) with Value Head for RLHF

model_args = ModelArguments(model_name_or_path="google/flan-t5-xl")
finetune_args = FinetuningArguments(finetuning_type="lora", stage="rm")

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
    add_valuehead=True,  # Injects value head for reward modeling

)

The framework selects AutoModelForSeq2SeqLM for the T5 architecture and adds the value head via the patcher module.

Summary

• Automatic Architecture Detection: src/llamafactory/model/loader.py inspects config types to select the correct Hugging Face AutoModel class (Causal LM, Seq2Seq, Image-Text, or Audio-Text).
• Unified Component Loading: Auto APIs handle config, tokenizer, and processor retrieval uniformly, with optional multimodal support via AutoProcessor.
• Generic Adapter Layer: src/llamafactory/model/adapter.py applies LoRA, OFT, or full fine-tuning via PEFT's architecture-agnostic interfaces.
• Extensible Backends: Conditional plugins like KTransformers and Unsloth integrate seamlessly without breaking the unified loading API.
• Single API for All Models: Users interact with identical load_model and load_tokenizer functions regardless of whether training LLaMA, T5, or LLaVA.

Frequently Asked Questions

What types of LLM architectures does LlamaFactory support?

LlamaFactory supports decoder-only models (LLaMA, Falcon), encoder-decoder models (T5, BART), vision-language models (LLaVA, Qwen-VL), and audio-text models (Qwen-Audio, Whisper). The framework automatically detects the correct architecture by inspecting the model config against AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoModelForImageTextToText, and AutoModelForTextToWaveform mappings in src/llamafactory/model/loader.py.

How does LlamaFactory apply fine-tuning adapters to different architectures uniformly?

The adapter logic in src/llamafactory/model/adapter.py manipulates the generic PreTrainedModel interface using the PEFT library. Because LoRA, OFT, and freeze strategies operate on parameter names rather than architecture-specific structures, they wrap any model type—whether causal LM or Seq2Seq—using identical LoraConfig objects and get_peft_model calls.

Can LlamaFactory load quantized versions of supported architectures?

Yes, but with constraints. When loading quantized models (4-bit or 8-bit), the framework restricts fine-tuning to adapter-based methods (LoRA or OFT) only, as implemented in src/llamafactory/model/adapter.py. Full-parameter fine-tuning is disabled for quantized weights to prevent gradient computation errors.

Which source file controls the automatic selection of model classes?

The automatic class selection logic resides in src/llamafactory/model/loader.py, specifically lines 66-74, where the code checks if type(config) in AutoModelForImageTextToText._model_mapping.keys() and similar conditions to dispatch to the appropriate AutoModel class.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}