# How LlamaFactory Supports Multiple LLM Architectures: Auto Detection and Unified Training

> Discover how LlamaFactory auto detects and unifies training for diverse LLM architectures like causal LM, vision-language, and more via Hugging Face AutoModel in a single pipeline.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: deep-dive
- Published: 2026-03-04

---

**LlamaFactory supports multiple LLM architectures by inspecting model configuration files in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) to dynamically select the appropriate Hugging Face `AutoModel` class—whether for causal language modeling, vision-language, encoder-decoder, or audio-text generation—enabling a single unified training pipeline.**

The open-source framework **hiyouga/LlamaFactory** eliminates manual architecture handling by routing any Transformers-compatible checkpoint through standardized loading and adapter logic. This approach to **LlamaFactory multiple LLM architectures** support allows researchers to fine-tune decoder-only models like LLaMA, multimodal vision-language models like LLaVA, and encoder-decoder models like T5 using identical Python APIs.

## Dynamic Architecture Detection via AutoModel Mapping

At the heart of LlamaFactory's architecture flexibility lies the conditional class selection logic in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) (lines 66-74). The loader inspects the `type` of the model's configuration object against the internal `_model_mapping` dictionaries of various Hugging Face Auto classes:

- **`AutoModelForCausalLM`** for decoder-only transformer architectures (e.g., LLaMA, Falcon, Mistral).
- **`AutoModelForImageTextToText`** for vision-language models accepting image and text inputs (e.g., LLaVA, Qwen-VL).
- **`AutoModelForSeq2SeqLM`** for encoder-decoder architectures (e.g., T5, BART, Flan-T5).
- **`AutoModelForTextToWaveform`** for audio-text and speech models (e.g., Qwen-Audio, Whisper).

```python

# src/llamafactory/model/loader.py (excerpt)

if type(config) in AutoModelForImageTextToText._model_mapping.keys():
    load_class = AutoModelForImageTextToText
elif type(config) in AutoModelForSeq2SeqLM._model_mapping.keys():
    load_class = AutoModelForSeq2SeqLM
elif type(config) in AutoModelForTextToWaveform._model_mapping.keys():
    load_class = AutoModelForTextToWaveform
else:
    load_class = AutoModelForCausalLM

```

This dynamic dispatch ensures that the correct model class is instantiated without user intervention, regardless of whether the checkpoint is a standard text-only LLM or a complex multimodal architecture.

## Unified Config, Tokenizer, and Processor Loading

Before model instantiation, `load_config` and `load_tokenizer` in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) (lines 26-30) utilize the `transformers` **Auto** APIs to retrieve architecture-specific components:

```python
config = AutoConfig.from_pretrained(model_args.model_name_or_path, **init_kwargs)
tokenizer = AutoTokenizer.from_pretrained(...)
processor = AutoProcessor.from_pretrained(...)  # optional, may be None

```

The optional `processor` object handles multimodal inputs (images or audio) when present, returning `None` for text-only models. After loading, the framework applies unified patches via [`src/llamafactory/model/patcher.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/patcher.py), which may inject value heads for RLHF or apply Liger kernels for optimized training, regardless of the underlying architecture.

## Architecture-Agnostic Fine-Tuning with PEFT

All fine-tuning strategies are implemented in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py), which operates on the generic `PreTrainedModel` interface. Because the adapter layer manipulates parameters by name and leverages the PEFT library's generic wrappers, it supports any architecture loaded by the framework:

- **Full/Freeze fine-tuning**: Directly manipulates `requires_grad` flags on selected parameters.
- **LoRA/OFT/PiSSA**: Creates `LoraConfig` or `OFTConfig` objects (lines 51-62) and wraps the model with `get_peft_model`, applying low-rank adapters to attention layers irrespective of model type.

```python

# src/llamafactory/model/adapter.py (excerpt)

if finetuning_args.finetuning_type == "lora":
    peft_config = LoraConfig(task_type=TaskType.CAUSAL_LM, **peft_kwargs)
elif finetuning_args.finetuning_type == "oft":
    peft_config = OFTConfig(task_type=TaskType.CAUSAL_LM, **peft_kwargs)

model = get_peft_model(model, peft_config)

```

For quantized models, special handling in [`adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/adapter.py) ensures only LoRA and OFT adapters are attached, bypassing full-parameter updates that are incompatible with quantized weights.

## Extensible Backend Plugins

LlamaFactory further extends **multiple LLM architectures** support through conditional backend plugins wired into the loading pipeline. When flags like `use_kt` (KTransformers) or `use_unsloth` are enabled in `ModelArguments`, the loader applies monkey-patches or swaps in optimized implementations without changing the core training logic. These plugins hook into the same Auto class detection system, allowing advanced inference engines or memory-efficient implementations to handle any supported architecture.

## Practical Loading Examples

### Decoder-Only LLM (LLaMA-3) for Full Fine-Tuning

```python
from llamafactory.model.loader import load_tokenizer, load_model
from llamafactory.hparams import ModelArguments, FinetuningArguments

model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B-Instruct",
    trust_remote_code=False,
)
finetune_args = FinetuningArguments(finetuning_type="full", stage="sft")

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
)

```

The loader automatically selects `AutoModelForCausalLM` based on the LLaMA configuration.

### Multimodal Vision-Language Model (LLaVA) for LoRA

```python
model_args = ModelArguments(model_name_or_path="liuhaodong/llava-v1.5-13b")
finetune_args = FinetuningArguments(finetuning_type="lora", lora_target=["q_proj", "v_proj"])

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
)

```

Here, `load_model` detects the LLaVA architecture and instantiates `AutoModelForImageTextToText`, while [`adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/adapter.py) applies LoRA adapters uniformly.

### Encoder-Decoder Model (FLAN-T5) with Value Head for RLHF

```python
model_args = ModelArguments(model_name_or_path="google/flan-t5-xl")
finetune_args = FinetuningArguments(finetuning_type="lora", stage="rm")

tokenizer_mod = load_tokenizer(model_args)
model = load_model(
    tokenizer=tokenizer_mod["tokenizer"],
    model_args=model_args,
    finetuning_args=finetune_args,
    is_trainable=True,
    add_valuehead=True,  # Injects value head for reward modeling

)

```

The framework selects `AutoModelForSeq2SeqLM` for the T5 architecture and adds the value head via the patcher module.

## Summary

- **Automatic Architecture Detection**: [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) inspects config types to select the correct Hugging Face AutoModel class (Causal LM, Seq2Seq, Image-Text, or Audio-Text).
- **Unified Component Loading**: Auto APIs handle config, tokenizer, and processor retrieval uniformly, with optional multimodal support via `AutoProcessor`.
- **Generic Adapter Layer**: [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py) applies LoRA, OFT, or full fine-tuning via PEFT's architecture-agnostic interfaces.
- **Extensible Backends**: Conditional plugins like KTransformers and Unsloth integrate seamlessly without breaking the unified loading API.
- **Single API for All Models**: Users interact with identical `load_model` and `load_tokenizer` functions regardless of whether training LLaMA, T5, or LLaVA.

## Frequently Asked Questions

### What types of LLM architectures does LlamaFactory support?

LlamaFactory supports decoder-only models (LLaMA, Falcon), encoder-decoder models (T5, BART), vision-language models (LLaVA, Qwen-VL), and audio-text models (Qwen-Audio, Whisper). The framework automatically detects the correct architecture by inspecting the model config against `AutoModelForCausalLM`, `AutoModelForSeq2SeqLM`, `AutoModelForImageTextToText`, and `AutoModelForTextToWaveform` mappings in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py).

### How does LlamaFactory apply fine-tuning adapters to different architectures uniformly?

The adapter logic in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py) manipulates the generic `PreTrainedModel` interface using the PEFT library. Because LoRA, OFT, and freeze strategies operate on parameter names rather than architecture-specific structures, they wrap any model type—whether causal LM or Seq2Seq—using identical `LoraConfig` objects and `get_peft_model` calls.

### Can LlamaFactory load quantized versions of supported architectures?

Yes, but with constraints. When loading quantized models (4-bit or 8-bit), the framework restricts fine-tuning to adapter-based methods (LoRA or OFT) only, as implemented in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py). Full-parameter fine-tuning is disabled for quantized weights to prevent gradient computation errors.

### Which source file controls the automatic selection of model classes?

The automatic class selection logic resides in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py), specifically lines 66-74, where the code checks `if type(config) in AutoModelForImageTextToText._model_mapping.keys()` and similar conditions to dispatch to the appropriate AutoModel class.