# How LlamaFactory Handles Model Loading and Configuration: A Deep Dive into src/llamafactory/model/loader.py

> Discover how LlamaFactory manages model loading and configuration. Explore the key functions in src/llamafactory/model/loader.py for seamless initialization and optimization.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: deep-dive
- Published: 2026-03-04

---

**LlamaFactory centralizes all model initialization logic in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py), exposing three core helpers—`load_config`, `load_tokenizer`, and `load_model`—that orchestrate everything from AutoConfig initialization to PEFT adapter injection and specialized backend optimization.**

Loading and configuring large language models for fine-tuning or inference requires orchestrating multiple components—configs, tokenizers, model weights, and adapters. LlamaFactory simplifies this complexity through a unified loading architecture in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) that supports everything from standard Hugging Face models to specialized backends like Unsloth and k-transformers.

## Core Model Loading API

The repository exposes three public functions in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py) that form the complete model initialization pipeline. All training scripts, the Gradio web UI, and benchmark utilities import these helpers from `llamafactory.model`.

### Loading Model Configurations with load_config

The `load_config` function prepares the Transformers configuration object used for downstream model class selection.

- Builds a dictionary of initialization arguments via `_get_init_kwargs(model_args)`, capturing `trust_remote_code`, `cache_dir`, `revision`, and `token`
- Calls `AutoConfig.from_pretrained` with the model path and prepared kwargs
- Returns a config object that determines which model auto-class to instantiate later

```python
init_kwargs = _get_init_kwargs(model_args)
config = AutoConfig.from_pretrained(
    model_args.model_name_or_path, 
    **init_kwargs
)

```

This logic appears in lines 26-30 of [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py).

### Tokenizer and Processor Initialization with load_tokenizer

The `load_tokenizer` function handles text tokenization and optional multimodal processor loading for vision-language models.

- Attempts `AutoTokenizer.from_pretrained` with the user-specified `use_fast` flag; automatically retries with the opposite flag on failure
- Applies LlamaFactory-specific patches via `patch_tokenizer(tokenizer, model_args)` to handle special tokens and chat templates
- Optionally loads `AutoProcessor`; if the returned object is not a subclass of `Processor`, it is discarded
- Patches the processor with the tokenizer via `patch_processor(processor, tokenizer, model_args)` when applicable

```python
tokenizer = AutoTokenizer.from_pretrained(
    model_args.model_name_or_path, 
    use_fast=model_args.use_fast_tokenizer,
    **init_kwargs
)
patch_tokenizer(tokenizer, model_args)

```

This implementation spans lines 72-124 in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py).

### Model Assembly with load_model

The `load_model` function is the heavyweight orchestrator spanning lines 132-247, handling backend-specific optimizations, weight loading, and adapter attachment.

**Preparation phase:**
- Calls `_get_init_kwargs`, `load_config`, and `patch_config` to add tokenizer-dependent fields and set trainable flags
- Optionally applies Liger kernels via `apply_liger_kernel` for memory-efficient attention

**Model class selection:**
Based on the configuration type, LlamaFactory selects the appropriate auto-class:

- `AutoModelForImageTextToText` for image-text models
- `AutoModelForSeq2SeqLM` for audio-text models  
- `AutoModelForTextToWaveform` for Qwen-Omni audio generation
- `AutoModelForCausalLM` for standard causal language models

**Specialized loading paths:**

- **k-transformers** (`use_kt=True`): Installs monkey-patches and loads via `load_kt_pretrained_model`
- **Unsloth** (`use_unsloth=True`): Defers loading with lazy initialization or uses `load_unsloth_pretrained_model`
- **Mixture-of-Depths** (`mixture_of_depths` active): Either loads a MoD-compatible checkpoint via `load_mod_pretrained_model` or converts a standard checkpoint using `convert_pretrained_model_to_mod`

**Weight loading strategy:**
- If `train_from_scratch` is enabled, instantiates the model using `from_config`
- Otherwise loads pretrained weights via `from_pretrained` with `torch_dtype="auto"`

**Adapter and head attachment:**
- Attaches LoRA or Prompt-tuning adapters via `init_adapter` (implemented in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py))
- Wraps with value-head for RLHF via `AutoModelForCausalLMWithValueHead` when `add_valuehead=True`, optionally loading saved value-head parameters via `load_valuehead_params`

**Finalization:**
- Applies `patch_model` and `register_autoclass` for serialization compatibility
- Optionally applies NPU-specific optimizations via `apply_default_kernels`
- Sets gradient requirements and prints parameter statistics

## Practical Implementation Examples

### Minimal Inference Script

The following demonstrates loading a model for read-only inference using the public API:

```python
from llamafactory.model import load_tokenizer, load_model
from llamafactory.hparams import ModelArguments, FinetuningArguments
import torch

# Configure arguments (typically from CLI or YAML)

model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    trust_remote_code=True,
    use_fast_tokenizer=True,
)

finetuning_args = FinetuningArguments(
    stage="sft",
    train_from_scratch=False,
)

# Load tokenizer and model

tokenizer = load_tokenizer(model_args)["tokenizer"]
model = load_model(
    tokenizer, 
    model_args, 
    finetuning_args, 
    is_trainable=False
)

# Run inference

inputs = tokenizer("Hello, LlamaFactory!", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
print(outputs.logits.shape)  # (1, seq_len, vocab_size)

```

### Loading with LoRA Adapters for Fine-Tuning

To resume training or inference with existing PEFT adapters:

```python
model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    adapter_name_or_path="path/to/lora_adapter",
    finetuning_type="lora",
    use_fast_tokenizer=True,
)

finetuning_args = FinetuningArguments(
    stage="sft",
    train_from_scratch=False,
)

tokenizer = load_tokenizer(model_args)["tokenizer"]
model = load_model(
    tokenizer,
    model_args,
    finetuning_args,
    is_trainable=True,  # Enable gradients for training

)

```

The `init_adapter` function in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py) automatically attaches the LoRA weights to the base model.

### Web UI Integration

When users click **"Load Model"** in the Gradio interface, [`src/llamafactory/webui/chatter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/chatter.py) (lines 101-141) translates UI fields into arguments and triggers the same loading pipeline:

```python

# Inside src/llamafactory/webui/chatter.py

args = build_ui_arguments()  # Converts UI fields to dict

self.engine = EngineFactory.create_engine(**args)  # Calls load_model internally

```

## Summary

- **Centralized architecture**: All loading logic resides in [`src/llamafactory/model/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/loader.py), consumed by training scripts and the web UI
- **Three-phase workflow**: Configuration loading (`load_config`) → Tokenizer initialization (`load_tokenizer`) → Model assembly (`load_model`)
- **Backend flexibility**: Automatic routing for Unsloth, k-transformers, Mixture-of-Depths, and Liger kernels
- **PEFT integration**: Native support for LoRA and Prompt-tuning via `init_adapter` in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py)
- **Multimodal support**: Handles vision-language and audio-text models through AutoProcessor detection
- **RLHF ready**: Built-in value-head wrapping for reward model training and PPO

## Frequently Asked Questions

### How does LlamaFactory handle tokenizer initialization failures?

The `load_tokenizer` function implements automatic fallback logic: if `AutoTokenizer.from_pretrained` fails with the user-specified `use_fast` flag, it catches the exception and retries with the opposite setting. After successful loading, it applies proprietary patches via `patch_tokenizer` to ensure compatibility with LlamaFactory's chat templating and special token handling.

### Can I load models with existing LoRA adapters?

Yes. Set `adapter_name_or_path` to the checkpoint directory and `finetuning_type="lora"` in your `ModelArguments`. The `load_model` function calls `init_adapter` (defined in [`src/llamafactory/model/adapter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model/adapter.py)) to attach the existing adapter weights to the base model before returning it.

### What optimized backends does LlamaFactory support for model loading?

LlamaFactory supports several optimization backends through the `load_model` function: **Unsloth** for lazy loading and memory-efficient finetuning, **k-transformers** for CPU/GNU hybrid inference via monkey-patching, **Liger kernels** for fused attention operations, and **Mixture-of-Depths** conversion for sparse model architectures. Each backend is activated via specific flags in `ModelArguments`.

### How does the web UI integrate with the model loading pipeline?

The Gradio interface in [`src/llamafactory/webui/chatter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/chatter.py) collects user inputs, converts them into `ModelArguments` and `FinetuningArguments` dataclasses, and passes them to the same `load_model` and `load_tokenizer` functions used by the CLI. This ensures consistency between interactive UI sessions and command-line training jobs.