How LlamaFactory Handles Model Loading and Configuration: A Deep Dive into src/llamafactory/model/loader.py

LlamaFactory centralizes all model initialization logic in src/llamafactory/model/loader.py, exposing three core helpers—load_config, load_tokenizer, and load_model—that orchestrate everything from AutoConfig initialization to PEFT adapter injection and specialized backend optimization.

Loading and configuring large language models for fine-tuning or inference requires orchestrating multiple components—configs, tokenizers, model weights, and adapters. LlamaFactory simplifies this complexity through a unified loading architecture in src/llamafactory/model/loader.py that supports everything from standard Hugging Face models to specialized backends like Unsloth and k-transformers.

Core Model Loading API

The repository exposes three public functions in src/llamafactory/model/loader.py that form the complete model initialization pipeline. All training scripts, the Gradio web UI, and benchmark utilities import these helpers from llamafactory.model.

Loading Model Configurations with load_config

The load_config function prepares the Transformers configuration object used for downstream model class selection.

• Builds a dictionary of initialization arguments via _get_init_kwargs(model_args), capturing trust_remote_code, cache_dir, revision, and token
• Calls AutoConfig.from_pretrained with the model path and prepared kwargs
• Returns a config object that determines which model auto-class to instantiate later

init_kwargs = _get_init_kwargs(model_args)
config = AutoConfig.from_pretrained(
    model_args.model_name_or_path, 
    **init_kwargs
)

This logic appears in lines 26-30 of src/llamafactory/model/loader.py.

Tokenizer and Processor Initialization with load_tokenizer

The load_tokenizer function handles text tokenization and optional multimodal processor loading for vision-language models.

• Attempts AutoTokenizer.from_pretrained with the user-specified use_fast flag; automatically retries with the opposite flag on failure
• Applies LlamaFactory-specific patches via patch_tokenizer(tokenizer, model_args) to handle special tokens and chat templates
• Optionally loads AutoProcessor; if the returned object is not a subclass of Processor, it is discarded
• Patches the processor with the tokenizer via patch_processor(processor, tokenizer, model_args) when applicable

tokenizer = AutoTokenizer.from_pretrained(
    model_args.model_name_or_path, 
    use_fast=model_args.use_fast_tokenizer,
    **init_kwargs
)
patch_tokenizer(tokenizer, model_args)

This implementation spans lines 72-124 in src/llamafactory/model/loader.py.

Model Assembly with load_model

The load_model function is the heavyweight orchestrator spanning lines 132-247, handling backend-specific optimizations, weight loading, and adapter attachment.

Preparation phase:

• Calls _get_init_kwargs, load_config, and patch_config to add tokenizer-dependent fields and set trainable flags
• Optionally applies Liger kernels via apply_liger_kernel for memory-efficient attention

Model class selection: Based on the configuration type, LlamaFactory selects the appropriate auto-class:

• AutoModelForImageTextToText for image-text models
• AutoModelForSeq2SeqLM for audio-text models
• AutoModelForTextToWaveform for Qwen-Omni audio generation
• AutoModelForCausalLM for standard causal language models

Specialized loading paths:

• k-transformers (use_kt=True): Installs monkey-patches and loads via load_kt_pretrained_model
• Unsloth (use_unsloth=True): Defers loading with lazy initialization or uses load_unsloth_pretrained_model
• Mixture-of-Depths (mixture_of_depths active): Either loads a MoD-compatible checkpoint via load_mod_pretrained_model or converts a standard checkpoint using convert_pretrained_model_to_mod

Weight loading strategy:

• If train_from_scratch is enabled, instantiates the model using from_config
• Otherwise loads pretrained weights via from_pretrained with torch_dtype="auto"

Adapter and head attachment:

• Attaches LoRA or Prompt-tuning adapters via init_adapter (implemented in src/llamafactory/model/adapter.py)
• Wraps with value-head for RLHF via AutoModelForCausalLMWithValueHead when add_valuehead=True, optionally loading saved value-head parameters via load_valuehead_params

Finalization:

• Applies patch_model and register_autoclass for serialization compatibility
• Optionally applies NPU-specific optimizations via apply_default_kernels
• Sets gradient requirements and prints parameter statistics

Practical Implementation Examples

Minimal Inference Script

The following demonstrates loading a model for read-only inference using the public API:

from llamafactory.model import load_tokenizer, load_model
from llamafactory.hparams import ModelArguments, FinetuningArguments
import torch

# Configure arguments (typically from CLI or YAML)

model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    trust_remote_code=True,
    use_fast_tokenizer=True,
)

finetuning_args = FinetuningArguments(
    stage="sft",
    train_from_scratch=False,
)

# Load tokenizer and model

tokenizer = load_tokenizer(model_args)["tokenizer"]
model = load_model(
    tokenizer, 
    model_args, 
    finetuning_args, 
    is_trainable=False
)

# Run inference

inputs = tokenizer("Hello, LlamaFactory!", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
print(outputs.logits.shape)  # (1, seq_len, vocab_size)

Loading with LoRA Adapters for Fine-Tuning

To resume training or inference with existing PEFT adapters:

model_args = ModelArguments(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    adapter_name_or_path="path/to/lora_adapter",
    finetuning_type="lora",
    use_fast_tokenizer=True,
)

finetuning_args = FinetuningArguments(
    stage="sft",
    train_from_scratch=False,
)

tokenizer = load_tokenizer(model_args)["tokenizer"]
model = load_model(
    tokenizer,
    model_args,
    finetuning_args,
    is_trainable=True,  # Enable gradients for training

)

The init_adapter function in src/llamafactory/model/adapter.py automatically attaches the LoRA weights to the base model.

Web UI Integration

When users click "Load Model" in the Gradio interface, src/llamafactory/webui/chatter.py (lines 101-141) translates UI fields into arguments and triggers the same loading pipeline:

# Inside src/llamafactory/webui/chatter.py

args = build_ui_arguments()  # Converts UI fields to dict

self.engine = EngineFactory.create_engine(**args)  # Calls load_model internally

Summary

• Centralized architecture: All loading logic resides in src/llamafactory/model/loader.py, consumed by training scripts and the web UI
• Three-phase workflow: Configuration loading (load_config) → Tokenizer initialization (load_tokenizer) → Model assembly (load_model)
• Backend flexibility: Automatic routing for Unsloth, k-transformers, Mixture-of-Depths, and Liger kernels
• PEFT integration: Native support for LoRA and Prompt-tuning via init_adapter in src/llamafactory/model/adapter.py
• Multimodal support: Handles vision-language and audio-text models through AutoProcessor detection
• RLHF ready: Built-in value-head wrapping for reward model training and PPO

Frequently Asked Questions

How does LlamaFactory handle tokenizer initialization failures?

The load_tokenizer function implements automatic fallback logic: if AutoTokenizer.from_pretrained fails with the user-specified use_fast flag, it catches the exception and retries with the opposite setting. After successful loading, it applies proprietary patches via patch_tokenizer to ensure compatibility with LlamaFactory's chat templating and special token handling.

Can I load models with existing LoRA adapters?

Yes. Set adapter_name_or_path to the checkpoint directory and finetuning_type="lora" in your ModelArguments. The load_model function calls init_adapter (defined in src/llamafactory/model/adapter.py) to attach the existing adapter weights to the base model before returning it.

What optimized backends does LlamaFactory support for model loading?

LlamaFactory supports several optimization backends through the load_model function: Unsloth for lazy loading and memory-efficient finetuning, k-transformers for CPU/GNU hybrid inference via monkey-patching, Liger kernels for fused attention operations, and Mixture-of-Depths conversion for sparse model architectures. Each backend is activated via specific flags in ModelArguments.

How does the web UI integrate with the model loading pipeline?

The Gradio interface in src/llamafactory/webui/chatter.py collects user inputs, converts them into ModelArguments and FinetuningArguments dataclasses, and passes them to the same load_model and load_tokenizer functions used by the CLI. This ensures consistency between interactive UI sessions and command-line training jobs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}