# LlamaFactory Inference Backends: The Complete Guide to Engine Selection

> Explore LlamaFactory inference backends including HuggingFace Transformers vLLM SGLang and K-Transformers. Optimize your LLM deployment by selecting the best engine.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory supports four inference backends—HuggingFace Transformers, vLLM, SGLang, and K-Transformers—selected via the `infer_backend` parameter that maps to the `EngineName` enumeration.**

LlamaFactory provides flexible model serving through multiple inference engines optimized for different deployment scenarios. Whether you need the compatibility of standard Transformers or the throughput of vLLM, the framework abstracts backend complexity behind a unified `ChatModel` interface. This guide examines the four supported LlamaFactory inference backends and their implementation across the codebase.

## The Four LlamaFactory Inference Backends

The framework defines supported engines in the `EngineName` enumeration located in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py)【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/extras/constants.py#L22-L27】. Each backend serves distinct operational requirements:

| Backend | EngineName Value | Description | Requirements |
|---------|------------------|-------------|--------------|
| **HuggingFace** | `EngineName.HF` (`"huggingface"`) | Standard 🤗 Transformers pipeline. Default option compatible with any model loadable into RAM/VRAM. | `transformers` |
| **vLLM** | `EngineName.VLLM` (`"vllm"`) | High‑performance serving engine with tensor parallelism and flash attention. Optimized for throughput. | `pip install vllm` |
| **SGLang** | `EngineName.SGLANG` (`"sglang"`) | Efficient token‑streaming backend with tool‑call support and structured generation. | `pip install sglang[all]` |
| **K‑Transformers** | `EngineName.KT` (`"ktransformers"`) | Inference engine for extremely large models using heterogeneous computing (CPU/GPU offloading). | `pip install ktransformers` |

## How Backend Selection Works in LlamaFactory

The architecture routes backend selection through three key components:

1. **Constant Definition**: [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) defines the `EngineName` enum with values `"huggingface"`, `"vllm"`, `"sglang"`, and `"ktransformers"`【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/extras/constants.py#L22-L27】.

2. **Argument Parsing**: [`src/llamafactory/hparams/model_args.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/model_args.py) declares `infer_backend: EngineName = EngineName.HF` as a dataclass field, defaulting to HuggingFace【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/hparams/model_args.py#L164-L166】.

3. **Engine Instantiation**: [`src/llamafactory/chat/chat_model.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/chat/chat_model.py) implements the factory logic in `ChatModel.__init__`. Lines 50‑78 inspect `model_args.infer_backend` and import the corresponding engine class (`HuggingfaceEngine`, `VllmEngine`, `SGLangEngine`, or `KTransformersEngine`)【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/chat/chat_model.py#L50-L78】.

## Using Different Backends in Practice

### Programmatic API Usage

You can specify the backend when constructing a `ChatModel` instance:

```python
from llamafactory.chat.chat_model import ChatModel

# Configuration dict mapping to ModelArguments

args = {
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B",
    "infer_backend": "vllm",  # Options: "huggingface", "vllm", "sglang", "ktransformers"

    "infer_dtype": "auto",
}

# Engine instantiation occurs here based on infer_backend value

chat = ChatModel(args)
response = chat.chat([{"role": "user", "content": "Explain quantum computing"}])
print(response[0]["content"])

```

The `ChatModel` class dynamically imports the engine implementation based on the `infer_backend` string value【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/chat/chat_model.py#L50-L78】.

### Command-Line Interface

Pass the `--infer_backend` flag to `llamafactory-cli`:

```bash

# HuggingFace (default)

llamafactory-cli infer \
    --model_name_or_path meta-llama/Meta-Llama-3-8B \
    --infer_backend huggingface

# vLLM backend

llamafactory-cli infer \
    --model_name_or_path meta-llama/Meta-Llama-3-8B \
    --infer_backend vllm

# SGLang backend

llamafactory-cli infer \
    --model_name_or_path meta-llama/Meta-Llama-3-8B \
    --infer_backend sglang

# K-Transformers backend

llamafactory-cli infer \
    --model_name_or_path deepseek-ai/DeepSeek-V2 \
    --infer_backend ktransformers

```

The CLI parser validates that non-HuggingFace backends are only used in inference contexts, enforced in [`src/llamafactory/hparams/parser.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/parser.py)【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/hparams/parser.py#L388-L389】.

### Web UI Selection

The Gradio interface exposes backend selection through a dropdown in [`src/llamafactory/webui/components/infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/components/infer.py):

```python
import gradio as gr

# Component definition (lines 36-38)

infer_backend = gr.Dropdown(
    choices=["huggingface", "vllm", "sglang"],  # Note: ktransformers not yet exposed in UI

    value="huggingface",
    label="Inference Backend"
)

```

Users select the desired engine from the dropdown before launching inference【/cache/repos/github.com/hiyouga/LlamaFactory/main/src/llamafactory/webui/components/infer.py#L36-L38】.

## Key Implementation Files

| File | Purpose | Key Lines |
|------|---------|-----------|
| [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) | Defines `EngineName` enum with four backend values | 22‑27 |
| [`src/llamafactory/hparams/model_args.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/model_args.py) | Declares `infer_backend` argument defaulting to `EngineName.HF` | 164‑166 |
| [`src/llamafactory/chat/chat_model.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/chat/chat_model.py) | Factory logic that instantiates specific engine classes | 50‑78 |
| [`src/llamafactory/webui/components/infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/components/infer.py) | Gradio dropdown for backend selection in Web UI | 36‑38 |
| [`src/llamafactory/hparams/parser.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/parser.py) | Validation logic ensuring non-HF backends are inference-only | 388‑389 |

## Summary

- LlamaFactory supports **four inference backends**: HuggingFace Transformers, vLLM, SGLang, and K-Transformers.
- Backend selection is controlled by the `infer_backend` parameter, defined in `ModelArguments` and defaulting to `"huggingface"`.
- The `EngineName` enum in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) provides the canonical identifiers for each backend.
- `ChatModel` acts as a factory, dynamically importing and instantiating the appropriate engine class based on the selected backend.
- All four backends are available via Python API and CLI, while the Web UI currently exposes HuggingFace, vLLM, and SGLang.

## Frequently Asked Questions

### What is the default inference backend in LlamaFactory?

**HuggingFace Transformers is the default backend.** When you do not specify the `infer_backend` argument, `ModelArguments` automatically sets it to `EngineName.HF` (value `"huggingface"`), which uses the standard 🤗 Transformers pipeline for model loading and generation.

### Can I use vLLM or SGLang for training in LlamaFactory?

**No, vLLM, SGLang, and K-Transformers are inference-only.** The argument parser in [`src/llamafactory/hparams/parser.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/parser.py) explicitly validates that these backends are only used during inference operations. For training workflows, you must use the HuggingFace backend.

### How do I install dependencies for a specific backend?

**Each backend requires its own optional dependency.** While HuggingFace works with the base `transformers` installation, you must install:
- **vLLM**: `pip install vllm`
- **SGLang**: `pip install sglang[all]`
- **K-Transformers**: `pip install ktransformers`

These packages are not included in the base LlamaFactory installation to keep the core footprint minimal.

### Why is K-Transformers not available in the Web UI dropdown?

**The Web UI currently exposes only HuggingFace, vLLM, and SGLang.** The Gradio dropdown component in [`src/llamafactory/webui/components/infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/components/infer.py) defines `choices=["huggingface", "vllm", "sglang"]`, excluding K-Transformers. Users requiring K-Transformers must use the Python API or CLI interface instead.