# Inference Capabilities of LlamaFactory: A Complete Guide to vLLM, HuggingFace, and SGLang Backends

> Explore LlamaFactory's inference capabilities with vLLM HuggingFace and SGLang backends. Deploy seamlessly across single-GPU multi-GPU and server environments.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: deep-dive
- Published: 2026-03-04

---

**LlamaFactory provides a unified inference layer supporting three interchangeable backends—HuggingFace, vLLM, and SGLang—enabling seamless deployment across single-GPU, multi-GPU, and server-based environments.**

The inference capabilities of LlamaFactory center on a flexible architecture that abstracts model execution behind a common API. According to the hiyouga/LlamaFactory source code, this design allows researchers and engineers to switch between optimized backends without changing application logic, supporting everything from CPU prototyping to high-throughput tensor-parallel serving.

## Unified Inference Architecture

At the core of LlamaFactory's inference system sits the abstract base class `BaseEngine` defined in [`src/llamafactory/v1/core/utils/inference_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/utils/inference_engine.py). This class establishes a strict contract that all concrete implementations must follow.

### The BaseEngine Contract

The `BaseEngine` class mandates two primary asynchronous methods:

```python
class BaseEngine(ABC):
    async def generate(self, messages, tools=None) -> AsyncGenerator[str, None]:
        ...
    
    async def batch_infer(self, dataset) -> List[Sample]:
        ...

```

All three inference engines—**`HuggingFaceEngine`**, **`VllmEngine`**, and **`SGLangEngine`**—inherit from this base class, ensuring consistent behavior regardless of the underlying infrastructure. The factory function `get_engine()` instantiates the appropriate concrete class based on the `backend` parameter, enabling runtime backend selection.

## Available Inference Backends

LlamaFactory implements three distinct backends, each optimized for specific deployment scenarios.

### HuggingFace Engine

The **`HuggingFaceEngine`** in [`src/llamafactory/v1/core/utils/inference_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/utils/inference_engine.py) (line 76) provides the most accessible entry point. It runs on CPU or any single-GPU torch device using the standard `transformers` library.

Key characteristics include:

- **Single-device execution** compatible with any 🤗 Transformers model
- **Background threading** utilizing `AsyncTextIteratorStreamer` to convert synchronous generation into asynchronous token streaming
- **Universal compatibility** with models that lack specialized inference optimizations

This backend excels during development and prototyping when simplicity matters more than throughput.

### vLLM Engine

For production workloads requiring maximum throughput, the **`VllmEngine`** in [`src/llamafactory/chat/vllm_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/chat/vllm_engine.py) (line 46) leverages the vLLM inference library.

Advanced capabilities include:

- **Tensor parallelism** via `tensor_parallel_size` parameter for multi-GPU scaling
- **Pipeline parallelism** support through `pipeline_parallel_size`
- **Extended context windows** handled via `max_model_len` configuration
- **Dynamic LoRA loading** using `vllm.lora` for adapter switching without model restarts

The vLLM backend automatically manages KV-cache memory and continuous batching, delivering significantly higher tokens-per-second than standard Transformers generation.

### SGLang Engine

The **`SGLangEngine`** in [`src/llamafactory/chat/sglang_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/chat/sglang_engine.py) (line 46) targets server-side deployments requiring HTTP-based inference with multimodal support.

Notable features include:

- **Dedicated server process** launched via `launch_server_cmd` and communicated with over HTTP
- **Multimodal inputs** with server-side processing of images, video, and audio
- **Dynamic LoRA paths** specified via `--lora-paths` argument for runtime adapter selection
- **Streaming chunks** returned directly from the SGLang server endpoint

This backend is ideal for microservice architectures and applications requiring strict separation between client and inference infrastructure.

## Key Inference Features

Regardless of backend selection, LlamaFactory exposes a consistent feature set through the unified API.

### Streaming and Batch Generation

All engines support **streaming generation** through the `generate()` and `stream_chat()` methods, yielding tokens incrementally for real-time user interfaces. For offline processing, the **`batch_infer()`** method processes entire datasets efficiently, as implemented in [`scripts/vllm_infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/scripts/vllm_infer.py) for the vLLM backend.

### Multimodal Support

The inference layer automatically processes multimodal inputs by inserting placeholders like `<image>`, `<video>`, and `<audio>` into prompts. These tokens route to the model's multimodal plugin for encoding, enabling vision-language and audio-language model inference without manual preprocessing.

### LoRA Adapter Loading

All three backends support on-the-fly **LoRA adapter** loading via the `adapter_name_or_path` parameter. This allows switching between fine-tuned variants without reloading the base model weights, critical for multi-tenant serving scenarios.

### Sampling Controls and Tool Usage

The engines expose comprehensive sampling parameters including `temperature`, `top_p`, `top_k`, `repetition_penalty`, `max_new_tokens`, and custom stop tokens. Additionally, the API accepts optional `system` prompts and `tools` specifications for function-calling models, enabling agentic workflows across all backends.

## Practical Usage Examples

### CLI Batch Inference with vLLM

For high-throughput offline generation, use the standalone script:

```bash
python scripts/vllm_infer.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --template llama \
    --dataset alpaca_en_demo \
    --vllm_config '{"gpu_memory_utilization": 0.8}'

```

This script initializes the `VllmEngine`, processes the entire dataset, and writes results to `generated_predictions.jsonl` (line 47 of the script).

### Interactive Web UI Inference

The Gradio interface constructs inference tabs dynamically:

```python
from llamafactory.webui.components.infer import create_infer_tab

```

The `create_infer_tab` function in [`src/llamafactory/webui/components/infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/components/infer.py) (line 32) renders backend selection dropdowns, model load/unload controls, and a streaming chat interface. Users can toggle between HuggingFace, vLLM, and SGLang backends through the UI without code changes.

### Programmatic Python API

For custom applications, instantiate engines directly:

```python
from llamafactory.chat import get_engine
from llamafactory.hparams import (
    ModelArguments, DataArguments, 
    FinetuningArguments, GeneratingArguments
)

model_args = ModelArguments(model_name_or_path="meta-llama/Llama-2-7b-hf")
data_args = DataArguments()
finetune_args = FinetuningArguments(stage="sft")
gen_args = GeneratingArguments(max_new_tokens=128)

engine = get_engine(
    backend="vllm",  # or "huggingface", "sglang"

    model_args=model_args,
    data_args=data_args,
    finetuning_args=finetune_args,
    generating_args=gen_args,
)

# Streaming inference

async for token in engine.chat(
    messages=[{"role": "user", "content": "Explain quantum computing"}]
):
    print(token, end='', flush=True)

```

The `get_engine` factory handles backend instantiation, while `engine.chat()` provides uniform access to streaming responses.

## Summary

- **LlamaFactory unifies inference** across HuggingFace, vLLM, and SGLang backends through the abstract `BaseEngine` class in [`src/llamafactory/v1/core/utils/inference_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/utils/inference_engine.py).
- **Three backend implementations** cater to different scales: single-device prototyping (`HuggingFaceEngine`), high-throughput tensor-parallel serving (`VllmEngine`), and HTTP-based multimodal servers (`SGLangEngine`).
- **Common capabilities** include streaming generation, batch inference, multimodal placeholder processing, dynamic LoRA loading, and comprehensive sampling controls.
- **Multiple interfaces** support various workflows: CLI scripts for batch processing ([`scripts/vllm_infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/scripts/vllm_infer.py)), web UI components ([`src/llamafactory/webui/components/infer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/webui/components/infer.py)), and programmatic Python APIs.

## Frequently Asked Questions

### Does LlamaFactory support multi-GPU inference for large models?

**Yes.** The `VllmEngine` automatically distributes models across multiple GPUs using tensor parallelism via the `tensor_parallel_size` parameter. You can also utilize pipeline parallelism through `pipeline_parallel_size` for extremely large models that exceed single-node GPU memory.

### Can I switch between different LoRA adapters without restarting the inference server?

**Yes.** All three backends support dynamic LoRA loading. For HuggingFace and vLLM engines, specify `adapter_name_or_path` during initialization or use the vLLM-specific LoRA interface. The SGLang engine accepts `--lora-paths` arguments to load multiple adapters simultaneously without reloading base model weights.

### How does LlamaFactory handle multimodal inputs like images and video?

**The inference engine automatically inserts placeholder tokens** (e.g., `<image>`, `<video>`, `<audio>`) into prompts and routes these to the model's multimodal processor. This works uniformly across all backends, allowing vision-language models to process mixed-media inputs through the standard `chat()` or `generate()` interfaces.

### What is the difference between using the HuggingFace backend versus vLLM for inference?

**The HuggingFace backend** uses standard `transformers` generation with `AsyncTextIteratorStreamer` for single-device CPU or GPU execution, prioritizing compatibility over speed. **The vLLM backend** implements PagedAttention with continuous batching and tensor parallelism, delivering significantly higher throughput for production serving but requiring GPU resources and the vLLM library installation.