Inference Capabilities of LlamaFactory: A Complete Guide to vLLM, HuggingFace, and SGLang Backends

LlamaFactory provides a unified inference layer supporting three interchangeable backends—HuggingFace, vLLM, and SGLang—enabling seamless deployment across single-GPU, multi-GPU, and server-based environments.

The inference capabilities of LlamaFactory center on a flexible architecture that abstracts model execution behind a common API. According to the hiyouga/LlamaFactory source code, this design allows researchers and engineers to switch between optimized backends without changing application logic, supporting everything from CPU prototyping to high-throughput tensor-parallel serving.

Unified Inference Architecture

At the core of LlamaFactory's inference system sits the abstract base class BaseEngine defined in src/llamafactory/v1/core/utils/inference_engine.py. This class establishes a strict contract that all concrete implementations must follow.

The BaseEngine Contract

The BaseEngine class mandates two primary asynchronous methods:

class BaseEngine(ABC):
    async def generate(self, messages, tools=None) -> AsyncGenerator[str, None]:
        ...
    
    async def batch_infer(self, dataset) -> List[Sample]:
        ...

All three inference engines—HuggingFaceEngine, VllmEngine, and SGLangEngine—inherit from this base class, ensuring consistent behavior regardless of the underlying infrastructure. The factory function get_engine() instantiates the appropriate concrete class based on the backend parameter, enabling runtime backend selection.

Available Inference Backends

LlamaFactory implements three distinct backends, each optimized for specific deployment scenarios.

HuggingFace Engine

The HuggingFaceEngine in src/llamafactory/v1/core/utils/inference_engine.py (line 76) provides the most accessible entry point. It runs on CPU or any single-GPU torch device using the standard transformers library.

Key characteristics include:

• Single-device execution compatible with any 🤗 Transformers model
• Background threading utilizing AsyncTextIteratorStreamer to convert synchronous generation into asynchronous token streaming
• Universal compatibility with models that lack specialized inference optimizations

This backend excels during development and prototyping when simplicity matters more than throughput.

vLLM Engine

For production workloads requiring maximum throughput, the VllmEngine in src/llamafactory/chat/vllm_engine.py (line 46) leverages the vLLM inference library.

Advanced capabilities include:

• Tensor parallelism via tensor_parallel_size parameter for multi-GPU scaling
• Pipeline parallelism support through pipeline_parallel_size
• Extended context windows handled via max_model_len configuration
• Dynamic LoRA loading using vllm.lora for adapter switching without model restarts

The vLLM backend automatically manages KV-cache memory and continuous batching, delivering significantly higher tokens-per-second than standard Transformers generation.

SGLang Engine

The SGLangEngine in src/llamafactory/chat/sglang_engine.py (line 46) targets server-side deployments requiring HTTP-based inference with multimodal support.

Notable features include:

• Dedicated server process launched via launch_server_cmd and communicated with over HTTP
• Multimodal inputs with server-side processing of images, video, and audio
• Dynamic LoRA paths specified via --lora-paths argument for runtime adapter selection
• Streaming chunks returned directly from the SGLang server endpoint

This backend is ideal for microservice architectures and applications requiring strict separation between client and inference infrastructure.

Key Inference Features

Regardless of backend selection, LlamaFactory exposes a consistent feature set through the unified API.

Streaming and Batch Generation

All engines support streaming generation through the generate() and stream_chat() methods, yielding tokens incrementally for real-time user interfaces. For offline processing, the batch_infer() method processes entire datasets efficiently, as implemented in scripts/vllm_infer.py for the vLLM backend.

Multimodal Support

The inference layer automatically processes multimodal inputs by inserting placeholders like <image>, <video>, and <audio> into prompts. These tokens route to the model's multimodal plugin for encoding, enabling vision-language and audio-language model inference without manual preprocessing.

LoRA Adapter Loading

All three backends support on-the-fly LoRA adapter loading via the adapter_name_or_path parameter. This allows switching between fine-tuned variants without reloading the base model weights, critical for multi-tenant serving scenarios.

Sampling Controls and Tool Usage

The engines expose comprehensive sampling parameters including temperature, top_p, top_k, repetition_penalty, max_new_tokens, and custom stop tokens. Additionally, the API accepts optional system prompts and tools specifications for function-calling models, enabling agentic workflows across all backends.

Practical Usage Examples

CLI Batch Inference with vLLM

For high-throughput offline generation, use the standalone script:

python scripts/vllm_infer.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --template llama \
    --dataset alpaca_en_demo \
    --vllm_config '{"gpu_memory_utilization": 0.8}'

This script initializes the VllmEngine, processes the entire dataset, and writes results to generated_predictions.jsonl (line 47 of the script).

Interactive Web UI Inference

The Gradio interface constructs inference tabs dynamically:

from llamafactory.webui.components.infer import create_infer_tab

The create_infer_tab function in src/llamafactory/webui/components/infer.py (line 32) renders backend selection dropdowns, model load/unload controls, and a streaming chat interface. Users can toggle between HuggingFace, vLLM, and SGLang backends through the UI without code changes.

Programmatic Python API

For custom applications, instantiate engines directly:

from llamafactory.chat import get_engine
from llamafactory.hparams import (
    ModelArguments, DataArguments, 
    FinetuningArguments, GeneratingArguments
)

model_args = ModelArguments(model_name_or_path="meta-llama/Llama-2-7b-hf")
data_args = DataArguments()
finetune_args = FinetuningArguments(stage="sft")
gen_args = GeneratingArguments(max_new_tokens=128)

engine = get_engine(
    backend="vllm",  # or "huggingface", "sglang"

    model_args=model_args,
    data_args=data_args,
    finetuning_args=finetune_args,
    generating_args=gen_args,
)

# Streaming inference

async for token in engine.chat(
    messages=[{"role": "user", "content": "Explain quantum computing"}]
):
    print(token, end='', flush=True)

The get_engine factory handles backend instantiation, while engine.chat() provides uniform access to streaming responses.

Summary

• LlamaFactory unifies inference across HuggingFace, vLLM, and SGLang backends through the abstract BaseEngine class in src/llamafactory/v1/core/utils/inference_engine.py.
• Three backend implementations cater to different scales: single-device prototyping (HuggingFaceEngine), high-throughput tensor-parallel serving (VllmEngine), and HTTP-based multimodal servers (SGLangEngine).
• Common capabilities include streaming generation, batch inference, multimodal placeholder processing, dynamic LoRA loading, and comprehensive sampling controls.
• Multiple interfaces support various workflows: CLI scripts for batch processing (scripts/vllm_infer.py), web UI components (src/llamafactory/webui/components/infer.py), and programmatic Python APIs.

Frequently Asked Questions

Does LlamaFactory support multi-GPU inference for large models?

Yes. The VllmEngine automatically distributes models across multiple GPUs using tensor parallelism via the tensor_parallel_size parameter. You can also utilize pipeline parallelism through pipeline_parallel_size for extremely large models that exceed single-node GPU memory.

Can I switch between different LoRA adapters without restarting the inference server?

Yes. All three backends support dynamic LoRA loading. For HuggingFace and vLLM engines, specify adapter_name_or_path during initialization or use the vLLM-specific LoRA interface. The SGLang engine accepts --lora-paths arguments to load multiple adapters simultaneously without reloading base model weights.

How does LlamaFactory handle multimodal inputs like images and video?

The inference engine automatically inserts placeholder tokens (e.g., <image>, <video>, <audio>) into prompts and routes these to the model's multimodal processor. This works uniformly across all backends, allowing vision-language models to process mixed-media inputs through the standard chat() or generate() interfaces.

What is the difference between using the HuggingFace backend versus vLLM for inference?

The HuggingFace backend uses standard transformers generation with AsyncTextIteratorStreamer for single-device CPU or GPU execution, prioritizing compatibility over speed. The vLLM backend implements PagedAttention with continuous batching and tensor parallelism, delivering significantly higher throughput for production serving but requiring GPU resources and the vLLM library installation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}