How to Use the OpenAI-Compatible API Provided by LlamaFactory

LlamaFactory provides a FastAPI-based HTTP server that exposes OpenAI-compatible endpoints, allowing you to use standard OpenAI SDKs and tools with any supported model by simply changing the base URL.

The OpenAI-compatible API in LlamaFactory enables seamless integration of fine-tuned or base models into existing applications built for OpenAI's platform. Built on FastAPI, the service implements the same REST contract as the official OpenAI API, including support for chat completions, streaming responses, and model listing.

Architecture Overview

The API implementation consists of four core components that work together to translate OpenAI-style requests into LlamaFactory's internal inference pipeline:

Component	Role	Source File
FastAPI Application	Creates the web server, configures CORS, implements API-key authentication, and registers routes.	src/llamafactory/api/app.py
Protocol Definitions	Pydantic models that mirror OpenAI's JSON schemas for requests and responses (e.g., ChatCompletionRequest, ChatCompletionResponse).	src/llamafactory/api/protocol.py
Chat Endpoint Logic	Core implementation handling request validation, multimodal input processing, tool-call extraction, and streaming logic.	src/llamafactory/api/chat.py
Server Entry Point	CLI launcher that constructs the ChatModel, builds the FastAPI app, and starts uvicorn.	src/api.py

Request Processing Flow

When a client sends a request to the OpenAI-compatible API, the following sequence occurs:

1. The HTTP request arrives at the FastAPI server (e.g., POST /v1/chat/completions) defined in src/llamafactory/api/app.py.
2. The route handler validates the optional bearer token against the API_KEY environment variable.
3. The request payload is forwarded to _process_request in src/llamafactory/api/chat.py, which converts the OpenAI-style messages into LlamaFactory's internal format.
4. The converted request is passed to a ChatModel instance via chat_model.achat (for standard responses) or chat_model.astream_chat (for streaming).
5. Results are wrapped into OpenAI-compatible response models (ChatCompletionResponse or ChatCompletionStreamResponse) and returned to the client.

Configuration Environment Variables

The API server behavior is controlled through environment variables read at startup:

Variable	Description	Default
API_HOST	Host IP address the server binds to	0.0.0.0
API_PORT	TCP port for incoming connections	8000
API_KEY	Optional bearer token required for authentication	none
API_MODEL_NAME	Model identifier returned by /v1/models	gpt-3.5-turbo
FASTAPI_ROOT_PATH	URL prefix for all routes (useful behind reverse proxies)	empty

Starting the API Server

To launch the OpenAI-compatible API, install the optional dependencies and start the server using the provided entry point:

# Install FastAPI and server dependencies

pip install "llamafactory[api]"

# Configure the environment

export API_HOST=0.0.0.0
export API_PORT=8000
export API_KEY=your-secret-key
export API_MODEL_NAME=llama2-7b-chat

# Launch the server

python -m src.api

Upon startup, the server logs the OpenAPI documentation URL:

Visit http://localhost:8000/docs for API document.

API Endpoints and Usage

The server exposes three primary endpoints that mirror OpenAI's API specification.

List Available Models

Retrieve the model metadata configured via API_MODEL_NAME:

curl -H "Authorization: Bearer your-secret-key" \
  http://localhost:8000/v1/models

Chat Completions (Non-Streaming)

Send a standard chat completion request:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": [{"role": "user", "content": "Explain quantum computing in simple terms."}]
  }'

Streaming Chat Completions

Enable streaming by setting stream: true. The response uses Server-Sent Events (SSE):

curl -N -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": [{"role": "user", "content": "Tell me a joke."}],
    "stream": true
  }'

Each line contains a JSON chunk conforming to the ChatCompletionStreamResponse schema defined in src/llamafactory/api/protocol.py.

Score Evaluation

Evaluate text sequences using the score endpoint:

curl -X POST http://localhost:8000/v1/score/evaluation \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": ["The quick brown fox jumps over the lazy dog."],
    "max_length": 10
  }'

Using the Official OpenAI Python Client

Because LlamaFactory implements the OpenAI API contract, you can use the official openai Python library by simply redirecting the base URL:

import os
import openai

# Configure the client to point to LlamaFactory

os.environ["OPENAI_API_BASE"] = "http://localhost:8000/v1"
os.environ["OPENAI_API_KEY"] = "your-secret-key"

client = openai.OpenAI(
    base_url=os.getenv("OPENAI_API_BASE"),
    api_key=os.getenv("OPENAI_API_KEY")
)

# Standard chat completion call

response = client.chat.completions.create(
    model="llama2-7b-chat",
    messages=[{"role": "user", "content": "Write a haiku about machine learning."}]
)

print(response.choices[0].message.content)

This approach requires zero code changes to existing applications using the OpenAI SDK—only the environment variables need updating.

Key Source Files

The OpenAI-compatible API implementation spans these critical files in the LlamaFactory repository:

File	Purpose	Location
src/api.py	Entry point that constructs the ChatModel, initializes the FastAPI application, and launches uvicorn.	src/api.py
src/llamafactory/api/app.py	Defines create_app(), configures CORS middleware, implements API-key validation, and registers the /v1/* routes.	src/llamafactory/api/app.py
src/llamafactory/api/protocol.py	Contains Pydantic models (ChatCompletionRequest, ChatCompletionResponse, ModelList, etc.) that ensure strict OpenAI API compatibility.	src/llamafactory/api/protocol.py
src/llamafactory/api/chat.py	Implements _process_request() and the streaming/non-streaming handlers that bridge OpenAI-style payloads to LlamaFactory's internal ChatModel interface.	src/llamafactory/api/chat.py

Summary

• LlamaFactory provides an OpenAI-compatible API built on FastAPI that exposes standard endpoints including /v1/chat/completions, /v1/models, and /v1/score/evaluation.
• The implementation relies on four core files: src/api.py (entry point), src/llamafactory/api/app.py (FastAPI setup), src/llamafactory/api/protocol.py (Pydantic schemas), and src/llamafactory/api/chat.py (request processing).
• Configuration is handled via environment variables (API_HOST, API_PORT, API_KEY, API_MODEL_NAME) read at startup in app.py and api.py.
• You can interact with the server using standard curl commands, HTTP clients, or the official OpenAI Python SDK by simply changing the base_url to point to your local LlamaFactory instance.

Frequently Asked Questions

What endpoints does the LlamaFactory OpenAI-compatible API support?

The API supports three primary endpoints: GET /v1/models for listing available models, POST /v1/chat/completions for chat-based inference with optional streaming, and POST /v1/score/evaluation for scoring text sequences. These routes are registered in src/llamafactory/api/app.py and processed by the handlers in src/llamafactory/api/chat.py.

How do I enable authentication for the API server?

Set the API_KEY environment variable before starting the server. When defined, the create_app function in src/llamafactory/api/app.py enforces bearer token validation on all incoming requests. Clients must then include the header Authorization: Bearer your-secret-key in every request, as demonstrated in the curl examples.

Can I use the official OpenAI Python library with LlamaFactory?

Yes. Because LlamaFactory implements the same request/response schemas defined in src/llamafactory/api/protocol.py, you can use the official openai Python package by simply configuring the base_url to point to your LlamaFactory server (e.g., http://localhost:8000/v1) and providing the matching API_KEY. No other code changes are required.

How does streaming work in the chat completions endpoint?

When the request payload includes "stream": true, the handler in src/llamafactory/api/chat.py invokes chat_model.astream_chat and returns a Server-Sent Events (SSE) stream. Each event contains a JSON chunk conforming to the ChatCompletionStreamResponse schema from src/llamafactory/api/protocol.py, allowing clients to consume tokens incrementally exactly as they would with the official OpenAI API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}