# How to Use the OpenAI-Compatible API Provided by LlamaFactory

> Learn how to use LlamaFactory's OpenAI-compatible API. Connect any model to standard OpenAI SDKs by changing the base URL and leverage powerful LLMs with ease.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory provides a FastAPI-based HTTP server that exposes OpenAI-compatible endpoints, allowing you to use standard OpenAI SDKs and tools with any supported model by simply changing the base URL.**

The **OpenAI-compatible API** in LlamaFactory enables seamless integration of fine-tuned or base models into existing applications built for OpenAI's platform. Built on **FastAPI**, the service implements the same REST contract as the official OpenAI API, including support for chat completions, streaming responses, and model listing.

## Architecture Overview

The API implementation consists of four core components that work together to translate OpenAI-style requests into LlamaFactory's internal inference pipeline:

| Component | Role | Source File |
|-----------|------|-------------|
| **FastAPI Application** | Creates the web server, configures CORS, implements API-key authentication, and registers routes. | [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) |
| **Protocol Definitions** | Pydantic models that mirror OpenAI's JSON schemas for requests and responses (e.g., `ChatCompletionRequest`, `ChatCompletionResponse`). | [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py) |
| **Chat Endpoint Logic** | Core implementation handling request validation, multimodal input processing, tool-call extraction, and streaming logic. | [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py) |
| **Server Entry Point** | CLI launcher that constructs the `ChatModel`, builds the FastAPI app, and starts **uvicorn**. | [`src/api.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/api.py) |

### Request Processing Flow

When a client sends a request to the OpenAI-compatible API, the following sequence occurs:

1. The HTTP request arrives at the FastAPI server (e.g., `POST /v1/chat/completions`) defined in [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py).
2. The route handler validates the optional bearer token against the `API_KEY` environment variable.
3. The request payload is forwarded to `_process_request` in [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py), which converts the OpenAI-style messages into LlamaFactory's internal format.
4. The converted request is passed to a `ChatModel` instance via `chat_model.achat` (for standard responses) or `chat_model.astream_chat` (for streaming).
5. Results are wrapped into OpenAI-compatible response models (`ChatCompletionResponse` or `ChatCompletionStreamResponse`) and returned to the client.

## Configuration Environment Variables

The API server behavior is controlled through environment variables read at startup:

| Variable | Description | Default |
|----------|-------------|---------|
| `API_HOST` | Host IP address the server binds to | `0.0.0.0` |
| `API_PORT` | TCP port for incoming connections | `8000` |
| `API_KEY` | Optional bearer token required for authentication | *none* |
| `API_MODEL_NAME` | Model identifier returned by `/v1/models` | `gpt-3.5-turbo` |
| `FASTAPI_ROOT_PATH` | URL prefix for all routes (useful behind reverse proxies) | *empty* |

## Starting the API Server

To launch the OpenAI-compatible API, install the optional dependencies and start the server using the provided entry point:

```bash

# Install FastAPI and server dependencies

pip install "llamafactory[api]"

# Configure the environment

export API_HOST=0.0.0.0
export API_PORT=8000
export API_KEY=your-secret-key
export API_MODEL_NAME=llama2-7b-chat

# Launch the server

python -m src.api

```

Upon startup, the server logs the OpenAPI documentation URL:

```

Visit http://localhost:8000/docs for API document.

```

## API Endpoints and Usage

The server exposes three primary endpoints that mirror OpenAI's API specification.

### List Available Models

Retrieve the model metadata configured via `API_MODEL_NAME`:

```bash
curl -H "Authorization: Bearer your-secret-key" \
  http://localhost:8000/v1/models

```

### Chat Completions (Non-Streaming)

Send a standard chat completion request:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": [{"role": "user", "content": "Explain quantum computing in simple terms."}]
  }'

```

### Streaming Chat Completions

Enable streaming by setting `stream: true`. The response uses Server-Sent Events (SSE):

```bash
curl -N -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": [{"role": "user", "content": "Tell me a joke."}],
    "stream": true
  }'

```

Each line contains a JSON chunk conforming to the `ChatCompletionStreamResponse` schema defined in [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py).

### Score Evaluation

Evaluate text sequences using the score endpoint:

```bash
curl -X POST http://localhost:8000/v1/score/evaluation \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-secret-key" \
  -d '{
    "model": "llama2-7b-chat",
    "messages": ["The quick brown fox jumps over the lazy dog."],
    "max_length": 10
  }'

```

## Using the Official OpenAI Python Client

Because LlamaFactory implements the OpenAI API contract, you can use the official `openai` Python library by simply redirecting the base URL:

```python
import os
import openai

# Configure the client to point to LlamaFactory

os.environ["OPENAI_API_BASE"] = "http://localhost:8000/v1"
os.environ["OPENAI_API_KEY"] = "your-secret-key"

client = openai.OpenAI(
    base_url=os.getenv("OPENAI_API_BASE"),
    api_key=os.getenv("OPENAI_API_KEY")
)

# Standard chat completion call

response = client.chat.completions.create(
    model="llama2-7b-chat",
    messages=[{"role": "user", "content": "Write a haiku about machine learning."}]
)

print(response.choices[0].message.content)

```

This approach requires zero code changes to existing applications using the OpenAI SDK—only the environment variables need updating.

## Key Source Files

The OpenAI-compatible API implementation spans these critical files in the LlamaFactory repository:

| File | Purpose | Location |
|------|---------|----------|
| [`src/api.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/api.py) | Entry point that constructs the `ChatModel`, initializes the FastAPI application, and launches **uvicorn**. | [src/api.py](https://github.com/hiyouga/LlamaFactory/blob/main/src/api.py) |
| [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) | Defines `create_app()`, configures CORS middleware, implements API-key validation, and registers the `/v1/*` routes. | [src/llamafactory/api/app.py](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) |
| [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py) | Contains Pydantic models (`ChatCompletionRequest`, `ChatCompletionResponse`, `ModelList`, etc.) that ensure strict OpenAI API compatibility. | [src/llamafactory/api/protocol.py](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py) |
| [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py) | Implements `_process_request()` and the streaming/non-streaming handlers that bridge OpenAI-style payloads to LlamaFactory's internal `ChatModel` interface. | [src/llamafactory/api/chat.py](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py) |

## Summary

- LlamaFactory provides an **OpenAI-compatible API** built on FastAPI that exposes standard endpoints including `/v1/chat/completions`, `/v1/models`, and `/v1/score/evaluation`.
- The implementation relies on four core files: [`src/api.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/api.py) (entry point), [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) (FastAPI setup), [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py) (Pydantic schemas), and [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py) (request processing).
- Configuration is handled via environment variables (`API_HOST`, `API_PORT`, `API_KEY`, `API_MODEL_NAME`) read at startup in [`app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/app.py) and [`api.py`](https://github.com/hiyouga/LlamaFactory/blob/main/api.py).
- You can interact with the server using standard `curl` commands, HTTP clients, or the official OpenAI Python SDK by simply changing the `base_url` to point to your local LlamaFactory instance.

## Frequently Asked Questions

### What endpoints does the LlamaFactory OpenAI-compatible API support?

The API supports three primary endpoints: `GET /v1/models` for listing available models, `POST /v1/chat/completions` for chat-based inference with optional streaming, and `POST /v1/score/evaluation` for scoring text sequences. These routes are registered in [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) and processed by the handlers in [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py).

### How do I enable authentication for the API server?

Set the `API_KEY` environment variable before starting the server. When defined, the `create_app` function in [`src/llamafactory/api/app.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/app.py) enforces bearer token validation on all incoming requests. Clients must then include the header `Authorization: Bearer your-secret-key` in every request, as demonstrated in the `curl` examples.

### Can I use the official OpenAI Python library with LlamaFactory?

Yes. Because LlamaFactory implements the same request/response schemas defined in [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py), you can use the official `openai` Python package by simply configuring the `base_url` to point to your LlamaFactory server (e.g., `http://localhost:8000/v1`) and providing the matching `API_KEY`. No other code changes are required.

### How does streaming work in the chat completions endpoint?

When the request payload includes `"stream": true`, the handler in [`src/llamafactory/api/chat.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/chat.py) invokes `chat_model.astream_chat` and returns a Server-Sent Events (SSE) stream. Each event contains a JSON chunk conforming to the `ChatCompletionStreamResponse` schema from [`src/llamafactory/api/protocol.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/api/protocol.py), allowing clients to consume tokens incrementally exactly as they would with the official OpenAI API.