# How to Use llama-server with OpenAI API Compatible Endpoints: A Complete Guide

> Easily connect OpenAI clients to local GGUF models using llama-server. This guide shows how to leverage OpenAI API compatible endpoints for seamless interaction with your models.

- Repository: [ggml/llama.cpp](https://github.com/ggml-org/llama.cpp)
- Tags: how-to-guide
- Published: 2026-02-27

---

**The llama.cpp `server` binary exposes OpenAI-compatible REST endpoints at `/v1/completions`, `/v1/chat/completions`, and `/v1/embeddings`, allowing any standard OpenAI client library to interact with local GGUF models without code modifications.**

The [`llama.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/llama.cpp) project includes a lightweight HTTP server that implements the OpenAI API contract, enabling developers to serve quantized LLMs through familiar REST interfaces. This implementation maps incoming OpenAI request fields directly to the internal `llama_*` inference API, making it compatible with existing tools like the OpenAI Python SDK, LangChain, and curl.

## Architecture Overview

The server implementation resides in `tools/server/` and consists of modular components that handle HTTP routing, request validation, and model inference.

### Core Components

- **[`server.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server.cpp)** — Entry point located at [`tools/server/server.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server.cpp) that parses command-line arguments, loads the GGUF model, instantiates `server_context`, and starts the HTTP listener on the specified port.

- **[`server-http.h`](https://github.com/ggml-org/llama.cpp/blob/main/server-http.h) and [`server-http.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-http.cpp)** — Implements the HTTP server using `httplib`, routing requests to `/v1/completions`, `/v1/chat/completions`, `/v1/embeddings`, and health-check endpoints. These files serialize responses into the exact JSON schema expected by OpenAI clients.

- **[`server-context.h`](https://github.com/ggml-org/llama.cpp/blob/main/server-context.h) and [`server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-context.cpp)** — Contains the `server_context` class that manages per-model state including the tokenizer, KV cache, and sampling parameters. This layer converts OpenAI parameters like `max_tokens`, `temperature`, and `top_p` into internal llama.cpp API calls and formats the output to include `usage` statistics and `finish_reason` fields.

- **[`server-common.h`](https://github.com/ggml-org/llama.cpp/blob/main/server-common.h) and [`server-common.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-common.cpp)** — Shared utilities using `nlohmann::json` for parsing, request validation logic, and HTTP status code mapping for OpenAI-compatible error responses.

When a request arrives, [`server-http.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-http.cpp) parses the JSON payload, validates required fields like `model` and `prompt` (or `messages`), delegates inference to `server_context::run_completion` or `run_chat` (which invoke `llama_eval`), then serializes the response into the OpenAI schema.

## Building the Server

Before using the OpenAI-compatible endpoints, compile the server binary from the llama.cpp source code.

```bash

# Clone the repository

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# Configure build with server support enabled

cmake -B build -DLLAMA_SERVER=ON -DLLAMA_BUILD_EXAMPLES=ON

# Compile the server target

cmake --build build --target server

```

The compiled binary appears at `build/bin/server`. Enable GPU acceleration by adding `-DLLAMA_CUDA=ON` (for NVIDIA) or `-DLLAMA_METAL=ON` (for Apple Silicon) to the CMake configuration.

## Starting the Server

Launch the server by specifying a GGUF model and context window size. The server loads the model into memory and begins listening for HTTP requests.

```bash
./build/bin/server \
    -m models/llama-2-7b-chat.Q4_0.gguf \
    -c 4096 \
    -ngl 32 \
    --port 8080

```

- **`-m`** — Path to the GGUF model file.
- **`-c`** — Context size in tokens (e.g., `4096` or `8192`).
- **`-ngl`** — Number of GPU layers to offload for acceleration.
- **`--port`** — HTTP port (defaults to `8080`).

Standard output confirms the server status:

```

[2024-03-01 12:00:00] server: listening on http://0.0.0.0:8080
[2024-03-01 12:00:00] model loaded: llama-2-7b-chat.Q4_0.gguf (4096 ctx)

```

## OpenAI-Compatible Endpoints

The server implements three primary endpoints that mirror the OpenAI API specification.

### Text Completions (/v1/completions)

Send a prompt to the model using the legacy completions endpoint. This maps to the `run_completion` method in [`server-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-context.cpp).

```bash
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "prompt": "Explain quantum entanglement in simple terms.",
        "max_tokens": 150,
        "temperature": 0.7,
        "stream": false
      }'

```

The response follows the OpenAI schema exactly:

```json
{
  "id": "cmpl-01a2b3c4d5e6f7g8h9i0j",
  "object": "text_completion",
  "created": 1709283456,
  "model": "llama-2-7b-chat",
  "choices": [
    {
      "text": "Quantum entanglement is a phenomenon where...",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 147,
    "total_tokens": 159
  }
}

```

### Chat Completions (/v1/chat/completions)

Use the chat endpoint for conversational interactions. The server processes the `messages` array through `server_context::run_chat` and applies the appropriate chat template.

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the meaning of life?"}
        ],
        "max_tokens": 200,
        "temperature": 0.5,
        "stream": false
      }'

```

Response structure:

```json
{
  "id": "chatcmpl-12345",
  "object": "chat.completion",
  "created": 1709283500,
  "model": "llama-2-7b-chat",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "The meaning of life is subjective..."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 24, "completion_tokens": 56, "total_tokens": 80}
}

```

### Streaming Responses

Enable real-time token delivery by setting `stream: true`. The server uses Server-Sent Events (SSE) format, emitting partial JSON objects identical to OpenAI's streaming implementation.

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
        "model": "llama-2-7b-chat",
        "messages": [{"role":"user","content":"Tell me a joke."}],
        "stream": true
      }'

```

Each line contains a `delta` object with incremental content, allowing UI clients to display text as it generates.

### Embeddings (/v1/embeddings)

Generate vector embeddings for input text. The server returns an array of floating-point values compatible with OpenAI's embedding format.

```bash
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "input": "Hello world"
      }'

```

Example response:

```json
{
  "object": "list",
  "data": [
    {"object":"embedding","embedding":[0.12, -0.08, 0.05, ...],"index":0}
  ],
  "model": "llama-2-7b-chat",
  "usage": {"prompt_tokens":2,"total_tokens":2}
}

```

## Configuring OpenAI Clients

Any client library that accepts a custom `base_url` can connect to the llama.cpp server. Set the base URL to `http://localhost:8080` (or your host:port) and provide any string for the API key.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="llama-2-7b-chat",  # Model name is ignored by server but required

    messages=[{"role": "user", "content": "Hello!"}]
)

```

This configuration works because [`server-http.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-http.cpp) implements the exact route structure and JSON schema that OpenAI clients expect.

## Summary

- **Build** the server with `-DLLAMA_SERVER=ON` to enable OpenAI-compatible endpoints.
- **Start** the server using `./build/bin/server` with flags `-m` for the model and `-c` for context size.
- **Query** `/v1/completions` for text generation and `/v1/chat/completions` for conversational AI.
- **Stream** responses by setting `stream: true` in your JSON payload.
- **Connect** any OpenAI SDK client by pointing `base_url` to the local server address.

## Frequently Asked Questions

### What endpoints does llama-server implement?

The server implements `/v1/completions`, `/v1/chat/completions`, `/v1/embeddings`, and health-check endpoints. These routes are defined in [`tools/server/server-http.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/tools/server/server-http.cpp) and return JSON structures matching the OpenAI API specification, including fields like `id`, `object`, `created`, `model`, `choices`, and `usage`.

### Does llama-server support API authentication?

The server does not implement API key validation by default. Authentication must be handled via a reverse proxy (such as nginx) if required for production deployments. The [`server-common.cpp`](https://github.com/ggml-org/llama.cpp/blob/main/server-common.cpp) file handles request validation but focuses on parameter checking rather than authorization headers.

### How do I enable GPU acceleration when running the server?

Add the `-ngl` (number of GPU layers) flag when starting the server: `./build/bin/server -m model.gguf -ngl 99`. The maximum value depends on your GPU memory. For CUDA builds, ensure you compiled with `-DLLAMA_CUDA=ON`; for Apple Silicon, use `-DLLAMA_METAL=ON`.

### Can I use the OpenAI Python SDK with llama-server?

Yes. Configure the OpenAI client with `base_url="http://localhost:8080/v1"` and any placeholder for `api_key`. The SDK will communicate with the llama.cpp server exactly as it would with OpenAI's official API, allowing you to use llama-server with OpenAI API compatible endpoints in existing applications without code changes.