How to Use llama-server with OpenAI API Compatible Endpoints: A Complete Guide

The llama.cpp server binary exposes OpenAI-compatible REST endpoints at /v1/completions, /v1/chat/completions, and /v1/embeddings, allowing any standard OpenAI client library to interact with local GGUF models without code modifications.

The llama.cpp project includes a lightweight HTTP server that implements the OpenAI API contract, enabling developers to serve quantized LLMs through familiar REST interfaces. This implementation maps incoming OpenAI request fields directly to the internal llama_* inference API, making it compatible with existing tools like the OpenAI Python SDK, LangChain, and curl.

Architecture Overview

The server implementation resides in tools/server/ and consists of modular components that handle HTTP routing, request validation, and model inference.

Core Components

• server.cpp — Entry point located at tools/server/server.cpp that parses command-line arguments, loads the GGUF model, instantiates server_context, and starts the HTTP listener on the specified port.

• server-http.h and server-http.cpp — Implements the HTTP server using httplib, routing requests to /v1/completions, /v1/chat/completions, /v1/embeddings, and health-check endpoints. These files serialize responses into the exact JSON schema expected by OpenAI clients.

• server-context.h and server-context.cpp — Contains the server_context class that manages per-model state including the tokenizer, KV cache, and sampling parameters. This layer converts OpenAI parameters like max_tokens, temperature, and top_p into internal llama.cpp API calls and formats the output to include usage statistics and finish_reason fields.

• server-common.h and server-common.cpp — Shared utilities using nlohmann::json for parsing, request validation logic, and HTTP status code mapping for OpenAI-compatible error responses.

When a request arrives, server-http.cpp parses the JSON payload, validates required fields like model and prompt (or messages), delegates inference to server_context::run_completion or run_chat (which invoke llama_eval), then serializes the response into the OpenAI schema.

Building the Server

Before using the OpenAI-compatible endpoints, compile the server binary from the llama.cpp source code.

# Clone the repository

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# Configure build with server support enabled

cmake -B build -DLLAMA_SERVER=ON -DLLAMA_BUILD_EXAMPLES=ON

# Compile the server target

cmake --build build --target server

The compiled binary appears at build/bin/server. Enable GPU acceleration by adding -DLLAMA_CUDA=ON (for NVIDIA) or -DLLAMA_METAL=ON (for Apple Silicon) to the CMake configuration.

Starting the Server

Launch the server by specifying a GGUF model and context window size. The server loads the model into memory and begins listening for HTTP requests.

./build/bin/server \
    -m models/llama-2-7b-chat.Q4_0.gguf \
    -c 4096 \
    -ngl 32 \
    --port 8080

• -m — Path to the GGUF model file.
• -c — Context size in tokens (e.g., 4096 or 8192).
• -ngl — Number of GPU layers to offload for acceleration.
• --port — HTTP port (defaults to 8080).

Standard output confirms the server status:

[2024-03-01 12:00:00] server: listening on http://0.0.0.0:8080
[2024-03-01 12:00:00] model loaded: llama-2-7b-chat.Q4_0.gguf (4096 ctx)

OpenAI-Compatible Endpoints

The server implements three primary endpoints that mirror the OpenAI API specification.

Text Completions (/v1/completions)

Send a prompt to the model using the legacy completions endpoint. This maps to the run_completion method in server-context.cpp.

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "prompt": "Explain quantum entanglement in simple terms.",
        "max_tokens": 150,
        "temperature": 0.7,
        "stream": false
      }'

The response follows the OpenAI schema exactly:

{
  "id": "cmpl-01a2b3c4d5e6f7g8h9i0j",
  "object": "text_completion",
  "created": 1709283456,
  "model": "llama-2-7b-chat",
  "choices": [
    {
      "text": "Quantum entanglement is a phenomenon where...",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 147,
    "total_tokens": 159
  }
}

Chat Completions (/v1/chat/completions)

Use the chat endpoint for conversational interactions. The server processes the messages array through server_context::run_chat and applies the appropriate chat template.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the meaning of life?"}
        ],
        "max_tokens": 200,
        "temperature": 0.5,
        "stream": false
      }'

Response structure:

{
  "id": "chatcmpl-12345",
  "object": "chat.completion",
  "created": 1709283500,
  "model": "llama-2-7b-chat",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "The meaning of life is subjective..."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 24, "completion_tokens": 56, "total_tokens": 80}
}

Streaming Responses

Enable real-time token delivery by setting stream: true. The server uses Server-Sent Events (SSE) format, emitting partial JSON objects identical to OpenAI's streaming implementation.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
        "model": "llama-2-7b-chat",
        "messages": [{"role":"user","content":"Tell me a joke."}],
        "stream": true
      }'

Each line contains a delta object with incremental content, allowing UI clients to display text as it generates.

Embeddings (/v1/embeddings)

Generate vector embeddings for input text. The server returns an array of floating-point values compatible with OpenAI's embedding format.

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-2-7b-chat",
        "input": "Hello world"
      }'

Example response:

{
  "object": "list",
  "data": [
    {"object":"embedding","embedding":[0.12, -0.08, 0.05, ...],"index":0}
  ],
  "model": "llama-2-7b-chat",
  "usage": {"prompt_tokens":2,"total_tokens":2}
}

Configuring OpenAI Clients

Any client library that accepts a custom base_url can connect to the llama.cpp server. Set the base URL to http://localhost:8080 (or your host:port) and provide any string for the API key.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="llama-2-7b-chat",  # Model name is ignored by server but required

    messages=[{"role": "user", "content": "Hello!"}]
)

This configuration works because server-http.cpp implements the exact route structure and JSON schema that OpenAI clients expect.

Summary

• Build the server with -DLLAMA_SERVER=ON to enable OpenAI-compatible endpoints.
• Start the server using ./build/bin/server with flags -m for the model and -c for context size.
• Query /v1/completions for text generation and /v1/chat/completions for conversational AI.
• Stream responses by setting stream: true in your JSON payload.
• Connect any OpenAI SDK client by pointing base_url to the local server address.

Frequently Asked Questions

What endpoints does llama-server implement?

The server implements /v1/completions, /v1/chat/completions, /v1/embeddings, and health-check endpoints. These routes are defined in tools/server/server-http.cpp and return JSON structures matching the OpenAI API specification, including fields like id, object, created, model, choices, and usage.

Does llama-server support API authentication?

The server does not implement API key validation by default. Authentication must be handled via a reverse proxy (such as nginx) if required for production deployments. The server-common.cpp file handles request validation but focuses on parameter checking rather than authorization headers.

How do I enable GPU acceleration when running the server?

Add the -ngl (number of GPU layers) flag when starting the server: ./build/bin/server -m model.gguf -ngl 99. The maximum value depends on your GPU memory. For CUDA builds, ensure you compiled with -DLLAMA_CUDA=ON; for Apple Silicon, use -DLLAMA_METAL=ON.

Can I use the OpenAI Python SDK with llama-server?

Yes. Configure the OpenAI client with base_url="http://localhost:8080/v1" and any placeholder for api_key. The SDK will communicate with the llama.cpp server exactly as it would with OpenAI's official API, allowing you to use llama-server with OpenAI API compatible endpoints in existing applications without code changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}