How to Serve Embedding Models with llama.cpp and Use the Embedding Endpoint

Start the llama.cpp server with the --embedding flag and POST to /v1/embeddings to generate vector representations using models like BERT, Nomic Embed, or Gemma.

The llama.cpp server provides a high-performance HTTP interface for running embedding models locally. By enabling embedding mode, you can serve quantized GGUF versions of popular encoders and access them through both legacy and OpenAI-compatible endpoints. This guide covers the complete pipeline from CLI configuration to vector extraction, referencing the actual implementation in ggml-org/llama.cpp.

Enabling Embedding Mode on the Server

To activate embedding support, launch the server binary with the --embedding flag. This mode changes how the server initializes the model and validates batch configurations.

In tools/server/server.cpp (lines 78-84), the server enforces that the batch size (n_batch) equals the unified batch size (n_ubatch). Embeddings must be computed in a single ubatch to prevent the model from splitting prompts and producing incoherent vectors:

./server -m models/bert-bge-small.gguf --embedding --pooling_type mean --port 8080

Pooling Strategies

The --pooling_type parameter controls how token-level embeddings aggregate into a single sentence vector:

• last (default): Returns the normalized vector of the last token in the sequence.
• mean: Computes the average of all token embeddings.
• none: Returns one embedding per token without aggregation.

Note that none disables the OpenAI-compatible endpoint because the schema requires a single vector per input.

Available HTTP Endpoints and Routing

Once embedding mode is active, the server registers three HTTP routes in tools/server/server.cpp (lines 184-186):

Route	Compatibility
POST /embedding	Legacy (non-OpenAI)
POST /embeddings	Legacy (non-OpenAI)
POST /v1/embeddings	OpenAI-compatible

All routes dispatch to handle_embeddings_impl, but the OpenAI-compatible version (post_embeddings_oai) sets a flag that triggers format_embeddings_response_oaicompat for schema-compliant JSON formatting.

Processing Pipeline and Vector Extraction

When a request hits the embedding endpoint, handle_embeddings_impl creates a server_task with type SERVER_TASK_TYPE_EMBEDDING. After model evaluation, send_embedding (in tools/server/server-context.cpp, lines 1502-1542) extracts the vectors:

1. Allocates a server_task_result_embd container.
2. Determines output dimension via llama_model_n_embd_out.
3. Iterates over processed tokens (batch.n_tokens).
4. For each token in the current slot:
   • If no pooling (LLAMA_POOLING_TYPE_NONE): fetches per-token embedding with llama_get_embeddings_ith.
   • Otherwise: fetches pooled sequence embedding with llama_get_embeddings_seq.
5. Optionally L2-normalizes the vector via common_embd_normalize if --embd_normalize is enabled.
6. Pushes vectors into res->embedding and returns to the HTTP layer.

The function guarantees error-filled placeholder vectors if the model fails to produce embeddings, ensuring the HTTP layer receives a complete response.

Response Format and OpenAI Compatibility

The HTTP layer calls format_embeddings_response_oaicompat (in tools/server/server-common.cpp, lines 1625-1668) to construct the JSON payload:

{
  "model": "text-embedding-3-small",
  "object": "list",
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 5
  },
  "data": [
    {
      "index": 0,
      "object": "embedding",
      "embedding": [0.0123, -0.0456, ...]
    }
  ]
}

If the request specifies encoding_format: "base64", the embedding array is base64-encoded to reduce JSON payload size. Otherwise, raw float arrays are emitted.

Practical Usage Examples

Legacy Endpoint with cURL

curl -X POST http://localhost:8080/embeddings \
     -H "Content-Type: application/json" \
     -d '{"input":"The quick brown fox"}'

OpenAI-Compatible Endpoint with Base64

curl -X POST http://localhost:8080/v1/embeddings \
     -H "Content-Type: application/json" \
     -d '{
       "model":"text-embedding-3-small",
       "input":["Hello world","Another sentence"],
       "encoding_format":"base64"
     }'

Python with OpenAI Client

from openai import OpenAI

client = OpenAI(api_key="dummy",
                base_url="http://127.0.0.1:8080/v1")

resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=["Llama cpp embeddings", "Fast inference"]
)

print([len(r.embedding) for r in resp.data])  # [vector_dim, vector_dim]

Async Batch Processing

The repository includes examples/server_embd.py, which demonstrates async batch requests to the legacy /embedding endpoint and computes cosine similarity matrices between results.

Error Handling and Constraints

The embedding endpoint enforces several constraints to ensure valid vector generation:

• Batch size validation: If n_batch does not equal n_ubatch, the server exits with an error because embeddings require single-batch processing.
• Context window limits: Prompts exceeding the model's context window return HTTP 400 with a "too large" error message.
• Disabled embeddings: Requests to /embedding* when the server started without --embedding return a 404-style error stating "This server does not support embeddings".
• Pooling incompatibility: Using pooling_type: none with /v1/embeddings returns HTTP 400 because the OpenAI schema requires one vector per input, not per-token vectors.

Summary

• Start the server with --embedding to enable vector generation, ensuring n_batch equals n_ubatch for coherent results.
• Choose --pooling_type (last, mean, or none) to control how token embeddings aggregate into sentence vectors.
• Access embeddings via /v1/embeddings for OpenAI compatibility or legacy /embeddings for simpler JSON.
• Use send_embedding in server-context.cpp to extract vectors via llama_get_embeddings_seq or llama_get_embeddings_ith, with optional L2 normalization.
• Handle errors for context limits, missing flags, and pooling mismatches to ensure robust production deployments.

Frequently Asked Questions

What models work with the llama.cpp embedding endpoint?

The server supports any GGUF-converted embedding model, including BERT variants (BGE, MiniLM), Nomic Embed, and Gemma embedding checkpoints. The model must be loaded with a pooling type compatible with your use case—last or mean for sentence embeddings, none for token-level analysis.

Why does the server require --embedding instead of auto-detecting the model type?

The --embedding flag triggers specific validation logic in server.cpp that ensures n_batch equals n_ubatch. Embeddings must be computed in a single unified batch to prevent sequence splitting, which would produce incoherent vectors. This flag also registers the /embedding* HTTP routes and initializes the embedding-specific task queue handlers.

Can I use the OpenAI Python client with llama.cpp embeddings?

Yes. Point the OpenAI client to your local server by setting base_url="http://127.0.0.1:8080/v1" and using a dummy API key. The /v1/embeddings endpoint accepts the same schema as OpenAI's embedding API, including input arrays, encoding_format: "base64", and model parameters. Note that pooling_type: none is incompatible with this endpoint and returns HTTP 400.

What is the difference between pooling types and when should I use each?

last returns the embedding of the final token in the sequence, suitable for causal language models adapted for embeddings. mean averages all token embeddings, generally preferred for bidirectional encoders like BERT. none returns per-token embeddings without aggregation, useful for token classification or analysis tasks, but incompatible with the OpenAI-compatible endpoint which expects one vector per input string.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}