How to Implement Reranking Models for Search Applications with llama.cpp

llama.cpp provides a built-in reranking pipeline that exposes HTTP endpoints to score document relevance using classification-head models like Jina or BGE-reranker.

The llama.cpp inference engine now supports reranking models for search applications, allowing you to compute query-document relevance scores within the same binary used for embeddings and text generation. By configuring the pooling type and enabling specific server flags, you can deploy cross-encoder models that return float scores indicating how well each candidate document matches a search query.

Enable the Reranking Endpoint

To activate the reranking functionality, start the server with four specific arguments. The --reranking flag (aliased as --rerank) toggles the server_reranking boolean parsed in common/arg.cpp, while --pooling rank sets the internal pooling type to LLAMA_POOLING_TYPE_RANK.

./server \
  --model /path/to/ggml-bge-reranker-v2-m3-f16.gguf \
  --embedding \
  --pooling rank \
  --reranking

This configuration registers the POST routes /rerank, /v1/rerank, and /v1/reranking in tools/server/server.cpp. If you omit the --reranking flag, the handlers return an error referencing the missing argument.

Architecture of the Reranking Pipeline

The implementation spans three tightly coupled layers that handle model definition, inference, and HTTP serving:

• Model Definition: The header include/llama.h defines LLAMA_POOLING_TYPE_RANK with value 4, instructing the compute graph to preserve the classification head output instead of mean or CLS pooling.
• Inference Graph: During the forward pass in src/llama-context.cpp, the engine checks cparams.pooling_type == LLAMA_POOLING_TYPE_RANK and extracts the rerank score from the cls_out tensor, producing one float per input sequence.
• Server Integration: tools/server/server-context.cpp orchestrates prompt formatting, batch inference, and JSON response construction, while tools/server/server-common.cpp provides the formatting utilities.

Construct Reranking Prompts with Chat Templates

Reranking models require specific query-document formatting. The server calls format_prompt_rerank in tools/server/server-common.cpp to retrieve a chat template named "rerank" via llama_model_chat_template. When available, the template structures input as:

<BOS> query <EOS> <SEP> document_i <EOS>

If the model lacks a custom rerank template, the code falls back to basic concatenation of the query and document strings before tokenization.

Extracting Relevance Scores

After tokenization and the forward pass through llama_decode, the engine extracts the relevance score from the classification head. In src/llama-context.cpp, the logic specifically handles the LLAMA_POOLING_TYPE_RANK case by reading the cls_out tensor and copying the float value into the result structure. This produces a single scalar score per query-document pair, where higher values indicate stronger relevance.

Query the Reranking API

With the server running, send a POST request to /v1/rerank with a JSON payload containing the query, a list of documents, and an optional top_n parameter to limit results.

curl http://127.0.0.1:8012/v1/rerank \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
        "query": "What is the capital of France?",
        "documents": [
          "Paris is the capital of France.",
          "Berlin is the capital of Germany."
        ],
        "top_n": 2
      }'

The format_response_rerank function in tools/server/server-common.cpp returns a JSON object:

{
  "scores": [0.842, 0.123],
  "documents": ["Paris is the capital of France.", "Berlin is the capital of Germany."]
}

The scores array aligns index-wise with the input documents array, allowing you to sort candidates by descending relevance.

Complete Implementation Examples

Starting the Server with a Jina Reranker

Download a compatible GGUF model and launch the server:

# Download model (example using Jina tiny)

./scripts/download-model.sh jina-reranker-v1-tiny-en

# Launch with reranking enabled

./server \
  --model ./models/jina-reranker-v1-tiny-en/ggml-model-f16.gguf \
  --embedding \
  --pooling rank \
  --reranking \
  --port 8012

Python Client for Search Applications

Use the requests library to integrate reranking into your retrieval pipeline:

import requests

def rerank_documents(query: str, documents: list[str], top_n: int = 5):
    payload = {
        "query": query,
        "documents": documents,
        "top_n": top_n
    }
    
    resp = requests.post(
        "http://127.0.0.1:8012/v1/rerank",
        json=payload,
        timeout=30
    )
    resp.raise_for_status()
    return resp.json()  # {"scores": [...], "documents": [...]}

# Example usage

results = rerank_documents(
    "Explain transformer attention mechanisms",
    [
        "Transformers use self-attention to weight token importance globally.",
        "Convolutional neural networks process data with local receptive fields.",
        "Recurrent networks maintain hidden states across sequence steps."
    ],
    top_n=2
)

# Sort by score

scored = list(zip(results["scores"], results["documents"]))
scored.sort(reverse=True)
for score, doc in scored:
    print(f"Score: {score:.3f} | {doc}")

Low-Level C++ Integration

For embedding the reranker directly in a C++ application without the HTTP server:

// Load model with rank pooling
llama_model_params mparams = llama_model_default_params();
llama_model * model = llama_load_model_from_file(
    "ggml-bge-reranker-v2-m3-f16.gguf", 
    mparams
);

llama_context_params cparams = llama_context_default_params();
cparams.pooling_type = LLAMA_POOLING_TYPE_RANK;  // Value 4 from llama.h L174
llama_context * ctx = llama_new_context_with_model(model, cparams);

// Format prompt using rerank template
const char * tmpl = llama_model_chat_template(model, "rerank");
std::string prompt = fmt::format("{} {} {}", 
    query, 
    separator_token, 
    document_text
);

// Tokenize and decode
auto tokens = llama_tokenize(ctx, prompt, true);
llama_batch batch = llama_batch_get_one(tokens.data(), tokens.size());
llama_decode(ctx, batch);

// Extract score (simplified; actual extraction in llama-context.cpp L1250-1676)
float relevance_score = ctx->logits[0];

Summary

• llama.cpp implements reranking through the LLAMA_POOLING_TYPE_RANK pooling type defined in include/llama.h, which preserves the classification head output.
• Enable the HTTP endpoints by starting the server with --embedding --pooling rank --reranking, parsed in common/arg.cpp.
• Relevance scores are extracted from the cls_out tensor during the forward pass in src/llama-context.cpp.
• The server exposes /v1/rerank and aliases, handling request formatting in tools/server/server-context.cpp and response building in tools/server/server-common.cpp.
• Compatible models include Jina-reranker variants and BGE-reranker-v2-m3 converted to GGUF format.

Frequently Asked Questions

What model architectures does llama.cpp reranking support?

llama.cpp supports encoder-only transformer models with classification heads, such as Jina-reranker-v1-tiny-en, BGE-reranker-v2-m3, or similar cross-encoders converted to GGUF. The model must expose a classification output tensor (cls_out) that the engine extracts when the pooling type is set to LLAMA_POOLING_TYPE_RANK (value 4), as defined in include/llama.h.

How does the server format prompts for reranking models?

The server uses format_prompt_rerank in tools/server/server-common.cpp to retrieve a chat template named "rerank" via llama_model_chat_template. If the model defines this template, it structures the input as <BOS> query <EOS> <SEP> document <EOS>; otherwise, it falls back to concatenating the query and document text with basic separators.

Can I access reranking scores without running the HTTP server?

Yes. You can programmatically extract scores using the C++ API by loading the model with LLAMA_POOLING_TYPE_RANK, formatting prompts with the "rerank" chat template via llama_model_chat_template, and accessing the output tensor after llama_decode. The score extraction logic resides in src/llama-context.cpp, mirroring the implementation used by the server endpoint.

Why does the rerank endpoint return an error about missing flags?

If the endpoint returns an error referencing the --reranking flag, the server instance was not started with this CLI argument. The route handlers in tools/server/server.cpp check this flag before processing requests, and the server returns a specific error message if the feature is disabled, as implemented in tools/server/server-context.cpp.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}