# Troubleshooting Poor RAG Retrieval Results in PrivateGPT: A Complete Guide

> Troubleshoot poor RAG retrieval in PrivateGPT. Learn to fix mismatched embedding models, optimize similarity_top_k, manage doc_id metadata, and re-ingest vector indices for better results.

- Repository: [Zylon/private-gpt](https://github.com/zylon-ai/private-gpt)
- Tags: how-to-guide
- Published: 2026-03-06

---

**Poor RAG retrieval in PrivateGPT is almost always caused by mismatched embedding models, incorrect `similarity_top_k` values, missing `doc_id` metadata, or stale vector indices that require re-ingestion.**

PrivateGPT retrieves relevant document chunks from a vector store and feeds them to the LLM for context-aware answers. When this retrieval fails—returning irrelevant, missing, or too few chunks—the generated responses suffer. This guide walks through systematic troubleshooting steps based on the actual source code in the `zylon-ai/private-gpt` repository.

## Understanding the RAG Pipeline Architecture

PrivateGPT's retrieval pipeline consists of three integrated components that must align perfectly.

**Embedding & Ingestion** transforms raw files into chunked `Document` objects and stores their embeddings. This logic lives in [`private_gpt/components/ingest/ingest_component.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/components/ingest/ingest_component.py), where the `IngestionHelper.transform_file_into_documents` method ensures each chunk carries a `doc_id` metadata field required for filtering.

**Vector Store** persists embeddings and provides fast nearest-neighbour search. The `VectorStoreComponent` class in [`private_gpt/components/vector_store/vector_store_component.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/components/vector_store/vector_store_component.py) initializes the backend (Chroma, Qdrant, Postgres, Milvus, or ClickHouse) and returns a `VectorIndexRetriever`.

**RAG Settings** control retrieval behavior through the `RagSettings` model in [`private_gpt/settings/settings.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/settings/settings.py), specifically the `similarity_top_k` (default: `2`) and `similarity_value` parameters.

## 10 Critical Troubleshooting Steps

Follow this systematic checklist to diagnose why your retrieval results are poor.

### 1. Verify Vector Store Connectivity

Ensure the backend database is reachable and client libraries are installed. The `VectorStoreComponent.__init__` constructor explicitly raises an `ImportError` if required backend libraries are missing.

- Re-install the specific extra dependency: `poetry install --extras vector-stores-chroma` (or `vector-stores-qdrant`, etc.)
- Confirm the database process is running (e.g., `curl http://localhost:6333` for Qdrant).

### 2. Enforce Embedding Model Consistency

The embedding model used during **ingestion** must match the model used at **query-time** exactly, including output dimensions. Both stages read from [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml) under the `embedding.model` key.

**Critical**: After changing the embedding model in settings, you **must re-ingest** all documents. Mismatched dimensions cause silent retrieval failures or empty result sets.

### 3. Validate Document Metadata (doc_id)

Each chunk must contain a `doc_id` metadata field for context-filtering to work. The `IngestComponent` adds this automatically via `IngestionHelper.transform_file_into_documents`.

Verify the field exists in stored vectors by inspecting the database directly or querying the store's client. Missing `doc_id` fields cause `ContextFilter` operations to return empty results.

### 4. Adjust the similarity_top_k Parameter

The `similarity_top_k` setting (found in `RagSettings` at lines 398-401 of [`private_gpt/settings/settings.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/settings/settings.py)) controls how many nearest neighbours are returned. The default value is `2`.

- **Too low** (`2` or `3`): Missing relevant context, causing hallucinations.
- **Too high** (`10+`): Noisy context dilutes the LLM's focus.

Adjust via [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml) or at runtime:

```python
from private_gpt.settings.settings import Settings

settings = Settings.load()
settings.rag.similarity_top_k = 5  # Increase for broader context

```

### 5. Re-index After Data Changes

Adding, deleting, or updating files does **not** automatically refresh the vector store. You must explicitly call the ingestion methods in [`private_gpt/components/ingest/ingest_component.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/components/ingest/ingest_component.py).

For single files:

```python
ingestion_component = get_ingestion_component(settings)
ingestion_component.ingest(file_path)

```

For bulk updates, use `bulk_ingest([...])`, then call `vector_store_component.close()` to flush pending writes.

### 6. Check the similarity_value Threshold

The optional `similarity_value` parameter (0-1 range) filters out results below a similarity score. If set too high (e.g., `0.9`), you may get zero results for legitimate queries.

Check `settings.rag.similarity_value` in your configuration. Remove or lower the value (e.g., `0.3`) to retrieve more hits.

### 7. Audit Context Filter Usage

When passing a `ContextFilter` with `docs_ids` to `VectorStoreComponent.get_retriever`, the retriever restricts search to only those IDs. If the IDs are stale or incorrect, the result list will be empty.

Verify that `docs_ids` correspond to current document IDs in the vector store before applying the filter.

### 8. Inspect Persistent Storage Location

The index persists under `local_data_path` (defined in [`private_gpt/paths.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/paths.py)). Corruption or permission issues cause silent failures.

Verify the path exists and is writable. To force a fresh rebuild (after backing up), delete the folder and re-ingest all documents.

### 9. Resolve Backend-Specific Collection Conflicts

Some stores like Qdrant and Chroma use default collection names such as `"make_this_parameterizable_per_api_call"`. If you run multiple API instances simultaneously without unique collection names, queries may hit the wrong index.

Override the collection name during `VectorStoreComponent` initialization for each instance.

### 10. Enable Debug Logging

Set `LOG_LEVEL=DEBUG` in [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml) or export `PYTHONLOGGING=debug`. The logs from `VectorStoreComponent.get_retriever` reveal the final `similarity_top_k`, applied `doc_ids`, and filter objects, showing exactly which retrieval branch executes.

## Practical Code Examples

### Adjusting Retrieval Parameters at Runtime

Modify RAG settings programmatically when you need dynamic control over retrieval:

```python
from private_gpt.settings.settings import Settings
from private_gpt.components.vector_store.vector_store_component import VectorStoreComponent
from private_gpt.server.chat.chat_service import ChatService

# Load and adjust settings

settings = Settings.load()
settings.rag.similarity_top_k = 5
settings.rag.similarity_value = 0.3

# Re-initialize components with new config

vector_store = VectorStoreComponent(settings)
retriever = vector_store.get_retriever(index=existing_index)

# Use in chat service

chat = ChatService(settings=settings, retriever=retriever)
response = chat.ask("Explain the architecture of PrivateGPT")

```

### Re-ingesting After Changing the Embedding Model

After updating `embedding.model` in [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml), rebuild the entire index:

```bash

# Update settings.yaml first:

# embedding:

#   model: "sentence-transformers/all-MiniLM-L6-v2"

# Then run ingestion

python -m private_gpt.scripts.ingest_folder /path/to/your/docs

```

The `get_ingestion_component` function selects the appropriate ingest class (e.g., `BatchIngestComponent`) based on `settings.embedding.ingest_mode` and rebuilds the vector index under `local_data_path`.

### Verifying Document IDs in Chroma

Confirm your documents carry the correct metadata:

```python
from private_gpt.components.vector_store.vector_store_component import VectorStoreComponent
from private_gpt.settings.settings import Settings

settings = Settings.load()
vs = VectorStoreComponent(settings)

# Access underlying Chroma collection

collection = vs.vector_store._client.get_collection("make_this_parameterizable_per_api_call")
stored_ids = collection.get()["ids"]
print(f"Stored document IDs: {stored_ids}")

```

If the printed IDs do not match your `ContextFilter` requirements, re-run ingestion for those specific files.

## Summary

- **Vector store connectivity**: Ensure backend libraries are installed and the database process is running.
- **Embedding consistency**: Never mix models between ingestion and query; always re-ingest after model changes.
- **Metadata integrity**: Verify `doc_id` fields exist in所有 stored chunks for filtering to work.
- **Retrieval tuning**: Increase `similarity_top_k` from the default `2` if context is missing; lower or remove `similarity_value` if results are too sparse.
- **Explicit re-indexing**: Call `ingest()` or `bulk_ingest()` after any data changes, then flush with `vector_store_component.close()`.
- **Storage hygiene**: Check `local_data_path` permissions and use debug logging to trace retrieval execution paths.

## Frequently Asked Questions

### Why is PrivateGPT returning "I don't know" even when the answer is in my documents?

This usually indicates a **retrieval gap**. Check that `similarity_top_k` is not set too low (default is `2`), verify the `similarity_value` threshold is not filtering out valid results, and ensure you have re-ingested documents after any embedding model changes. Enable debug logging to confirm chunks are being fetched from `VectorStoreComponent`.

### How do I fix mismatched embedding dimensions errors?

Mismatched dimensions occur when you query using a different embedding model than the one used during ingestion. Both phases read from [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml)'s `embedding.model` field. To fix: (1) Set the correct model in settings, (2) Delete the existing index in `local_data_path`, and (3) Re-ingest all documents using `python -m private_gpt.scripts.ingest_folder`.

### What is the optimal value for similarity_top_k?

The default `similarity_top_k` of `2` works for focused queries but often fails for complex questions requiring broader context. According to the `RagSettings` implementation, values between `5` and `7` typically provide sufficient context without introducing noise. Adjust based on your document chunk size and query complexity, testing incrementally.

### Why are my ContextFilter queries returning empty results?

Empty results when using `ContextFilter` indicate the `docs_ids` parameter contains stale or incorrect identifiers. Verify that the IDs passed match the `doc_id` metadata stored in your vector store. Use the Chroma verification snippet above to list valid IDs, and ensure `IngestionHelper.transform_file_into_documents` successfully added the metadata during ingestion.