Best Practices for Optimizing Knowledge Base Retrieval Performance in AstrBot

AstrBot achieves optimal knowledge base retrieval performance through a four-stage hybrid pipeline that combines dense vector search, BM25 sparse retrieval, reciprocal rank fusion, and optional LLM reranking, with tunable parameters for chunk sizing, top-k thresholds, and index configuration.

Optimizing knowledge base retrieval performance in AstrBot requires understanding its hybrid architecture that leverages both dense embeddings and sparse keyword matching. The system stores each knowledge base as chunked documents indexed in FAISS vectors and BM25 term frequencies, coordinated by the RetrievalManager in astrbot/core/knowledge_base/retrieval/manager.py. By tuning chunking strategies, retrieval parameters, and index types, you can balance retrieval accuracy against latency for your specific use case.

Understanding AstrBot's Four-Stage Retrieval Pipeline

The retrieval process in astrbot/core/knowledge_base/retrieval/manager.py executes four distinct stages, each configurable via parameters stored in the knowledge base models at astrbot/core/knowledge_base/models.py:

1. Dense Retrieval (_dense_retrieve): Queries the per-KB FAISS vector database using cosine similarity. Controlled by top_k_dense (default 50).
2. Sparse Retrieval (astrbot/core/knowledge_base/retrieval/sparse_retriever.py): Executes BM25 keyword search on raw chunk text. Controlled by top_k_sparse (default 50).
3. Rank Fusion (astrbot/core/knowledge_base/retrieval/rank_fusion.py): Merges dense and sparse results using Reciprocal Rank Fusion (RRF). Controlled by top_k_fusion (default 20).
4. Reranking (_rerank): Optional LLM-based reranking if rerank_provider_id is configured. Returns top_m_final chunks (default 5).

Optimizing Chunk Size and Overlap

The RecursiveCharacterChunker in astrbot/core/knowledge_base/chunking/recursive.py determines how documents are segmented before indexing. Chunk sizing directly impacts vector database size and retrieval granularity.

Best practices for chunk configuration:

• Long-form prose: Use chunk_size=512 with chunk_overlap=50 to maintain context across boundaries while keeping vector counts manageable.
• Short FAQ-style content: Reduce chunk_size to 256-300 to minimize vector count and improve exact match retrieval.
• Code documentation: Use larger chunks (768-1024) with minimal overlap (20-30) to preserve function context.

Configure chunking when creating a knowledge base via kb_manager.create_kb in astrbot/core/knowledge_base/kb_mgr.py:

kb_helper = await kb_manager.create_kb(
    kb_name="TechnicalDocs",
    embedding_provider_id="openai-embeddings",
    chunk_size=256,
    chunk_overlap=30,
)

Tuning Top-K Parameters for Latency vs. Recall

The top_k parameters control the candidate pool size at each retrieval stage. Adjust these in kb_manager.update_kb to optimize for your latency budget:

• top_k_dense and top_k_sparse: Increase (e.g., to 100) for higher recall on ambiguous queries; decrease (e.g., to 20) for strict latency requirements.
• top_k_fusion: Controls how many fused candidates proceed to reranking. Values between 10-30 typically provide the best precision-latency balance.
• top_m_final: The actual chunks returned to users. Keep this ≤ 5 unless the downstream LLM context window supports large inputs.

Example configuration update:

await kb_manager.update_kb(
    kb_id=kb_helper.kb.kb_id,
    kb_name="TechnicalDocs",
    top_k_dense=30,
    top_k_sparse=30,
    top_k_fusion=15,
    top_m_final=3,
    rerank_provider_id=None,  # Disable reranker for speed

)

Configuring BM25 Stop-Words for Sparse Retrieval

The sparse retriever in astrbot/core/knowledge_base/retrieval/sparse_retriever.py loads stop-words from hit_stopwords.txt to exclude high-frequency, low-information terms from BM25 scoring.

Optimization strategies:

• Add domain-specific noise terms (e.g., "example", "demo", "test" in documentation KBs) to reduce false positives.
• Keep the stop-word list under 200 entries to maintain scoring performance.
• Modify the file at runtime, then restart the AstrBot process to reload the module.

Example stop-word extension:

from pathlib import Path

STOPWORDS_PATH = Path("astrbot/core/knowledge_base/retrieval/hit_stopwords.txt")
custom_stopwords = {"example", "demo", "test", "placeholder"}

with STOPWORDS_PATH.open("a", encoding="utf-8") as f:
    for word in custom_stopwords:
        f.write(f"{word}\n")

Selecting Vector Index Types: Flat vs. IVF

AstrBot uses FaissVecDB in astrbot/core/db/vec_db/faiss_impl.py for vector storage, defaulting to a flat index that loads entirely into RAM. For knowledge bases exceeding 100,000 chunks, consider migrating to an IVF (Inverted File) index to reduce memory footprint and query latency.

Index selection guidelines:

• Flat Index (IndexFlatIP or IndexFlatL2): Best for KBs under 50k chunks; provides exact nearest neighbor search.
• IVF Index (IndexIVFFlat): Partition vectors into nlist clusters; queries only the nearest clusters. Use nlist=4096 for 100k+ chunks, adjusting nprobe (clusters to search) for recall/latency trade-offs.

Note: Switching index types requires rebuilding the FAISS index from stored embeddings in astrbot/core/db/vec_db/faiss_impl.py.

Managing Embedding Providers and Rerankers

Provider configuration in astrbot/core/provider/manager.py significantly impacts retrieval quality and speed.

Embedding provider selection:

• OpenAI text-embedding-3-small: 1536 dimensions, lower latency, suitable for general-purpose KBs.
• OpenAI text-embedding-3-large: 3072 dimensions, higher quality, increased latency and storage cost.
• Local models: Use for air-gapped environments, but verify batch processing support in astrbot/core/provider/manager.py.

Reranker optimization:

• Enable reranking only when rerank_provider_id supports the rerank method (e.g., Cohere or OpenAI rerankers).
• Reranking adds 100-500ms latency per query due to additional LLM calls.
• Best practice: Disable reranking (rerank_provider_id=None) for latency-sensitive applications; enable only when precision is critical and top_m_final ≤ 5.

Monitoring Retrieval Performance

AstrBot logs detailed timing metrics via logger.debug in RetrievalManager.retrieve within astrbot/core/knowledge_base/retrieval/manager.py.

Key metrics to monitor:

Dense retrieval across 2 bases took 0.12s and returned 84 results.
Sparse retrieval across 2 bases took 0.04s and returned 100 results.
Rank fusion took 0.01s and returned 20 results.

Optimization workflow:

1. Enable debug logging to capture stage-specific timings.
2. Identify the bottleneck (typically dense retrieval for large KBs or reranking when enabled).
3. Adjust the corresponding top_k parameters or index type.
4. Target a combined retrieval time under 300ms for interactive chat applications.

Summary

• AstrBot's retrieval pipeline combines dense FAISS search, BM25 sparse retrieval, RRF fusion, and optional LLM reranking through astrbot/core/knowledge_base/retrieval/manager.py.
• Chunk sizing directly impacts vector count and retrieval granularity; use 256-512 tokens depending on content type via astrbot/core/knowledge_base/chunking/recursive.py.
• Top-K tuning balances latency and recall: reduce top_k_dense, top_k_sparse, and top_k_fusion for speed, or increase them for comprehensive recall.
• BM25 optimization requires maintaining a concise stop-word list in hit_stopwords.txt loaded by astrbot/core/knowledge_base/retrieval/sparse_retriever.py.
• Index selection matters for scale: use flat FAISS indices for small KBs and consider IVF indices for 100k+ chunks in astrbot/core/db/vec_db/faiss_impl.py.
• Reranker usage adds significant latency; disable rerank_provider_id unless precision is critical.

Frequently Asked Questions

How do I reduce latency for real-time chat applications using AstrBot's knowledge base?

To reduce latency, decrease the candidate pool sizes by setting top_k_dense and top_k_sparse to 20-30 instead of the default 50, and reduce top_k_fusion to 10-15. Disable the reranker by setting rerank_provider_id=None in astrbot/core/knowledge_base/models.py, as reranking adds 100-500ms per query. Finally, ensure your FAISS index fits in RAM to avoid disk swapping.

What is the optimal chunk size for technical documentation in AstrBot?

For technical documentation, use a chunk_size of 512 tokens with chunk_overlap of 50 tokens via the RecursiveCharacterChunker in astrbot/core/knowledge_base/chunking/recursive.py. This preserves code context and explanatory paragraphs while keeping vector database size manageable. For FAQ-style content with short answers, reduce chunk_size to 256-300 tokens to minimize vector count and improve exact-match retrieval.

When should I use an IVF index instead of the default flat FAISS index in AstrBot?

Switch to an IVF (Inverted File) index in astrbot/core/db/vec_db/faiss_impl.py when your knowledge base exceeds 100,000 chunks or when memory constraints prevent loading the entire flat index into RAM. IVF indices partition vectors into clusters (e.g., nlist=4096), querying only the nearest clusters to reduce both memory footprint and search latency. Note that IVF requires rebuilding the index from stored embeddings and may require tuning nprobe (clusters to search) to balance recall versus speed.

How does the reciprocal rank fusion (RRF) work in AstrBot's retrieval pipeline?

Reciprocal Rank Fusion in astrbot/core/knowledge_base/retrieval/rank_fusion.py combines results from dense FAISS retrieval and sparse BM25 retrieval without requiring score calibration between the different modalities. RRF assigns each document a fused score based on the sum of reciprocal ranks (1/(k + rank)) from each retrieval method, where k is a constant (typically 60). This ensures that documents appearing highly ranked in both dense and sparse results surface to the top, while the top_k_fusion parameter (default 20) controls how many fused candidates proceed to optional reranking or final output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}