Optimizing Embedding Performance When Dealing with Large Codebases in DeepWiki

DeepWiki uses batch-size tuning, token-based file filtering, and provider-specific caching to process massive repositories without hitting API limits or memory constraints.

When working with the AsyncFuncAI/deepwiki-open repository, optimizing embedding performance becomes critical as codebase size grows. DeepWiki processes every source file as a separate document and generates vector embeddings using providers like OpenAI, Google, Ollama, or Bedrock. Without optimization, large files or high document counts trigger slow processing, memory exhaustion, or provider rate limits. The following strategies, implemented directly in the source code, allow you to maintain high throughput while handling enterprise-scale repositories.

Built-in Strategies for Optimizing Embedding Performance

Tune Batch Sizes Per Provider

DeepWiki configures provider-specific batch sizes to maximize throughput without overwhelming API endpoints. In api/tools/embedder.py, the get_embedder factory constructs the embedding client, while api/data_pipeline.py reads the batch configuration and instantiates the ToEmbeddings component.

The default configurations are:

• OpenAI: 500 documents per batch
• Google/Bedrock: 100 documents per batch
• Ollama: Single-document streaming mode (no batching)

You can override these values in configs["embedder_*"]["batch_size"] to match your specific rate limits and latency requirements.

Skip Oversized Files with Token Limits

To prevent API rejections and wasted compute, DeepWiki implements token-based filtering in api/data_pipeline.py. The count_tokens() function uses tiktoken to estimate document size with provider-specific encodings (OpenAI's exact model encoding or cl100k for others).

The system applies these thresholds:

• Code files: Skipped if exceeding MAX_EMBEDDING_TOKENS * 10 (81,920 tokens for the default 8,192 limit)
• Documentation files: Skipped if exceeding MAX_EMBEDDING_TOKENS (8,192 tokens)

This prevents embedding failures on generated artifacts or minified bundles while preserving standard source files.

Leverage Embedding Caching

For batch-oriented providers like DashScope, DeepWiki implements persistent caching in api/dashscope_client.py. The DashScopeBatchEmbedder class stores full lists of EmbedderOutput objects as pickle files in ./embedding_cache/.

The cache key combines the embedding model name and a user-specified identifier:

self.cache_path = f"./embedding_cache/{embedding_cache_file_name}_{self.__class__.__name__}_{embedder.model}.pkl"

Subsequent pipeline runs load embeddings from disk instead of re-calling the remote API, dramatically reducing processing time for incremental updates or repeated analysis of the same repository.

Filter by File Type and Directory

The read_all_documents() function in api/data_pipeline.py supports granular inclusion and exclusion filters to reduce document volume before embedding:

• included_dirs: Process only specific subdirectories (e.g., ["src", "lib"])
• excluded_dirs: Skip test directories, build artifacts, or dependencies (e.g., ["node_modules", "tests"])
• included_files/excluded_files: Pattern-based file filtering

The internal should_process_file() helper implements this logic, ensuring only relevant source files consume embedding budget.

Use Provider-Specific Processing Modes

DeepWiki adapts its processing strategy based on embedder capabilities. In api/data_pipeline.py, the prepare_data_pipeline() function checks the embedder type and instantiates the appropriate transformer:

• Standard embedders (OpenAI, Google, Bedrock): Use ToEmbeddings with configurable batch sizes
• Ollama: Uses OllamaDocumentProcessor from api/ollama_patch.py for single-document streaming

For large codebases where local inference is viable, switching to Ollama eliminates batch overhead and network latency entirely.

Practical Code Examples for DeepWiki Optimization

Example 1: Configuring Batch Size for Google Embedder

Increase throughput for the Google embedder by adjusting the batch size in your configuration:

from api.config import configs
from api.data_pipeline import prepare_data_pipeline

# Override default batch size (100) for higher throughput

configs["embedder_google"]["batch_size"] = 200

# Build pipeline with optimized settings

pipeline = prepare_data_pipeline(embedder_type="google")

Example 2: Selective Directory Processing with Token Limits

Process only production code while skipping oversized generated files:

from api.data_pipeline import read_all_documents

# Focus on src directory, exclude tests, apply token filtering automatically

docs = read_all_documents(
    path="/path/to/repo",
    included_dirs=["src"],
    excluded_dirs=["src/tests", "src/__pycache__"],
    embedder_type="openai"
)

Example 3: Reusing DashScope Embeddings Cache

Avoid redundant API calls by leveraging the built-in cache mechanism:

from api.dashscope_client import DashScopeBatchEmbedder
from api.tools.embedder import get_embedder

embedder = get_embedder(embedder_type="dashscope")
batch_processor = DashScopeBatchEmbedder(
    embedder=embedder,
    batch_size=25,  # DashScope hard limit

    embedding_cache_file_name="my_large_repo"
)

# First run persists to ./embedding_cache/

vectors = batch_processor(["def example(): pass", "class Example: pass"])

# Second run loads from cache instantly

vectors_cached = batch_processor(["def example(): pass", "class Example: pass"])

Example 4: Switching to Ollama for Local Streaming

Eliminate network overhead for large repositories using local Ollama inference:

from api.data_pipeline import prepare_data_pipeline

# Uses OllamaDocumentProcessor for single-document streaming

pipeline = prepare_data_pipeline(embedder_type="ollama")

Key Configuration Files and Components

Understanding the codebase structure helps you locate optimization parameters:

File	Role
api/tools/embedder.py	Factory that constructs adal.Embedder instances for each provider.
api/config.py	Loads embedder.json and exposes configuration helpers like get_embedder_type.
api/config/embedder.json	Central configuration for provider-specific options including batch_size and model_kwargs.
api/data_pipeline.py	Orchestrates file discovery, token filtering via count_tokens(), and pipeline construction with prepare_data_pipeline().
api/dashscope_client.py	Implements DashScopeBatchEmbedder with persistent pickle-based caching in ./embedding_cache/.
api/ollama_patch.py	Provides OllamaDocumentProcessor for single-document streaming mode.

Summary

Optimizing embedding performance when dealing with large codebases in DeepWiki requires combining multiple built-in strategies:

• Tune batch sizes per provider in api/config/embedder.json to match API rate limits and maximize throughput.
• Filter oversized files automatically using token-based thresholds that skip code files exceeding 81,920 tokens and docs exceeding 8,192 tokens.
• Enable embedding caching for batch providers like DashScope to avoid redundant API calls across pipeline runs.
• Narrow document scope using included_dirs, excluded_dirs, and file pattern filters in read_all_documents().
• Select provider-specific modes such as Ollama's single-document streaming to eliminate batch overhead for local inference.

Frequently Asked Questions

What is the default batch size for OpenAI embeddings in DeepWiki?

The default batch size for OpenAI embeddings is 500 documents per request. This value is defined in the configuration and passed to the ToEmbeddings component in api/data_pipeline.py. You can override this in api/config/embedder.json if your API rate limits or latency requirements differ.

How does DeepWiki handle files that exceed token limits?

DeepWiki uses tiktoken to estimate token counts before embedding. In api/data_pipeline.py, the system skips code files exceeding 81,920 tokens (10× the base MAX_EMBEDDING_TOKENS of 8,192) and documentation files exceeding 8,192 tokens. This prevents API rejections and wasted compute on minified or generated artifacts.

Can I use local embedding models with DeepWiki for large repositories?

Yes, DeepWiki supports Ollama as a local embedding provider. When you set embedder_type="ollama", the pipeline uses OllamaDocumentProcessor from api/ollama_patch.py, which processes documents individually in streaming mode. This eliminates network latency and batch overhead, making it ideal for large codebases where local GPU resources are available.

Where is the embedding cache stored in DeepWiki?

For batch-oriented providers like DashScope, DeepWiki stores embeddings in ./embedding_cache/ as pickle files. The cache filename includes the embedder class and model name (e.g., ./embedding_cache/myrepo_DashScopeBatchEmbedder_dashscope_embeddings.pkl). This caching is implemented in api/dashscope_client.py and persists across pipeline runs to avoid redundant API calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}