# Optimizing Embedding Performance When Dealing with Large Codebases in DeepWiki

> Boost embedding performance for large codebases in DeepWiki with batch-size tuning, token filtering, and provider caching. Optimize processing, avoid limits.

- Repository: [ASYNCFUNC/deepwiki-open](https://github.com/asyncfuncai/deepwiki-open)
- Tags: performance
- Published: 2026-02-16

---

**DeepWiki uses batch-size tuning, token-based file filtering, and provider-specific caching to process massive repositories without hitting API limits or memory constraints.**

When working with the `AsyncFuncAI/deepwiki-open` repository, optimizing embedding performance becomes critical as codebase size grows. DeepWiki processes every source file as a separate document and generates vector embeddings using providers like OpenAI, Google, Ollama, or Bedrock. Without optimization, large files or high document counts trigger slow processing, memory exhaustion, or provider rate limits. The following strategies, implemented directly in the source code, allow you to maintain high throughput while handling enterprise-scale repositories.

## Built-in Strategies for Optimizing Embedding Performance

### Tune Batch Sizes Per Provider

DeepWiki configures provider-specific batch sizes to maximize throughput without overwhelming API endpoints. In [`api/tools/embedder.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py), the `get_embedder` factory constructs the embedding client, while [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) reads the batch configuration and instantiates the `ToEmbeddings` component.

The default configurations are:
- **OpenAI**: 500 documents per batch
- **Google/Bedrock**: 100 documents per batch  
- **Ollama**: Single-document streaming mode (no batching)

You can override these values in `configs["embedder_*"]["batch_size"]` to match your specific rate limits and latency requirements.

### Skip Oversized Files with Token Limits

To prevent API rejections and wasted compute, DeepWiki implements token-based filtering in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py). The `count_tokens()` function uses `tiktoken` to estimate document size with provider-specific encodings (OpenAI's exact model encoding or `cl100k` for others).

The system applies these thresholds:
- **Code files**: Skipped if exceeding `MAX_EMBEDDING_TOKENS * 10` (81,920 tokens for the default 8,192 limit)
- **Documentation files**: Skipped if exceeding `MAX_EMBEDDING_TOKENS` (8,192 tokens)

This prevents embedding failures on generated artifacts or minified bundles while preserving standard source files.

### Leverage Embedding Caching

For batch-oriented providers like DashScope, DeepWiki implements persistent caching in [`api/dashscope_client.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/dashscope_client.py). The `DashScopeBatchEmbedder` class stores full lists of `EmbedderOutput` objects as pickle files in `./embedding_cache/`.

The cache key combines the embedding model name and a user-specified identifier:

```python
self.cache_path = f"./embedding_cache/{embedding_cache_file_name}_{self.__class__.__name__}_{embedder.model}.pkl"

```

Subsequent pipeline runs load embeddings from disk instead of re-calling the remote API, dramatically reducing processing time for incremental updates or repeated analysis of the same repository.

### Filter by File Type and Directory

The `read_all_documents()` function in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) supports granular inclusion and exclusion filters to reduce document volume before embedding:

- `included_dirs`: Process only specific subdirectories (e.g., `["src", "lib"]`)
- `excluded_dirs`: Skip test directories, build artifacts, or dependencies (e.g., `["node_modules", "tests"]`)
- `included_files`/`excluded_files`: Pattern-based file filtering

The internal `should_process_file()` helper implements this logic, ensuring only relevant source files consume embedding budget.

### Use Provider-Specific Processing Modes

DeepWiki adapts its processing strategy based on embedder capabilities. In [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py), the `prepare_data_pipeline()` function checks the embedder type and instantiates the appropriate transformer:

- **Standard embedders** (OpenAI, Google, Bedrock): Use `ToEmbeddings` with configurable batch sizes
- **Ollama**: Uses `OllamaDocumentProcessor` from [`api/ollama_patch.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/ollama_patch.py) for single-document streaming

For large codebases where local inference is viable, switching to Ollama eliminates batch overhead and network latency entirely.

## Practical Code Examples for DeepWiki Optimization

### Example 1: Configuring Batch Size for Google Embedder

Increase throughput for the Google embedder by adjusting the batch size in your configuration:

```python
from api.config import configs
from api.data_pipeline import prepare_data_pipeline

# Override default batch size (100) for higher throughput

configs["embedder_google"]["batch_size"] = 200

# Build pipeline with optimized settings

pipeline = prepare_data_pipeline(embedder_type="google")

```

### Example 2: Selective Directory Processing with Token Limits

Process only production code while skipping oversized generated files:

```python
from api.data_pipeline import read_all_documents

# Focus on src directory, exclude tests, apply token filtering automatically

docs = read_all_documents(
    path="/path/to/repo",
    included_dirs=["src"],
    excluded_dirs=["src/tests", "src/__pycache__"],
    embedder_type="openai"
)

```

### Example 3: Reusing DashScope Embeddings Cache

Avoid redundant API calls by leveraging the built-in cache mechanism:

```python
from api.dashscope_client import DashScopeBatchEmbedder
from api.tools.embedder import get_embedder

embedder = get_embedder(embedder_type="dashscope")
batch_processor = DashScopeBatchEmbedder(
    embedder=embedder,
    batch_size=25,  # DashScope hard limit

    embedding_cache_file_name="my_large_repo"
)

# First run persists to ./embedding_cache/

vectors = batch_processor(["def example(): pass", "class Example: pass"])

# Second run loads from cache instantly

vectors_cached = batch_processor(["def example(): pass", "class Example: pass"])

```

### Example 4: Switching to Ollama for Local Streaming

Eliminate network overhead for large repositories using local Ollama inference:

```python
from api.data_pipeline import prepare_data_pipeline

# Uses OllamaDocumentProcessor for single-document streaming

pipeline = prepare_data_pipeline(embedder_type="ollama")

```

## Key Configuration Files and Components

Understanding the codebase structure helps you locate optimization parameters:

| File | Role |
|------|------|
| [`api/tools/embedder.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py) | Factory that constructs `adal.Embedder` instances for each provider. |
| [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) | Loads [`embedder.json`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/embedder.json) and exposes configuration helpers like `get_embedder_type`. |
| [`api/config/embedder.json`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config/embedder.json) | Central configuration for provider-specific options including `batch_size` and `model_kwargs`. |
| [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) | Orchestrates file discovery, token filtering via `count_tokens()`, and pipeline construction with `prepare_data_pipeline()`. |
| [`api/dashscope_client.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/dashscope_client.py) | Implements `DashScopeBatchEmbedder` with persistent pickle-based caching in `./embedding_cache/`. |
| [`api/ollama_patch.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/ollama_patch.py) | Provides `OllamaDocumentProcessor` for single-document streaming mode. |

## Summary

Optimizing embedding performance when dealing with large codebases in DeepWiki requires combining multiple built-in strategies:

- **Tune batch sizes** per provider in [`api/config/embedder.json`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config/embedder.json) to match API rate limits and maximize throughput.
- **Filter oversized files** automatically using token-based thresholds that skip code files exceeding 81,920 tokens and docs exceeding 8,192 tokens.
- **Enable embedding caching** for batch providers like DashScope to avoid redundant API calls across pipeline runs.
- **Narrow document scope** using `included_dirs`, `excluded_dirs`, and file pattern filters in `read_all_documents()`.
- **Select provider-specific modes** such as Ollama's single-document streaming to eliminate batch overhead for local inference.

## Frequently Asked Questions

### What is the default batch size for OpenAI embeddings in DeepWiki?

The default batch size for OpenAI embeddings is **500 documents per request**. This value is defined in the configuration and passed to the `ToEmbeddings` component in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py). You can override this in [`api/config/embedder.json`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config/embedder.json) if your API rate limits or latency requirements differ.

### How does DeepWiki handle files that exceed token limits?

DeepWiki uses `tiktoken` to estimate token counts before embedding. In [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py), the system skips **code files exceeding 81,920 tokens** (10× the base `MAX_EMBEDDING_TOKENS` of 8,192) and **documentation files exceeding 8,192 tokens**. This prevents API rejections and wasted compute on minified or generated artifacts.

### Can I use local embedding models with DeepWiki for large repositories?

Yes, DeepWiki supports **Ollama** as a local embedding provider. When you set `embedder_type="ollama"`, the pipeline uses `OllamaDocumentProcessor` from [`api/ollama_patch.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/ollama_patch.py), which processes documents individually in streaming mode. This eliminates network latency and batch overhead, making it ideal for large codebases where local GPU resources are available.

### Where is the embedding cache stored in DeepWiki?

For batch-oriented providers like DashScope, DeepWiki stores embeddings in `./embedding_cache/` as pickle files. The cache filename includes the embedder class and model name (e.g., `./embedding_cache/myrepo_DashScopeBatchEmbedder_dashscope_embeddings.pkl`). This caching is implemented in [`api/dashscope_client.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/dashscope_client.py) and persists across pipeline runs to avoid redundant API calls.