# How OpenDeepWiki's AI Smart Filtering Handles Large Repositories: A Technical Deep Dive

> Discover how OpenDeepWiki's AI smart filtering tackles massive repositories. Learn about its sub-second processing of markdown files using vector embeddings and efficient caching.

- Repository: [AIDotNet/OpenDeepWiki](https://github.com/aidotnet/opendeepwiki)
- Tags: deep-dive
- Published: 2026-02-19

---

**OpenDeepWiki processes hundreds of thousands of markdown files in sub-second time by combining dense vector embeddings, nearest-neighbor search, and incremental caching to avoid scanning entire repositories at query time.**

OpenDeepWiki is an open-source documentation platform designed to make massive codebases navigable through intelligent semantic search. When dealing with repositories containing hundreds of thousands of markdown files, traditional keyword matching breaks down under the weight of linear scans. This is where OpenDeepWiki's **AI smart filtering** becomes critical, leveraging pre-computed embeddings and vector similarity to surface relevant documents instantly regardless of repository scale.

## The Three-Stage AI Smart Filtering Pipeline

The core implementation resides in [`src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs), where the `GenerateAsync` method orchestrates the filtering logic. The pipeline consists of three tightly-coupled stages that transform raw repository files into ranked, paginated results.

### Document Indexing and Embedding Generation

During initial repository ingestion, `WikiService.GetRepositoryAsync` (lines 90-100 in [`src/OpenDeepWiki/Services/Wiki/WikiService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiService.cs)) creates a repository record. A background worker then walks the file tree, generating a `DocCatalog` entry for each markdown file.

The `EmbedService` class in [`src/OpenDeepWiki/Chat/Providers/EmbedService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Providers/EmbedService.cs) converts text content into dense vector embeddings using the `GetEmbeddingAsync` method. These embeddings are persisted in the `DocEmbedding` table via `CatalogStorage` (lines 33-45 in [`src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs)) and cached in memory using `OpenDeepWiki.Cache.Memory`.

### Vector Search and Semantic Filtering

When a user submits a query through the `GET /api/v1/wiki/catalog` endpoint, the `TextMessageMerger` (in [`src/OpenDeepWiki/Chat/Queue/TextMessageMerger.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Queue/TextMessageMerger.cs)) processes the request context. The search term is converted into an embedding vector, then `WikiGenerator.GenerateAsync` performs a nearest-neighbors lookup against the pre-computed embeddings.

This vector similarity search intersects with attribute filters such as `IsPublic`, `Branch`, and `Language` to narrow results before pagination occurs.

### Incremental Paging and Caching

For repositories with thousands of files, the system implements efficient result management. The `SessionManager` (in [`src/OpenDeepWiki/Chat/Sessions/SessionManager.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Sessions/SessionManager.cs)) handles pagination state across requests, while `WikiGeneratorOptions` configures the default page size of 50 results (lines 12-18 in [`src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs)).

The `MemoryCacheAdapter` stores vector index pages in memory, making subsequent page requests O(1). Meanwhile, the `RepositoryUpdateWorker` background job detects file changes via `GitTool` (lines 76-84 in [`src/OpenDeepWiki/Agents/Tools/GitTool.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Agents/Tools/GitTool.cs)), which parses `.gitignore` and triggers partial re-embedding only for modified files.

## How It Works for Large Repositories

The AI smart filtering system handles massive scale through four distinct operational phases:

1. **Initial Load** – When a repo is first added, the background worker generates embeddings for all markdown files and stores them in `DocEmbedding` via `CatalogStorage`.

2. **Smart Filtering Request** – Client calls to `/api/v1/wiki/catalog` trigger `WikiGenerator.GenerateAsync`, which converts search terms to vectors and performs nearest-neighbor lookups.

3. **Result Trimming** – The vector search returns a ranked list; the generator applies pagination (default 50 items) configured in `WikiGeneratorOptions`, dramatically reducing payload size even for repos with >10k files.

4. **Cache-Backed Updates** – When new commits arrive, `GitTool` filters unchanged files using `.gitignore` parsing and triggers partial re-indexing via `RepositoryUpdateWorker`, while `MemoryCacheAdapter` ensures subsequent queries remain instant.

## Practical Implementation Examples

Here are concrete examples of interacting with the AI smart filtering system:

```csharp
// Retrieve catalog with semantic search filter
var response = await httpClient.GetAsync(
    $"/api/v1/wiki/catalog?org=dotnet&repo=runtime&search=garbage%20collection");

// Fetch specific document content
var doc = await httpClient.GetAsync(
    $"/api/v1/wiki/doc/dotnet/runtime/System.GC");

// Direct embedding generation (used internally by the filter)
var embed = await embedService.GetEmbeddingAsync("dependency injection in ASP.NET Core");

```

## Key Source Files and Architecture

Understanding the following files is essential for working with the AI smart filtering system:

- [`src/OpenDeepWiki/Services/Wiki/WikiService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiService.cs) – API entry point for repository loading and branch/language resolution
- [`src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs) – Core implementation of the smart filter pipeline including vector search
- [`src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs) – Configuration for pagination size and similarity thresholds
- [`src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs) – Persistence layer for `DocCatalog` and `DocEmbedding` entities
- [`src/OpenDeepWiki/Agents/Tools/GitTool.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Agents/Tools/GitTool.cs) – Change detection and incremental indexing trigger
- [`src/OpenDeepWiki/Chat/Providers/EmbedService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Providers/EmbedService.cs) – LLM embedding generation wrapper
- [`src/OpenDeepWiki/Chat/Sessions/SessionManager.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Sessions/SessionManager.cs) – Pagination state management
- [`framework/OpenDeepWiki.Cache.Memory/MemoryCacheAdapter.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/framework/OpenDeepWiki.Cache.Memory/MemoryCacheAdapter.cs) – In-memory caching of vector indexes

## Summary

OpenDeepWiki's AI smart filtering system solves the large repository problem through three core strategies:

- **Dense vector embeddings** generated by `EmbedService` enable semantic understanding beyond keyword matching
- **Nearest-neighbor vector search** in `WikiGenerator` surfaces relevant documents in constant time regardless of repository size
- **Incremental indexing and caching** via `GitTool` and `MemoryCacheAdapter` ensure updates are efficient and repeated queries are instant

This architecture allows the system to handle repositories with hundreds of thousands of markdown files while maintaining sub-second query latency.

## Frequently Asked Questions

### How does OpenDeepWiki handle repositories with over 100,000 markdown files?

The system uses a combination of pre-computed embeddings and vector similarity search to avoid scanning every file at query time. The `RepositoryUpdateWorker` processes only changed files via `GitTool`, making incremental updates feasible even for massive repositories.

### What embedding model does OpenDeepWiki use for the smart filter?

The `EmbedService` class in [`src/OpenDeepWiki/Chat/Providers/EmbedService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Chat/Providers/EmbedService.cs) wraps the LLM embedding provider. The specific model is configurable, but the system typically uses dense vector embeddings that capture semantic meaning beyond simple keyword matching.

### How does the vector search handle pagination for large result sets?

The `SessionManager` maintains pagination state across requests, while `WikiGeneratorOptions` configures the default page size of 50 results. The `MemoryCacheAdapter` stores intermediate page indexes, making subsequent page requests O(1) operations regardless of the total result set size.

### Can the AI smart filtering work with private repositories?

Yes. The system respects the `IsPublic` attribute filter during the vector search phase. When querying private repositories, the `WikiGenerator` intersects the semantic similarity results with access control checks to ensure users only see documents they have permission to view.