How OpenDeepWiki's AI Smart Filtering Handles Large Repositories: A Technical Deep Dive

OpenDeepWiki processes hundreds of thousands of markdown files in sub-second time by combining dense vector embeddings, nearest-neighbor search, and incremental caching to avoid scanning entire repositories at query time.

OpenDeepWiki is an open-source documentation platform designed to make massive codebases navigable through intelligent semantic search. When dealing with repositories containing hundreds of thousands of markdown files, traditional keyword matching breaks down under the weight of linear scans. This is where OpenDeepWiki's AI smart filtering becomes critical, leveraging pre-computed embeddings and vector similarity to surface relevant documents instantly regardless of repository scale.

The Three-Stage AI Smart Filtering Pipeline

The core implementation resides in src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs, where the GenerateAsync method orchestrates the filtering logic. The pipeline consists of three tightly-coupled stages that transform raw repository files into ranked, paginated results.

Document Indexing and Embedding Generation

During initial repository ingestion, WikiService.GetRepositoryAsync (lines 90-100 in src/OpenDeepWiki/Services/Wiki/WikiService.cs) creates a repository record. A background worker then walks the file tree, generating a DocCatalog entry for each markdown file.

The EmbedService class in src/OpenDeepWiki/Chat/Providers/EmbedService.cs converts text content into dense vector embeddings using the GetEmbeddingAsync method. These embeddings are persisted in the DocEmbedding table via CatalogStorage (lines 33-45 in src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs) and cached in memory using OpenDeepWiki.Cache.Memory.

Vector Search and Semantic Filtering

When a user submits a query through the GET /api/v1/wiki/catalog endpoint, the TextMessageMerger (in src/OpenDeepWiki/Chat/Queue/TextMessageMerger.cs) processes the request context. The search term is converted into an embedding vector, then WikiGenerator.GenerateAsync performs a nearest-neighbors lookup against the pre-computed embeddings.

This vector similarity search intersects with attribute filters such as IsPublic, Branch, and Language to narrow results before pagination occurs.

Incremental Paging and Caching

For repositories with thousands of files, the system implements efficient result management. The SessionManager (in src/OpenDeepWiki/Chat/Sessions/SessionManager.cs) handles pagination state across requests, while WikiGeneratorOptions configures the default page size of 50 results (lines 12-18 in src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs).

The MemoryCacheAdapter stores vector index pages in memory, making subsequent page requests O(1). Meanwhile, the RepositoryUpdateWorker background job detects file changes via GitTool (lines 76-84 in src/OpenDeepWiki/Agents/Tools/GitTool.cs), which parses .gitignore and triggers partial re-embedding only for modified files.

How It Works for Large Repositories

The AI smart filtering system handles massive scale through four distinct operational phases:

1. Initial Load – When a repo is first added, the background worker generates embeddings for all markdown files and stores them in DocEmbedding via CatalogStorage.

2. Smart Filtering Request – Client calls to /api/v1/wiki/catalog trigger WikiGenerator.GenerateAsync, which converts search terms to vectors and performs nearest-neighbor lookups.

3. Result Trimming – The vector search returns a ranked list; the generator applies pagination (default 50 items) configured in WikiGeneratorOptions, dramatically reducing payload size even for repos with >10k files.

4. Cache-Backed Updates – When new commits arrive, GitTool filters unchanged files using .gitignore parsing and triggers partial re-indexing via RepositoryUpdateWorker, while MemoryCacheAdapter ensures subsequent queries remain instant.

Practical Implementation Examples

Here are concrete examples of interacting with the AI smart filtering system:

// Retrieve catalog with semantic search filter
var response = await httpClient.GetAsync(
    $"/api/v1/wiki/catalog?org=dotnet&repo=runtime&search=garbage%20collection");

// Fetch specific document content
var doc = await httpClient.GetAsync(
    $"/api/v1/wiki/doc/dotnet/runtime/System.GC");

// Direct embedding generation (used internally by the filter)
var embed = await embedService.GetEmbeddingAsync("dependency injection in ASP.NET Core");

Key Source Files and Architecture

Understanding the following files is essential for working with the AI smart filtering system:

• src/OpenDeepWiki/Services/Wiki/WikiService.cs – API entry point for repository loading and branch/language resolution
• src/OpenDeepWiki/Services/Wiki/WikiGenerator.cs – Core implementation of the smart filter pipeline including vector search
• src/OpenDeepWiki/Services/Wiki/WikiGeneratorOptions.cs – Configuration for pagination size and similarity thresholds
• src/OpenDeepWiki/Services/Wiki/CatalogStorage.cs – Persistence layer for DocCatalog and DocEmbedding entities
• src/OpenDeepWiki/Agents/Tools/GitTool.cs – Change detection and incremental indexing trigger
• src/OpenDeepWiki/Chat/Providers/EmbedService.cs – LLM embedding generation wrapper
• src/OpenDeepWiki/Chat/Sessions/SessionManager.cs – Pagination state management
• framework/OpenDeepWiki.Cache.Memory/MemoryCacheAdapter.cs – In-memory caching of vector indexes

Summary

OpenDeepWiki's AI smart filtering system solves the large repository problem through three core strategies:

• Dense vector embeddings generated by EmbedService enable semantic understanding beyond keyword matching
• Nearest-neighbor vector search in WikiGenerator surfaces relevant documents in constant time regardless of repository size
• Incremental indexing and caching via GitTool and MemoryCacheAdapter ensure updates are efficient and repeated queries are instant

This architecture allows the system to handle repositories with hundreds of thousands of markdown files while maintaining sub-second query latency.

Frequently Asked Questions

How does OpenDeepWiki handle repositories with over 100,000 markdown files?

The system uses a combination of pre-computed embeddings and vector similarity search to avoid scanning every file at query time. The RepositoryUpdateWorker processes only changed files via GitTool, making incremental updates feasible even for massive repositories.

What embedding model does OpenDeepWiki use for the smart filter?

The EmbedService class in src/OpenDeepWiki/Chat/Providers/EmbedService.cs wraps the LLM embedding provider. The specific model is configurable, but the system typically uses dense vector embeddings that capture semantic meaning beyond simple keyword matching.

How does the vector search handle pagination for large result sets?

The SessionManager maintains pagination state across requests, while WikiGeneratorOptions configures the default page size of 50 results. The MemoryCacheAdapter stores intermediate page indexes, making subsequent page requests O(1) operations regardless of the total result set size.

Can the AI smart filtering work with private repositories?

Yes. The system respects the IsPublic attribute filter during the vector search phase. When querying private repositories, the WikiGenerator intersects the semantic similarity results with access control checks to ensure users only see documents they have permission to view.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}