# DeepWiki Architecture and RAG Pipeline: How It Processes Repositories and Generates Wikis

> Explore the DeepWiki architecture and RAG pipeline. Learn how it processes repositories and generates wiki documentation using FastAPI and FAISS vector stores.

- Repository: [ASYNCFUNC/deepwiki-open](https://github.com/asyncfuncai/deepwiki-open)
- Tags: architecture
- Published: 2026-02-16

---

**DeepWiki uses a modular FastAPI backend built on AdalFlow that clones repositories, embeds source code into a FAISS vector store, and uses Retrieval-Augmented Generation (RAG) to produce structured wiki documentation.**

DeepWiki is an open-source tool that automatically generates comprehensive wiki documentation from code repositories. Understanding the DeepWiki architecture and RAG pipeline reveals how it transforms raw source code into searchable, AI-generated documentation through a sophisticated retrieval and generation system.

## DeepWiki Architecture Overview

The DeepWiki architecture follows a layered design that separates concerns between API handling, data processing, retrieval, and generation. Each layer is loosely coupled through configuration dictionaries defined in [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py), allowing you to switch providers or models without modifying core logic.

### Entry Point and API Layer

The application bootstrap occurs in [`api/main.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/main.py), which initializes the FastAPI server, loads environment variables, and mounts routes defined in [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py). The API layer handles HTTP and WebSocket routes for model configuration, repository processing, wiki caching, and streaming chat completions.

### RAG Core and Data Pipeline

At the heart of the system lies the `RAG` class in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py), which orchestrates embedding, retrieval, and generation. The `DatabaseManager` in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) handles repository cloning, document ingestion, token-aware splitting, and vector storage using AdalFlow's `LocalDB`.

### Frontend Integration

A React/Next.js frontend consumes these endpoints, displaying generated wiki trees and allowing users to tune model providers. The frontend communicates with the backend through REST APIs and WebSockets for real-time chat streaming.

## How the DeepWiki RAG Pipeline Works for Repository Processing

The DeepWiki RAG pipeline processes repositories through a linear ten-step flow, transforming raw code into queryable knowledge bases and generated documentation.

### Step 1: Repository Acquisition

The pipeline begins with `download_repo()` in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) (lines 72-100), which clones repositories from GitHub, GitLab, or Bitbucket. If the repository already exists locally, the download is skipped to save time.

### Step 2: Document Ingestion and Filtering

The `read_all_documents()` function (lines 99-124, 133-162) scans the repository path, applying exclusion filters (`DEFAULT_EXCLUDED_DIRS` and `DEFAULT_EXCLUDED_FILES`) while respecting optional inclusion patterns. Files are categorized into code and documentation groups based on extensions.

### Step 3: Token-Aware Text Splitting

Documents are processed through AdalFlow's `TextSplitter`, configured in `configs["text_splitter"]`. This occurs inside `prepare_data_pipeline` at line 106, ensuring chunks respect token limits while preserving semantic boundaries.

### Step 4: Embedding Generation

The `get_embedder()` factory in [`api/tools/embedder.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py) (lines 44-56) instantiates the appropriate embedding client based on `DEEPWIKI_EMBEDDER_TYPE`. For Ollama, a `OllamaDocumentProcessor` handles single-document processing (lines 111-115 in [`data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/data_pipeline.py)), while OpenAI and Google use batched `ToEmbeddings` transformers (lines 115-119).

### Step 5: Vector Database Persistence

Processed documents are stored via `LocalDB` in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) (lines 140-149):

```python
db = LocalDB()
db.register_transformer(transformer=data_transformer, key="split_and_embed")
db.load(documents)
db.transform(key="split_and_embed")
db.save_state(filepath)

```

The database is saved to `~/.adalflow/databases/<owner>_<repo>.pkl`. Subsequent runs use the fast load path in `DatabaseManager.prepare_db_index` (lines 68-85).

### Step 6: Retriever Construction with FAISS

The `RAG.prepare_retriever()` method in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) (lines 84-90) constructs a `FAISSRetriever`:

```python
self.retriever = FAISSRetriever(
    **configs["retriever"],
    embedder=retrieve_embedder,
    documents=self.transformed_docs,
    document_map_func=lambda doc: doc.vector,
)

```

The system validates consistent embedding sizes in `_validate_and_filter_embeddings` (lines 51-88) before building the FAISS index.

### Step 7: Query Embedding and Retrieval

When processing user queries, the `RAG` class embeds the query using `self.query_embedder`. For Ollama, this uses single-string embedding (lines 94-101), while other providers use standard batch embedding. The `FAISSRetriever` returns the top-k most relevant documents based on the configuration in `configs["retriever"]`.

### Step 8: Prompt Composition

The generation phase uses AdalFlow's `Generator` instantiated in `RAG.__init__` (lines 31-44). The prompt combines:
- `RAG_TEMPLATE` (system prompt + user query placeholder) from `api.prompts`
- `RAG_SYSTEM_PROMPT` as the system context
- Conversation history from `Memory` (`self.memory()`)
- Retrieved documents as context

Output parsing is enforced by `DataClassParser` for the `RAGAnswer` structure (lines 10-14).

### Step 9: LLM Generation

The `RAG.call()` method (lines 16-30) executes the retrieval and generation:

```python
retrieved_documents = self.retriever(query)
response = self.generator(...)  # internally called by adalflow

```

The LLM receives the retrieved snippets as contexts and produces a markdown-ready answer in `RAGAnswer.answer`.

### Step 10: Wiki Assembly and Caching

The frontend repeatedly calls `/chat/completions/stream` (defined in [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) lines 93-99) to generate section pages and structure. Once complete, the client POSTs to `/api/wiki_cache` (lines 86-102) to persist the structure and generated pages to `~/.adalflow/wikicache/`.

## Practical Implementation: Using the DeepWiki RAG System

Below is a complete example demonstrating how to use the DeepWiki RAG pipeline programmatically:

```python
from api.rag import RAG
from api.data_pipeline import DatabaseManager
import asyncio

async def build_wiki(repo_url: str, repo_type: str = "github", token: str | None = None):
    # 1️⃣ Initialise the RAG component (choose provider/model)

    rag = RAG(provider="google", model="gemini-1.5-flash")   # uses Google GenAI by default

    # 2️⃣ Prepare the retriever (download, embed, index)

    rag.prepare_retriever(
        repo_url_or_path=repo_url,
        type=repo_type,
        access_token=token,
        # optional: limit processing to src/ only

        included_dirs=["src"],
    )

    # 3️⃣ Ask a high‑level question – the generator will synthesize a wiki page

    answer, docs = rag.call(
        query="Generate an outline of the public API for this repository.",
        language="en"
    )
    print("Outline:", answer.answer)   # markdown ready

    # 4️⃣ Persist the cache for later UI consumption

    # (the FastAPI endpoints do this, but we can call the helper directly)

    from api.api import save_wiki_cache, WikiCacheRequest, WikiCacheData, WikiStructureModel, WikiPage

    # Build a minimal WikiStructureModel from the answer (omitted for brevity)

    # ...

# Run the async helper

asyncio.run(build_wiki("https://github.com/AsyncFuncAI/deepwiki-open"))

```

**Key implementation details:**

- `RAG(provider, model)` loads the model client via `config.get_model_config` and creates the `Generator` (lines 31-44 of [`rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/rag.py)).
- `prepare_retriever` calls `DatabaseManager.prepare_database` → `download_repo` → `read_all_documents` → `prepare_data_pipeline` → `transform_documents_and_save_to_db` (all in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py), lines 45-150).
- `rag.call` embeds the query (single-string for Ollama), retrieves docs via `FAISSRetriever`, then runs the LLM with the assembled prompt.

## Key Source Files in the DeepWiki Architecture

| File | Role | Direct link |
|------|------|------------|
| [`api/main.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/main.py) | Application bootstrap, env loading, dev-mode watchfiles, uvicorn start. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/main.py) |
| [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) | FastAPI routes for model config, auth, export, local repo view, wiki cache, health, processed-projects. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) |
| [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) | Core Retrieval-Augmented Generation class – memory, retriever, generator, embedding handling. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) |
| [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) | Repo cloning, document ingestion, token-aware splitting, embedding, DB persistence, validation. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) |
| [`api/tools/embedder.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py) | Factory mapping `DEEPWIKI_EMBEDDER_TYPE` to concrete `adalflow` embedder (OpenAI, Google, Ollama, Bedrock). | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py) |
| [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) | Loads JSON configs, resolves `${ENV_VAR}` placeholders, builds dictionaries for models, embedders, file filters. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) |
| [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) | WebSocket handler for streaming chat, uses `RAG` under the hood. | [view](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) |
| `src/components/**` | React UI components calling API endpoints, rendering wiki trees, model selection. | [tree](https://github.com/AsyncFuncAI/deepwiki-open/tree/main/src/components) |
| `src/utils/**` | Helper utilities for URL handling, WebSocket client, language context. | [tree](https://github.com/AsyncFuncAI/deepwiki-open/tree/main/src/utils) |

## Summary

- **DeepWiki architecture** consists of a FastAPI backend with clearly separated layers for entry points, API routing, RAG orchestration, and data processing, all built on the AdalFlow framework.
- The **RAG pipeline** follows a ten-step linear flow: repository acquisition, document filtering, token-aware splitting, embedding generation, vector database persistence, FAISS retriever construction, query embedding, prompt composition, LLM generation, and wiki caching.
- All components are **configuration-driven** through JSON files and environment variables, allowing seamless switching between embedding providers (OpenAI, Google, Ollama, Bedrock) and LLM backends without code changes.
- The system uses **FAISS** for efficient similarity search and **AdalFlow's LocalDB** for persistent storage of processed repository embeddings.

## Frequently Asked Questions

### How does DeepWiki handle different types of code repositories?

DeepWiki supports GitHub, GitLab, and Bitbucket repositories through the `download_repo()` function in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) (lines 72-100). The system automatically detects the repository type and handles authentication via access tokens when provided. Once cloned, the `read_all_documents()` function categorizes files into code and documentation groups based on extensions, applying configurable exclusion filters to skip build artifacts and dependencies.

### What embedding models does DeepWiki support?

DeepWiki supports multiple embedding providers through a factory pattern in [`api/tools/embedder.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/tools/embedder.py). The system can use OpenAI, Google GenAI, Ollama, and AWS Bedrock embeddings, selected via the `DEEPWIKI_EMBEDDER_TYPE` environment variable. Each provider is abstracted through AdalFlow's embedding interface, allowing the RAG pipeline in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) to remain agnostic to the specific embedding implementation.

### How does the RAG pipeline ensure relevant context retrieval?

The RAG pipeline uses FAISS (Facebook AI Similarity Search) for efficient vector similarity search, implemented in the `FAISSRetriever` class instantiated within `RAG.prepare_retriever()` (lines 84-90 of [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py)). Before building the index, the system validates embedding consistency through `_validate_and_filter_embeddings()` (lines 51-88). During query time, the pipeline embeds the user query using the same embedder and retrieves the top-k most similar document chunks, which are then injected into the LLM prompt along with conversation history from the `Memory` component.