DeepWiki Architecture and RAG Pipeline: How It Processes Repositories and Generates Wikis

DeepWiki uses a modular FastAPI backend built on AdalFlow that clones repositories, embeds source code into a FAISS vector store, and uses Retrieval-Augmented Generation (RAG) to produce structured wiki documentation.

DeepWiki is an open-source tool that automatically generates comprehensive wiki documentation from code repositories. Understanding the DeepWiki architecture and RAG pipeline reveals how it transforms raw source code into searchable, AI-generated documentation through a sophisticated retrieval and generation system.

DeepWiki Architecture Overview

The DeepWiki architecture follows a layered design that separates concerns between API handling, data processing, retrieval, and generation. Each layer is loosely coupled through configuration dictionaries defined in api/config.py, allowing you to switch providers or models without modifying core logic.

Entry Point and API Layer

The application bootstrap occurs in api/main.py, which initializes the FastAPI server, loads environment variables, and mounts routes defined in api/api.py. The API layer handles HTTP and WebSocket routes for model configuration, repository processing, wiki caching, and streaming chat completions.

RAG Core and Data Pipeline

At the heart of the system lies the RAG class in api/rag.py, which orchestrates embedding, retrieval, and generation. The DatabaseManager in api/data_pipeline.py handles repository cloning, document ingestion, token-aware splitting, and vector storage using AdalFlow's LocalDB.

Frontend Integration

A React/Next.js frontend consumes these endpoints, displaying generated wiki trees and allowing users to tune model providers. The frontend communicates with the backend through REST APIs and WebSockets for real-time chat streaming.

How the DeepWiki RAG Pipeline Works for Repository Processing

The DeepWiki RAG pipeline processes repositories through a linear ten-step flow, transforming raw code into queryable knowledge bases and generated documentation.

Step 1: Repository Acquisition

The pipeline begins with download_repo() in api/data_pipeline.py (lines 72-100), which clones repositories from GitHub, GitLab, or Bitbucket. If the repository already exists locally, the download is skipped to save time.

Step 2: Document Ingestion and Filtering

The read_all_documents() function (lines 99-124, 133-162) scans the repository path, applying exclusion filters (DEFAULT_EXCLUDED_DIRS and DEFAULT_EXCLUDED_FILES) while respecting optional inclusion patterns. Files are categorized into code and documentation groups based on extensions.

Step 3: Token-Aware Text Splitting

Documents are processed through AdalFlow's TextSplitter, configured in configs["text_splitter"]. This occurs inside prepare_data_pipeline at line 106, ensuring chunks respect token limits while preserving semantic boundaries.

Step 4: Embedding Generation

The get_embedder() factory in api/tools/embedder.py (lines 44-56) instantiates the appropriate embedding client based on DEEPWIKI_EMBEDDER_TYPE. For Ollama, a OllamaDocumentProcessor handles single-document processing (lines 111-115 in data_pipeline.py), while OpenAI and Google use batched ToEmbeddings transformers (lines 115-119).

Step 5: Vector Database Persistence

Processed documents are stored via LocalDB in api/data_pipeline.py (lines 140-149):

db = LocalDB()
db.register_transformer(transformer=data_transformer, key="split_and_embed")
db.load(documents)
db.transform(key="split_and_embed")
db.save_state(filepath)

The database is saved to ~/.adalflow/databases/<owner>_<repo>.pkl. Subsequent runs use the fast load path in DatabaseManager.prepare_db_index (lines 68-85).

Step 6: Retriever Construction with FAISS

The RAG.prepare_retriever() method in api/rag.py (lines 84-90) constructs a FAISSRetriever:

self.retriever = FAISSRetriever(
    **configs["retriever"],
    embedder=retrieve_embedder,
    documents=self.transformed_docs,
    document_map_func=lambda doc: doc.vector,
)

The system validates consistent embedding sizes in _validate_and_filter_embeddings (lines 51-88) before building the FAISS index.

Step 7: Query Embedding and Retrieval

When processing user queries, the RAG class embeds the query using self.query_embedder. For Ollama, this uses single-string embedding (lines 94-101), while other providers use standard batch embedding. The FAISSRetriever returns the top-k most relevant documents based on the configuration in configs["retriever"].

Step 8: Prompt Composition

The generation phase uses AdalFlow's Generator instantiated in RAG.__init__ (lines 31-44). The prompt combines:

• RAG_TEMPLATE (system prompt + user query placeholder) from api.prompts
• RAG_SYSTEM_PROMPT as the system context
• Conversation history from Memory (self.memory())
• Retrieved documents as context

Output parsing is enforced by DataClassParser for the RAGAnswer structure (lines 10-14).

Step 9: LLM Generation

The RAG.call() method (lines 16-30) executes the retrieval and generation:

retrieved_documents = self.retriever(query)
response = self.generator(...)  # internally called by adalflow

The LLM receives the retrieved snippets as contexts and produces a markdown-ready answer in RAGAnswer.answer.

Step 10: Wiki Assembly and Caching

The frontend repeatedly calls /chat/completions/stream (defined in api/api.py lines 93-99) to generate section pages and structure. Once complete, the client POSTs to /api/wiki_cache (lines 86-102) to persist the structure and generated pages to ~/.adalflow/wikicache/.

Practical Implementation: Using the DeepWiki RAG System

Below is a complete example demonstrating how to use the DeepWiki RAG pipeline programmatically:

from api.rag import RAG
from api.data_pipeline import DatabaseManager
import asyncio

async def build_wiki(repo_url: str, repo_type: str = "github", token: str | None = None):
    # 1️⃣ Initialise the RAG component (choose provider/model)

    rag = RAG(provider="google", model="gemini-1.5-flash")   # uses Google GenAI by default

    # 2️⃣ Prepare the retriever (download, embed, index)

    rag.prepare_retriever(
        repo_url_or_path=repo_url,
        type=repo_type,
        access_token=token,
        # optional: limit processing to src/ only

        included_dirs=["src"],
    )

    # 3️⃣ Ask a high‑level question – the generator will synthesize a wiki page

    answer, docs = rag.call(
        query="Generate an outline of the public API for this repository.",
        language="en"
    )
    print("Outline:", answer.answer)   # markdown ready

    # 4️⃣ Persist the cache for later UI consumption

    # (the FastAPI endpoints do this, but we can call the helper directly)

    from api.api import save_wiki_cache, WikiCacheRequest, WikiCacheData, WikiStructureModel, WikiPage

    # Build a minimal WikiStructureModel from the answer (omitted for brevity)

    # ...

# Run the async helper

asyncio.run(build_wiki("https://github.com/AsyncFuncAI/deepwiki-open"))

Key implementation details:

• RAG(provider, model) loads the model client via config.get_model_config and creates the Generator (lines 31-44 of rag.py).
• prepare_retriever calls DatabaseManager.prepare_database → download_repo → read_all_documents → prepare_data_pipeline → transform_documents_and_save_to_db (all in api/data_pipeline.py, lines 45-150).
• rag.call embeds the query (single-string for Ollama), retrieves docs via FAISSRetriever, then runs the LLM with the assembled prompt.

Key Source Files in the DeepWiki Architecture

File	Role	Direct link
api/main.py	Application bootstrap, env loading, dev-mode watchfiles, uvicorn start.	view
api/api.py	FastAPI routes for model config, auth, export, local repo view, wiki cache, health, processed-projects.	view
api/rag.py	Core Retrieval-Augmented Generation class – memory, retriever, generator, embedding handling.	view
api/data_pipeline.py	Repo cloning, document ingestion, token-aware splitting, embedding, DB persistence, validation.	view
api/tools/embedder.py	Factory mapping DEEPWIKI_EMBEDDER_TYPE to concrete adalflow embedder (OpenAI, Google, Ollama, Bedrock).	view
api/config.py	Loads JSON configs, resolves ${ENV_VAR} placeholders, builds dictionaries for models, embedders, file filters.	view
api/websocket_wiki.py	WebSocket handler for streaming chat, uses RAG under the hood.	view
src/components/**	React UI components calling API endpoints, rendering wiki trees, model selection.	tree
src/utils/**	Helper utilities for URL handling, WebSocket client, language context.	tree

Summary

• DeepWiki architecture consists of a FastAPI backend with clearly separated layers for entry points, API routing, RAG orchestration, and data processing, all built on the AdalFlow framework.
• The RAG pipeline follows a ten-step linear flow: repository acquisition, document filtering, token-aware splitting, embedding generation, vector database persistence, FAISS retriever construction, query embedding, prompt composition, LLM generation, and wiki caching.
• All components are configuration-driven through JSON files and environment variables, allowing seamless switching between embedding providers (OpenAI, Google, Ollama, Bedrock) and LLM backends without code changes.
• The system uses FAISS for efficient similarity search and AdalFlow's LocalDB for persistent storage of processed repository embeddings.

Frequently Asked Questions

How does DeepWiki handle different types of code repositories?

DeepWiki supports GitHub, GitLab, and Bitbucket repositories through the download_repo() function in api/data_pipeline.py (lines 72-100). The system automatically detects the repository type and handles authentication via access tokens when provided. Once cloned, the read_all_documents() function categorizes files into code and documentation groups based on extensions, applying configurable exclusion filters to skip build artifacts and dependencies.

What embedding models does DeepWiki support?

DeepWiki supports multiple embedding providers through a factory pattern in api/tools/embedder.py. The system can use OpenAI, Google GenAI, Ollama, and AWS Bedrock embeddings, selected via the DEEPWIKI_EMBEDDER_TYPE environment variable. Each provider is abstracted through AdalFlow's embedding interface, allowing the RAG pipeline in api/rag.py to remain agnostic to the specific embedding implementation.

How does the RAG pipeline ensure relevant context retrieval?

The RAG pipeline uses FAISS (Facebook AI Similarity Search) for efficient vector similarity search, implemented in the FAISSRetriever class instantiated within RAG.prepare_retriever() (lines 84-90 of api/rag.py). Before building the index, the system validates embedding consistency through _validate_and_filter_embeddings() (lines 51-88). During query time, the pipeline embeds the user query using the same embedder and retrieves the top-k most similar document chunks, which are then injected into the LLM prompt along with conversation history from the Memory component.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}