How to Troubleshoot DeepWiki Wiki Generation Failures on Large Repositories

To resolve DeepWiki wiki generation failures on large repositories, verify consistent embedding dimensions across all documents, implement inclusion and exclusion filters to reduce corpus size, and clear corrupted cache entries before regenerating.

DeepWiki, the open-source RAG-based documentation generator from AsyncFuncAI/deepwiki-open, can encounter failures when processing large repositories with thousands of files or massive source trees. These DeepWiki wiki generation failures typically stem from embedding mismatches, token limit overflows, or corpus size issues that overwhelm the retrieval pipeline. Understanding the specific error patterns and diagnostic locations in the codebase allows you to implement targeted fixes without rebuilding the entire infrastructure.

Common Symptoms of DeepWiki Wiki Generation Failures

When processing large codebases, the DeepWiki pipeline surfaces specific errors that map to distinct failure modes:

• 500 Internal Server Error with "No valid document embeddings found" — Indicates that inconsistent or missing embeddings across the document set caused the validator to discard all documents. Detected in api/websocket_wiki.py (lines 112‑115) and api/simple_chat.py (lines 117‑120).

• 500 Internal Server Error with "All embeddings should be of the same size" — Occurs when documents were embedded with different dimensionalities, often from mixed provider models. Detected in api/rag.py (lines 94‑98).

• 500 Internal Server Error with "maximum context length / token limit" — The prompt plus retrieved context exceeds the provider-specific token budget (approximately 8,000 tokens). Detected in api/websocket_wiki.py (lines 75‑84) and api/simple_chat.py (lines 80‑88).

• Empty or incomplete wiki pages — Retrieval returned no documents because they were filtered out by exclusion rules or size limits. Originates in api/rag.py through prepare_retriever → DatabaseManager → should_process_file.

• Slow or hanging responses — Heavy RAG processing on the full repository without size guards causes long embedding or indexing passes. Originates in api/data_pipeline.py in the document ingestion loop.

Verify the Embedding Pipeline for Consistency

The RAG pipeline in api/rag.py relies on uniform vector representations. When processing large repositories, embedding inconsistencies are the primary source of hard failures.

Fix Inconsistent Embedding Sizes

DeepWiki expects all documents to share the same embedding dimension. The validator _validate_and_filter_embeddings (lines 51‑86 in api/rag.py) logs the most common size and discards outliers. If you encounter the error "All embeddings should be of the same size" (detected at lines 94‑98), the pipeline has already filtered mismatched documents and aborted.

To resolve this:

1. Check the provider configuration in api/config.py to ensure the embedder matches your intended model.
2. Force a single embedder type by setting the EMBEDDER_TYPE environment variable (e.g., google, openai, or ollama) so all documents use the same client.
3. Inspect the validation log — the validator prints a sample of the first ten embedding sizes (see api/rag.py lines 998‑1014).

Resolve Missing Embeddings

If a document fails to embed, the validator logs "Document X has no embedding vector" and removes it. When every document is removed, you receive the "No valid document embeddings found" error, detected in api/websocket_wiki.py (lines 112‑115) and api/simple_chat.py (lines 117‑120).

Run the data pipeline manually on a small subset to confirm the embedder works:

from api.data_pipeline import DatabaseManager

dm = DatabaseManager()
docs = dm.prepare_database(
    repo_url_or_path='https://github.com/your/large-repo',
    type='github',
    access_token=None,
    embedder_type='google',  # match your config

    excluded_dirs=None,
    excluded_files=None,
    included_dirs=None,
    included_files=None,
)
print(f'Processed {len(docs)} documents')

If the pipeline crashes, check network connectivity to the embedder's API (e.g., Google Gemini, OpenAI).

Prevent Token-Limit Overflows in Large Repositories

Large repositories generate extensive retrieved context that can exceed model context windows.

Detect Oversized Inputs

Both the WebSocket and HTTP chat handlers compute the token count of the last user message (see api/websocket_wiki.py lines 80‑84 and api/simple_chat.py lines 85‑89). If it exceeds 8,000 tokens, they set input_too_large = True and later skip RAG to avoid blowing up the model context window.

To avoid this:

• Split huge requests into multiple smaller queries targeting specific files or sub-modules.
• Trim the message before sending by removing superfluous code snippets or verbose descriptions.

Handle Automatic Fallbacks

When the provider returns a token-limit error, the handlers rebuild a simplified prompt without any retrieved context (see api/websocket_wiki.py lines 720‑734 and api/simple_chat.py lines 63‑78).

You can:

• Accept the fallback — the response will be less informed but still functional.
• Reduce the retrieved context by narrowing the search scope via included_dirs or excluded_dirs parameters (see ChatCompletionRequest fields).

Optimize Repository Scope with Inclusion and Exclusion Filters

The RAG pipeline can be instructed to process only a subset of the repository, which is essential for large monorepos.

The filtering parameters are parsed in api/websocket_wiki.py (lines 96‑107) and passed to RAG.prepare_retriever. The actual filtering logic resides in should_process_file within api/data_pipeline.py (lines 235‑252).

Available parameters:

• excluded_dirs — comma-separated list of directories (URL-decoded), e.g., "tests\nnode_modules"
• excluded_files — comma-separated list of glob patterns, e.g., "*.md\n*.png"
• included_dirs — process only these directories, e.g., "src\ndocs"
• included_files — process only these file patterns, e.g., "*.py\n*.tsx"

Reducing the corpus drastically cuts embedding time and memory usage, preventing token-limit blow-outs.

Example request body:

{
  "repo_url": "https://github.com/AsyncFuncAI/deepwiki-open",
  "messages": [{ "role": "user", "content": "Explain the WebSocket protocol implementation." }],
  "excluded_dirs": "tests\nnode_modules",
  "included_dirs": "api"
}

Clear Corrupted Wiki Cache Entries

DeepWiki stores generated wiki structures in a per-project cache defined in api/api.py (WIKI_CACHE_DIR). A corrupted or incomplete cache can cause downstream failures.

To manage the cache:

• List cached projects — Query GET /api/processed_projects which reads the cache directory (api/api.py lines 776‑830).
• Delete a stale cache — Send DELETE /api/wiki_cache with the correct owner, repo, repo_type, and language parameters (see api/api.py lines 504‑538).

After clearing the cache, rerun the generation so the pipeline rebuilds the embeddings from scratch.

Diagnostic Logging for DeepWiki Failures

All components log to the standard logger configured by api/logging_config.py (setup_logging). The most useful log entries appear at INFO and WARNING level:

• Embedding validation — "Target embedding size: …", "Filtering out document … due to embedding size mismatch"
• RAG retrieval — "Retrieved X documents", "No documents retrieved from RAG"
• Prompt building — "Creating simplified prompt …"

Ensure the container (Docker, Docker‑Compose, or local process) streams logs to the console or a file, then grep for the keywords embedding, token, or retriever.

Quick-Fix Checklist for DeepWiki Generation Issues

Follow this systematic checklist when troubleshooting large repository failures:

1. Reduce the repository scope with included_dirs or included_files parameters.
2. Verify that EMBEDDER_TYPE matches the provider configured in api/config.py.
3. Run a tiny ingestion test (see the local pipeline debugging script) to confirm embeddings are produced.
4. If you hit a token-limit error, either split the user query or let the fallback run without context.
5. Clear the wiki cache for the project using DELETE /api/wiki_cache.
6. Review logs for messages about mismatched embedding sizes or missing vectors.
7. Re-run the generation request after the above adjustments.

Code Examples for Troubleshooting

Filtered Generation Request via cURL

Submit a generation request with explicit filters to reduce corpus size:

curl -X POST https://your-deepwiki-host/chat/completions/stream \
  -H "Content-Type: application/json" \
  -d '{
        "repo_url":"https://github.com/your/large-repo",
        "messages":[{"role":"user","content":"[DEEP RESEARCH] Summarize the authentication flow."}],
        "included_dirs":"src\nauth",
        "excluded_files":"*.test.js\n*.spec.ts",
        "provider":"openai",
        "model":"gpt-4o"
      }'

Cache Deletion with Python Requests

Delete a corrupted cache entry before regenerating:

import requests

params = {
    "owner": "your",
    "repo": "large-repo",
    "repo_type": "github",
    "language": "en",
    "authorization_code": "YOUR_CODE"      # required only if WIKI_AUTH_MODE is true

}
resp = requests.delete(
    "https://your-deepwiki-host/api/wiki_cache",
    params=params
)
print(resp.json())

Local Pipeline Debugging Script

Run the data pipeline locally to verify embedding generation:

python - <<'PY'
from api.data_pipeline import DatabaseManager
dm = DatabaseManager()
docs = dm.prepare_database(
    repo_url_or_path='https://github.com/your/large-repo',
    type='github',
    access_token=None,
    embedder_type='google',
    excluded_dirs='tests\nnode_modules',
    excluded_files='*.md',
    included_dirs=None,
    included_files=None,
)
print(f"Processed {len(docs)} documents")
PY

The script prints the number of documents that survived the filter and were successfully embedded—useful to confirm that the repository is not being pruned away entirely.

Key Files in the DeepWiki RAG Pipeline

Understanding these core files helps you trace failures quickly:

File	Responsibility	Important Sections
api/rag.py	Core RAG class, embedding validation, retriever creation	_validate_and_filter_embeddings (lines 51‑86)
api/websocket_wiki.py	WebSocket chat endpoint, token‑size guard, fallback logic	Input‑size check (lines 75‑84) and token‑limit fallback (lines 719‑734)
api/simple_chat.py	HTTP chat endpoint (POST /chat/completions/stream), same guards as WebSocket	Token check (lines 80‑89) and fallback (lines 63‑78)
api/api.py	Wiki cache API (GET/POST/DELETE), export endpoints, processed‑projects list	Cache helpers (get_wiki_cache_path, read_wiki_cache, save_wiki_cache) and /api/processed_projects (lines 776‑830)
api/data_pipeline.py	Document ingestion, embedding via various providers, file‑filter logic	should_process_file (lines 235‑252) and prepare_database (lines 462‑525)
api/config.py	Central configuration for providers, models, embedder type	get_embedder_type, get_model_config
api/logging_config.py	Logging setup used throughout the service	setup_logging

These files together constitute the pipeline that processes a repository, builds embeddings, stores a searchable index, and generates the final wiki. Most failure modes originate from mismatches in these stages, so reviewing the corresponding sections will give you the needed context to debug issues on large repositories.

Summary

• Embedding consistency is critical — Ensure all documents use the same embedder type via EMBEDDER_TYPE and check api/rag.py validation logs to catch size mismatches early.
• Token limits require guards — The WebSocket and HTTP handlers in api/websocket_wiki.py and api/simple_chat.py automatically detect inputs exceeding 8,000 tokens and fall back to context-free prompts.
• Scope reduction prevents overload — Use included_dirs, excluded_dirs, and file glob patterns parsed in api/websocket_wiki.py to limit the corpus processed by api/data_pipeline.py.
• Cache corruption causes silent failures — Delete stale entries via DELETE /api/wiki_cache (implemented in api/api.py lines 504‑538) to force fresh embedding generation.
• Logs provide definitive diagnostics — Check api/logging_config.py output for embedding validation messages, retrieval counts, and prompt simplification events.

Frequently Asked Questions

Why does DeepWiki return "All embeddings should be of the same size" when processing large repositories?

This error originates in api/rag.py (lines 94‑98) when the _validate_and_filter_embeddings function detects documents with varying vector dimensionalities. Large repositories often trigger this if different file types were processed with different embedder configurations or if the EMBEDDER_TYPE environment variable changed between ingestion runs. To fix this, set a consistent EMBEDDER_TYPE (e.g., google, openai, or ollama) in api/config.py and re-ingest the repository from scratch after clearing the cache.

How can I prevent token limit errors when generating wikis for monorepos?

DeepWiki implements an 8,000-token guard in both api/websocket_wiki.py (lines 80‑84) and api/simple_chat.py (lines 85‑89) that sets input_too_large = True when the last user message exceeds this threshold. To prevent hitting this limit in monorepos, use the included_dirs and included_files parameters to scope the RAG retrieval to specific subdirectories (e.g., "src\nauth"), or split your query into smaller, file-specific questions that generate less retrieved context.

What causes the "No valid document embeddings found" error in DeepWiki?

This error appears in api/websocket_wiki.py (lines 112‑115) and api/simple_chat.py (lines 117‑120) when the _validate_and_filter_embeddings function in api/rag.py removes every document from the corpus due to missing vectors or size mismatches. On large repositories, this typically occurs when the embedding provider API fails silently for batches of files or when the EMBEDDER_TYPE configuration does not match the actual model used during ingestion. Run the local pipeline debugging script against a small subset to verify the embedder is producing vectors before processing the entire repository.

Where does DeepWiki store cached wiki data and how do I clear it?

DeepWiki stores generated wiki structures in a per-project cache directory defined by WIKI_CACHE_DIR in api/api.py. You can list cached projects via GET /api/processed_projects (implemented at lines 776‑830), which reads the cache directory structure. To delete a corrupted cache entry, send a DELETE request to /api/wiki_cache with the owner, repo, repo_type, and language parameters (handled at lines 504‑538). After clearing the cache, the next generation request will rebuild the embeddings from scratch, resolving issues caused by stale or incomplete index data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}