# How to Troubleshoot DeepWiki Wiki Generation Failures on Large Repositories

> Fix DeepWiki wiki generation failures for large repos. Verify embedding dimensions, use filters, clear cache, and regenerate to resolve issues.

- Repository: [ASYNCFUNC/deepwiki-open](https://github.com/asyncfuncai/deepwiki-open)
- Tags: how-to-guide
- Published: 2026-02-16

---

**To resolve DeepWiki wiki generation failures on large repositories, verify consistent embedding dimensions across all documents, implement inclusion and exclusion filters to reduce corpus size, and clear corrupted cache entries before regenerating.**

DeepWiki, the open-source RAG-based documentation generator from AsyncFuncAI/deepwiki-open, can encounter failures when processing large repositories with thousands of files or massive source trees. These DeepWiki wiki generation failures typically stem from embedding mismatches, token limit overflows, or corpus size issues that overwhelm the retrieval pipeline. Understanding the specific error patterns and diagnostic locations in the codebase allows you to implement targeted fixes without rebuilding the entire infrastructure.

## Common Symptoms of DeepWiki Wiki Generation Failures

When processing large codebases, the DeepWiki pipeline surfaces specific errors that map to distinct failure modes:

- **`500` Internal Server Error with "No valid document embeddings found"** — Indicates that inconsistent or missing embeddings across the document set caused the validator to discard all documents. Detected in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 112‑115) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) (lines 117‑120).

- **`500` Internal Server Error with "All embeddings should be of the same size"** — Occurs when documents were embedded with different dimensionalities, often from mixed provider models. Detected in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) (lines 94‑98).

- **`500` Internal Server Error with "maximum context length / token limit"** — The prompt plus retrieved context exceeds the provider-specific token budget (approximately 8,000 tokens). Detected in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 75‑84) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) (lines 80‑88).

- **Empty or incomplete wiki pages** — Retrieval returned no documents because they were filtered out by exclusion rules or size limits. Originates in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) through `prepare_retriever` → `DatabaseManager` → `should_process_file`.

- **Slow or hanging responses** — Heavy RAG processing on the full repository without size guards causes long embedding or indexing passes. Originates in [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) in the document ingestion loop.

## Verify the Embedding Pipeline for Consistency

The RAG pipeline in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) relies on uniform vector representations. When processing large repositories, embedding inconsistencies are the primary source of hard failures.

### Fix Inconsistent Embedding Sizes

DeepWiki expects **all documents** to share the same embedding dimension. The validator `_validate_and_filter_embeddings` (lines 51‑86 in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py)) logs the most common size and discards outliers. If you encounter the error *"All embeddings should be of the same size"* (detected at lines 94‑98), the pipeline has already filtered mismatched documents and aborted.

To resolve this:

1. **Check the provider configuration** in [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) to ensure the embedder matches your intended model.
2. **Force a single embedder type** by setting the `EMBEDDER_TYPE` environment variable (e.g., `google`, `openai`, or `ollama`) so all documents use the same client.
3. **Inspect the validation log** — the validator prints a sample of the first ten embedding sizes (see [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) lines 998‑1014).

### Resolve Missing Embeddings

If a document fails to embed, the validator logs *"Document X has no embedding vector"* and removes it. When **every** document is removed, you receive the "No valid document embeddings found" error, detected in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 112‑115) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) (lines 117‑120).

Run the data pipeline manually on a small subset to confirm the embedder works:

```python
from api.data_pipeline import DatabaseManager

dm = DatabaseManager()
docs = dm.prepare_database(
    repo_url_or_path='https://github.com/your/large-repo',
    type='github',
    access_token=None,
    embedder_type='google',  # match your config

    excluded_dirs=None,
    excluded_files=None,
    included_dirs=None,
    included_files=None,
)
print(f'Processed {len(docs)} documents')

```

If the pipeline crashes, check network connectivity to the embedder's API (e.g., Google Gemini, OpenAI).

## Prevent Token-Limit Overflows in Large Repositories

Large repositories generate extensive retrieved context that can exceed model context windows.

### Detect Oversized Inputs

Both the WebSocket and HTTP chat handlers compute the token count of the **last user message** (see [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) lines 80‑84 and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) lines 85‑89). If it exceeds **8,000** tokens, they set `input_too_large = True` and later skip RAG to avoid blowing up the model context window.

To avoid this:

- **Split huge requests** into multiple smaller queries targeting specific files or sub-modules.
- **Trim the message** before sending by removing superfluous code snippets or verbose descriptions.

### Handle Automatic Fallbacks

When the provider returns a token-limit error, the handlers rebuild a *simplified prompt* without any retrieved context (see [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) lines 720‑734 and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) lines 63‑78).

You can:

- **Accept the fallback** — the response will be less informed but still functional.
- **Reduce the retrieved context** by narrowing the search scope via `included_dirs` or `excluded_dirs` parameters (see `ChatCompletionRequest` fields).

## Optimize Repository Scope with Inclusion and Exclusion Filters

The RAG pipeline can be instructed to process only a subset of the repository, which is essential for large monorepos.

The filtering parameters are parsed in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 96‑107) and passed to `RAG.prepare_retriever`. The actual filtering logic resides in `should_process_file` within [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) (lines 235‑252).

Available parameters:

- `excluded_dirs` — comma-separated list of directories (URL-decoded), e.g., `"tests\nnode_modules"`
- `excluded_files` — comma-separated list of glob patterns, e.g., `"*.md\n*.png"`
- `included_dirs` — process *only* these directories, e.g., `"src\ndocs"`
- `included_files` — process *only* these file patterns, e.g., `"*.py\n*.tsx"`

Reducing the corpus drastically cuts embedding time and memory usage, preventing token-limit blow-outs.

Example request body:

```json
{
  "repo_url": "https://github.com/AsyncFuncAI/deepwiki-open",
  "messages": [{ "role": "user", "content": "Explain the WebSocket protocol implementation." }],
  "excluded_dirs": "tests\nnode_modules",
  "included_dirs": "api"
}

```

## Clear Corrupted Wiki Cache Entries

DeepWiki stores generated wiki structures in a per-project cache defined in [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) (`WIKI_CACHE_DIR`). A corrupted or incomplete cache can cause downstream failures.

To manage the cache:

- **List cached projects** — Query `GET /api/processed_projects` which reads the cache directory ([`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) lines 776‑830).
- **Delete a stale cache** — Send `DELETE /api/wiki_cache` with the correct `owner`, `repo`, `repo_type`, and `language` parameters (see [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) lines 504‑538).

After clearing the cache, rerun the generation so the pipeline rebuilds the embeddings from scratch.

## Diagnostic Logging for DeepWiki Failures

All components log to the standard logger configured by [`api/logging_config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/logging_config.py) (`setup_logging`). The most useful log entries appear at **INFO** and **WARNING** level:

- **Embedding validation** — "Target embedding size: …", "Filtering out document … due to embedding size mismatch"
- **RAG retrieval** — "Retrieved X documents", "No documents retrieved from RAG"
- **Prompt building** — "Creating simplified prompt …"

Ensure the container (Docker, Docker‑Compose, or local process) streams logs to the console or a file, then grep for the keywords `embedding`, `token`, or `retriever`.

## Quick-Fix Checklist for DeepWiki Generation Issues

Follow this systematic checklist when troubleshooting large repository failures:

1. **Reduce the repository scope** with `included_dirs` or `included_files` parameters.
2. **Verify that `EMBEDDER_TYPE`** matches the provider configured in [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py).
3. **Run a tiny ingestion test** (see the local pipeline debugging script) to confirm embeddings are produced.
4. **If you hit a token-limit error**, either split the user query or let the fallback run without context.
5. **Clear the wiki cache** for the project using `DELETE /api/wiki_cache`.
6. **Review logs** for messages about mismatched embedding sizes or missing vectors.
7. **Re-run the generation request** after the above adjustments.

## Code Examples for Troubleshooting

### Filtered Generation Request via cURL

Submit a generation request with explicit filters to reduce corpus size:

```bash
curl -X POST https://your-deepwiki-host/chat/completions/stream \
  -H "Content-Type: application/json" \
  -d '{
        "repo_url":"https://github.com/your/large-repo",
        "messages":[{"role":"user","content":"[DEEP RESEARCH] Summarize the authentication flow."}],
        "included_dirs":"src\nauth",
        "excluded_files":"*.test.js\n*.spec.ts",
        "provider":"openai",
        "model":"gpt-4o"
      }'

```

### Cache Deletion with Python Requests

Delete a corrupted cache entry before regenerating:

```python
import requests

params = {
    "owner": "your",
    "repo": "large-repo",
    "repo_type": "github",
    "language": "en",
    "authorization_code": "YOUR_CODE"      # required only if WIKI_AUTH_MODE is true

}
resp = requests.delete(
    "https://your-deepwiki-host/api/wiki_cache",
    params=params
)
print(resp.json())

```

### Local Pipeline Debugging Script

Run the data pipeline locally to verify embedding generation:

```bash
python - <<'PY'
from api.data_pipeline import DatabaseManager
dm = DatabaseManager()
docs = dm.prepare_database(
    repo_url_or_path='https://github.com/your/large-repo',
    type='github',
    access_token=None,
    embedder_type='google',
    excluded_dirs='tests\nnode_modules',
    excluded_files='*.md',
    included_dirs=None,
    included_files=None,
)
print(f"Processed {len(docs)} documents")
PY

```

The script prints the number of documents that survived the filter and were successfully embedded—useful to confirm that the repository is not being pruned away entirely.

## Key Files in the DeepWiki RAG Pipeline

Understanding these core files helps you trace failures quickly:

| File | Responsibility | Important Sections |
|---|---|---|
| [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) | Core RAG class, embedding validation, retriever creation | `_validate_and_filter_embeddings` (lines 51‑86) |
| [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) | WebSocket chat endpoint, token‑size guard, fallback logic | Input‑size check (lines 75‑84) and token‑limit fallback (lines 719‑734) |
| [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) | HTTP chat endpoint (POST `/chat/completions/stream`), same guards as WebSocket | Token check (lines 80‑89) and fallback (lines 63‑78) |
| [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) | Wiki cache API (GET/POST/DELETE), export endpoints, processed‑projects list | Cache helpers (`get_wiki_cache_path`, `read_wiki_cache`, `save_wiki_cache`) and `/api/processed_projects` (lines 776‑830) |
| [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py) | Document ingestion, embedding via various providers, file‑filter logic | `should_process_file` (lines 235‑252) and `prepare_database` (lines 462‑525) |
| [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) | Central configuration for providers, models, embedder type | `get_embedder_type`, `get_model_config` |
| [`api/logging_config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/logging_config.py) | Logging setup used throughout the service | `setup_logging` |

These files together constitute the **pipeline** that processes a repository, builds embeddings, stores a searchable index, and generates the final wiki. Most failure modes originate from mismatches in these stages, so reviewing the corresponding sections will give you the needed context to debug issues on large repositories.

## Summary

- **Embedding consistency is critical** — Ensure all documents use the same embedder type via `EMBEDDER_TYPE` and check [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) validation logs to catch size mismatches early.
- **Token limits require guards** — The WebSocket and HTTP handlers in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) automatically detect inputs exceeding 8,000 tokens and fall back to context-free prompts.
- **Scope reduction prevents overload** — Use `included_dirs`, `excluded_dirs`, and file glob patterns parsed in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) to limit the corpus processed by [`api/data_pipeline.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/data_pipeline.py).
- **Cache corruption causes silent failures** — Delete stale entries via `DELETE /api/wiki_cache` (implemented in [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py) lines 504‑538) to force fresh embedding generation.
- **Logs provide definitive diagnostics** — Check [`api/logging_config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/logging_config.py) output for embedding validation messages, retrieval counts, and prompt simplification events.

## Frequently Asked Questions

### Why does DeepWiki return "All embeddings should be of the same size" when processing large repositories?

This error originates in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) (lines 94‑98) when the `_validate_and_filter_embeddings` function detects documents with varying vector dimensionalities. Large repositories often trigger this if different file types were processed with different embedder configurations or if the `EMBEDDER_TYPE` environment variable changed between ingestion runs. To fix this, set a consistent `EMBEDDER_TYPE` (e.g., `google`, `openai`, or `ollama`) in [`api/config.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/config.py) and re-ingest the repository from scratch after clearing the cache.

### How can I prevent token limit errors when generating wikis for monorepos?

DeepWiki implements an 8,000-token guard in both [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 80‑84) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) (lines 85‑89) that sets `input_too_large = True` when the last user message exceeds this threshold. To prevent hitting this limit in monorepos, use the `included_dirs` and `included_files` parameters to scope the RAG retrieval to specific subdirectories (e.g., `"src\nauth"`), or split your query into smaller, file-specific questions that generate less retrieved context.

### What causes the "No valid document embeddings found" error in DeepWiki?

This error appears in [`api/websocket_wiki.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/websocket_wiki.py) (lines 112‑115) and [`api/simple_chat.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/simple_chat.py) (lines 117‑120) when the `_validate_and_filter_embeddings` function in [`api/rag.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/rag.py) removes every document from the corpus due to missing vectors or size mismatches. On large repositories, this typically occurs when the embedding provider API fails silently for batches of files or when the `EMBEDDER_TYPE` configuration does not match the actual model used during ingestion. Run the local pipeline debugging script against a small subset to verify the embedder is producing vectors before processing the entire repository.

### Where does DeepWiki store cached wiki data and how do I clear it?

DeepWiki stores generated wiki structures in a per-project cache directory defined by `WIKI_CACHE_DIR` in [`api/api.py`](https://github.com/AsyncFuncAI/deepwiki-open/blob/main/api/api.py). You can list cached projects via `GET /api/processed_projects` (implemented at lines 776‑830), which reads the cache directory structure. To delete a corrupted cache entry, send a `DELETE` request to `/api/wiki_cache` with the `owner`, `repo`, `repo_type`, and `language` parameters (handled at lines 504‑538). After clearing the cache, the next generation request will rebuild the embeddings from scratch, resolving issues caused by stale or incomplete index data.