DeepWiki Server-Side Wiki Caching Mechanism: Implementation and Invalidation Strategies

DeepWiki implements a file-system-based server-side cache that stores generated wiki structures as JSON files under a dedicated directory, using deterministic filenames derived from repository metadata, with explicit invalidation via DELETE endpoints and implicit refresh through POST overwrites.

The AsyncFuncAI/deepwiki-open repository uses this server-side wiki caching mechanism to eliminate redundant computation costs. By persisting generated wiki structures and pages to the local file system, the system avoids re-running expensive generation pipelines for previously processed repositories.

How DeepWiki's Server-Side Cache Works

Cache Directory Structure and Initialization

On API startup, DeepWiki automatically creates a dedicated cache directory at <adalflow-root>/wikicache. This path is defined as WIKI_CACHE_DIR in api/api.py (lines 5-7). The directory is auto-created if missing, ensuring the cache is always available without manual intervention.

Deterministic Cache Key Generation

Cache entries are addressed using deterministic filenames derived from repository metadata. The function get_wiki_cache_path in api/api.py (lines 8-11) combines owner, repo, repo_type, and language into a filename formatted as:

<repo_type>_<owner>_<repo>_<language>.json

This deterministic mapping ensures that identical repository requests always resolve to the same cache file, while different languages or repository types receive separate cache entries.

Cache Storage and Retrieval Endpoints

Reading from cache: The GET /api/wiki_cache endpoint (implemented in get_cached_wiki, lines 61-84) constructs the file path from query parameters and loads the JSON into a WikiCacheData Pydantic model. If the file is missing, it returns HTTP 200 with a null body—a behavior the frontend expects to trigger fresh generation.

Writing to cache: The POST /api/wiki_cache endpoint (store_wiki_cache, lines 86-100) receives a WikiCacheRequest, constructs a WikiCacheData payload, and writes it as pretty-printed JSON to the computed file path. This operation overwrites any existing file, effectively refreshing the cache.

Cache Invalidation Strategies in DeepWiki

Explicit Deletion via REST API

The primary invalidation mechanism uses the DELETE /api/wiki_cache endpoint (delete_wiki_cache, lines 104-135). Clients send a request with the same identifying query parameters used for retrieval (owner, repo, repo_type, language). The endpoint validates optional authentication (controlled by DEEPWIKI_AUTH_MODE and DEEPWIKI_AUTH_CODE in api/config.py), then removes the corresponding cache file. Subsequent GET requests return null, forcing a full regeneration.

Implicit Invalidation Through Overwrites

Cache entries are implicitly invalidated when a new generation completes. The POST /api/wiki_cache endpoint writes the new wiki data to the same deterministic file path, overwriting the previous JSON. This atomic replacement ensures that readers always see the most recent generation without requiring an explicit delete operation first.

No TTL: Deterministic Cache Lifetime

Unlike many caching systems, DeepWiki does not implement time-based expiration (TTL). Cache lifetime is controlled entirely through explicit deletion or overwrite operations. This deterministic approach keeps the implementation simple and ensures that cached data remains available indefinitely until actively invalidated, which is appropriate for wiki content that changes only when the underlying repository is updated.

Practical API Usage Examples

Retrieve a Cached Wiki

curl "http://localhost:8000/api/wiki_cache?owner=AsyncFuncAI&repo=deepwiki-open&repo_type=github&language=en"

Cache hit response:

{
  "wiki_structure": {...},
  "generated_pages": [...],
  "repo": {"owner":"AsyncFuncAI","repo":"deepwiki-open","type":"github"},
  "provider":"openai",
  "model":"gpt-4"
}

Cache miss response:

null

Store a Newly Generated Wiki

curl -X POST "http://localhost:8000/api/wiki_cache" \
  -H "Content-Type: application/json" \
  -d '{
    "wiki_structure": {...},
    "generated_pages": [...],
    "repo": {"owner":"AsyncFuncAI","repo":"deepwiki-open","type":"github"},
    "language": "en",
    "provider": "openai",
    "model": "gpt-4"
  }'

Response:

{ "message": "Wiki cache saved successfully" }

Invalidate a Cache Entry

curl -X DELETE "http://localhost:8000/api/wiki_cache?owner=AsyncFuncAI&repo=deepwiki-open&repo_type=github&language=en"

If authentication is enabled, append &authorization_code=YOUR_CODE or set DEEPWIKI_AUTH_MODE=False in your environment.

Response:

{ "message": "Wiki cache for AsyncFuncAI/deepwiki-open (en) deleted successfully" }

List All Cached Projects

curl "http://localhost:8000/api/processed_projects"

Response:

[
  {
    "id":"deepwiki_cache_github_AsyncFuncAI_deepwiki-open_en.json",
    "owner":"AsyncFuncAI",
    "repo":"deepwiki-open",
    "name":"AsyncFuncAI/deepwiki-open",
    "repo_type":"github",
    "submittedAt":1708099200000,
    "language":"en"
  }
]

Key Source Files and Implementation Details

File	Role	Key Components
api/api.py	Core API implementation	WIKI_CACHE_DIR definition (lines 5-7), get_wiki_cache_path() (lines 8-11), get_cached_wiki() endpoint (lines 61-84), store_wiki_cache() endpoint (lines 86-100), delete_wiki_cache() endpoint (lines 104-135), get_processed_projects() endpoint (lines 76-115)
api/config.py	Configuration and authentication	DEEPWIKI_AUTH_MODE flag, DEEPWIKI_AUTH_CODE for securing the delete endpoint
api/models.py	Data models	WikiCacheData and WikiCacheRequest Pydantic models for JSON serialization/deserialization
api/utils.py	Path utilities	get_adalflow_default_root_path() used to locate the cache root directory

These files implement a straightforward, file-system-backed cache that supports CRUD operations through REST endpoints, with deterministic addressing and explicit invalidation controls.

Summary

• DeepWiki's server-side caching mechanism stores generated wiki structures as JSON files under <adalflow-root>/wikicache using deterministic filenames based on repository metadata.
• Cache keys are derived from repo_type, owner, repo, and language, producing filenames like github_AsyncFuncAI_deepwiki-open_en.json.
• Cache invalidation is handled explicitly via the DELETE /api/wiki_cache endpoint or implicitly when POST /api/wiki_cache overwrites existing entries.
• No TTL mechanism exists; cached data persists indefinitely until actively deleted or refreshed, simplifying the design and ensuring deterministic behavior.
• Authentication for destructive operations is controlled via DEEPWIKI_AUTH_MODE and DEEPWIKI_AUTH_CODE in api/config.py.

Frequently Asked Questions

How does DeepWiki determine the filename for a cached wiki?

DeepWiki constructs the cache filename by combining the repo_type, owner, repo, and language parameters into the format <repo_type>_<owner>_<repo>_<language>.json. This logic is implemented in the get_wiki_cache_path function within api/api.py (lines 8-11). The deterministic naming ensures that identical repository requests always map to the same file path.

What happens when a cached wiki entry is missing?

When the GET /api/wiki_cache endpoint cannot locate a file for the requested repository parameters, it returns HTTP status 200 with a null body. This behavior, implemented in get_cached_wiki (api/api.py, lines 61-84), signals to the frontend that no cached data exists and triggers the wiki generation pipeline to create a new entry.

Is there an automatic expiration time for cached wiki files?

No, DeepWiki does not implement time-based expiration (TTL) for cached entries. The cache persists indefinitely until explicitly invalidated through the DELETE /api/wiki_cache endpoint or overwritten by a new POST /api/wiki_cache request. This design choice, documented in the delete_wiki_cache implementation (api/api.py, lines 104-135), provides deterministic cache behavior suitable for wiki content that changes only when source repositories are updated.

How is the cache deletion endpoint secured?

The DELETE /api/wiki_cache endpoint supports optional authentication controlled by environment variables defined in api/config.py. When DEEPWIKI_AUTH_MODE is enabled, requests must include a valid authorization_code parameter matching DEEPWIKI_AUTH_CODE. If authentication fails, the endpoint returns HTTP 403. This security layer prevents unauthorized cache invalidation while allowing open access in trusted environments by setting DEEPWIKI_AUTH_MODE=False.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}