Understanding the File-System Architecture of LLM Wiki: A Deep Dive into the Knowledge Base Structure

The file-system architecture of LLM Wiki implements a three-layer data persistence model (Raw → Wiki → Schema) within each project directory, supported by a monorepo codebase that separates the React TypeScript frontend from the Rust Tauri backend.

LLM Wiki, an open-source knowledge management system from the nashsu/llm_wiki repository, organizes data through a deliberate file-system architecture inspired by Andrej Karpathy’s original LLM Wiki pattern. This design separates immutable source evidence from generated knowledge and structural rules, enabling the application to maintain a queryable graph while preserving raw document integrity.

Project-Level Knowledge Base Structure

When a user creates a new LLM Wiki project, the application generates a folder containing a strict hierarchical layout that separates raw inputs from processed outputs.

The Three-Layer Data Model

The architecture centers on three distinct directories that reflect a transformation pipeline:

• raw/sources/ – Stores immutable source documents including PDFs, DOCX files, Markdown, images, and web clips. The application watches this folder for external changes via file system monitors. Implementation logic resides in src/lib/url-source-import.ts and src/lib/wiki-page-resolver.ts.

• wiki/ – Contains generated Markdown files written by the LLM. Each file includes YAML front-matter recording its type, title, and originating sources via the sources: [] array. The resolution logic is handled in src/lib/wiki-page-resolver.ts and src/lib/wiki-filename.ts.

• schema.md – An optional configuration file defining structural rules, page-type definitions, and routing logic for the knowledge base. The parser is implemented in src/lib/wiki-schema.ts.

Configuration and Navigation Files

Beyond the core data layers, each project root contains specific metadata files:

• index.md – Serves as the entry point and navigation anchor for the LLM during query operations. The ingest pipeline creates and updates this file automatically.

• log.md – Maintains a chronological record of operations including ingest, delete, and lint actions. This enables "time-travel" reasoning where the LLM can parse historical state changes.

• purpose.md – An optional human-written statement defining project goals and research scope. The ingest pipeline reads this via loadProjectStore on every operation to align LLM behavior with user intent.

Hidden Application State

The .llmwiki/ directory stores persistent application configuration using Tauri’s plugin-store mechanism. Located at src/lib/project-store.ts, this layer manages settings, recent projects, and model overrides in a hidden subdirectory to separate application metadata from user content.

Repository Source Code Layout

The nashsu/llm_wiki repository follows a conventional monorepo structure that isolates frontend, backend, and auxiliary services.

Frontend TypeScript Core

The src/ directory contains the React-based user interface and core business logic:

• src/App.tsx – Top-level component responsible for booting the UI and restoring persisted state.
• src/main.tsx – React entry point that renders the application root.
• src/lib/ – Core TypeScript logic including:
  • wiki-graph.ts – Builds the knowledge graph, computes relevance scores, and runs Louvain community detection via detectCommunities.
  • wiki-page-resolver.ts – Resolves [[wikilinks]] to actual wiki pages and maintains source references.
  • wiki-schema.ts – Parses and applies routing rules from schema.md.
  • web-search.ts – Provides a unified façade for Firecrawl, SearXNG, and other search providers.
  • vision-caption.ts – Generates image captions using vision LLMs for multimodal ingest.
• src/stores/ – Zustand state management including wiki-store.ts, chat-store.ts, and update-store.ts.

Native Rust Backend

The src-tauri/ directory houses the Tauri-based native layer:

• src-tauri/src/lib.rs – Exposes core commands for file I/O and HTTP server functionality.
• src-tauri/src/agent/ – Contains the agent runtime responsible for tool-using chat and file generation.

Auxiliary Services

Additional directories provide supporting functionality:

• mcp-server/ – A lightweight local MCP (Model Context Protocol) server implemented in TypeScript. Contains src/api-client.ts for search operations and src/project-binding.ts for project integration.
• extension/ – Chrome extension code for one-click web clipping, including background.js and popup.html.
• assets/ – Static illustrations for UI and documentation.

Runtime Caching Strategy

LLM Wiki optimizes performance through strategic disk caching within the .llmwiki/ directory:

• .llmwiki/cache/ – Stores incremental SHA-256 hashes for source files. The ingest pipeline compares these hashes to skip unchanged files, significantly reducing redundant processing.

• .llmwiki/graph/ – Persists memoized graph data structures for fast UI rendering. The system respects the MAX_CACHED_PROJECT_GRAPHS limit defined in src/lib/wiki-graph.ts to prevent unbounded disk usage.

These caches refresh automatically when the ingest queue processes new sources or detects file modifications.

Interacting with the File System Programmatically

The codebase exposes several utilities for direct file-system manipulation from TypeScript.

Building the Knowledge Graph

The buildWikiGraph function constructs the complete knowledge representation by scanning the wiki/ directory:

import { buildWikiGraph } from "@/lib/wiki-graph";

// projectPath is the absolute path to the current LLM Wiki project
const result = await buildWikiGraph(projectPath);
console.log(result.nodes.length, "nodes", result.edges.length, "edges");

As implemented in src/lib/wiki-graph.ts, this function extracts wikilinks from every Markdown file, calculates relevance scores, and executes community detection algorithms.

Importing External Sources

The importUrlAsSource utility downloads remote content into the immutable raw layer:

import { importUrlAsSource } from "@/lib/url-source-import";

await importUrlAsSource({
  project,
  url: "https://arxiv.org/pdf/2302.00001.pdf",
  title: "Attention Is All You Need",
});

This writes the downloaded file into raw/sources/ and registers it with the auto-watcher, as defined in src/lib/url-source-import.ts.

Executing Web Searches

The unified search interface supports multiple providers through a single function:

import { searchWeb } from "@/lib/web-search";

const results = await searchWeb({
  query: "transformer architecture survey",
  provider: "firecrawl",   // or "searxng", "none"
});
console.log(results.slice(0, 3));

The implementation in src/lib/web-search.ts resolves API keys from application state and normalizes results across different search backends.

Managing Project Persistence

Application metadata access occurs through the project store API:

import { getRecentProjects, saveLastProject } from "@/lib/project-store";

const recent = await getRecentProjects();
await saveLastProject({ 
  id: "proj-1", 
  path: "/home/user/my-wiki", 
  name: "My Wiki" 
});

src/lib/project-store.ts wraps Tauri’s plugin-store to persist JSON state to .llmwiki/app-state.json.

Summary

• LLM Wiki employs a three-layer file-system architecture (Raw → Wiki → Schema) that treats source documents as immutable evidence while allowing LLM-generated content to evolve in the wiki/ directory.
• The monorepo structure cleanly separates the React frontend (src/), Rust Tauri backend (src-tauri/), MCP server (mcp-server/), and browser extension (extension/).
• Runtime caches in .llmwiki/cache/ and .llmwiki/graph/ optimize ingest performance and graph rendering while respecting configurable limits.
• Key implementation files including src/lib/wiki-graph.ts, src/lib/wiki-page-resolver.ts, and src/lib/project-store.ts provide deterministic access to the file-system layers.

Frequently Asked Questions

What is the purpose of the raw/sources/ directory in LLM Wiki?

The raw/sources/ directory stores immutable source documents such as PDFs, images, and web clips. According to the source code in src/lib/url-source-import.ts, this folder is watched for external changes, and its contents serve as the ground-truth evidence that the LLM references when generating wiki pages.

How does LLM Wiki handle application state persistence?

Application state persists through Tauri’s plugin-store API, wrapped by src/lib/project-store.ts. This stores configuration, recent projects, and model overrides in a hidden .llmwiki/ directory, specifically within .llmwiki/app-state.json, separate from user-generated knowledge content.

What role does schema.md play in the file-system architecture?

The optional schema.md file defines structural rules, page-type definitions, and routing logic for the knowledge base. Parsed by src/lib/wiki-schema.ts, it enforces consistency across the wiki/ directory by constraining how the LLM generates and links pages.

How does the Chrome extension integrate with the desktop application’s file system?

The extension in extension/background.js captures web pages and forwards them to the desktop app, which then writes the content into raw/sources/ using the same pipeline as importUrlAsSource. This creates a seamless bridge between browser content and the local knowledge base.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}