How LLM Wiki Handles Immutable Source Documents in the raw/sources Directory

LLM Wiki enforces strict read-only access to files in raw/sources through a three-layer architecture that copies, extracts, and references source materials without ever mutating the originals.

The nashsu/llm_wiki repository implements a robust content pipeline designed to treat files in the raw/sources directory as immutable source documents. This architectural pattern ensures that original research papers, PDFs, and data files remain pristine while the system generates editable wiki summaries. Understanding this immutability guarantee is essential for maintaining data integrity when building LLM-powered knowledge bases.

The Three-Layer Immutability Architecture

According to the source code in nashsu/llm_wiki, immutability is enforced through three complementary layers that operate during ingestion, processing, and resolution.

Layer 1: Source Watch Configuration

The first layer of protection resides in src/lib/source-watch-config.ts, where the isPathAllowedBySourceWatch function filters incoming files. This utility exclusively validates read eligibility by checking file extensions and path patterns while explicitly excluding binaries, temporary files, and hidden items. Crucially, the configuration never returns a write-permission flag, establishing a read-only contract at the entry point.

Layer 2: Ingestion and Lifecycle

Once admitted, files enter the lifecycle management system in src/lib/source-lifecycle.ts. Functions such as importSources, preprocessFile, and copyFile operate exclusively on a destination copy inside raw/sources. The pipeline extracts text content and generates a separate markdown summary under wiki/sources, ensuring the original file timestamp and bytes remain unaltered after the initial copy operation.

Layer 3: Reference Resolution

The final safeguard occurs during page rendering in src/lib/wiki-page-resolver.ts. When the resolveSourceName function encounters a front-matter entry like sources: [paper.pdf], it searches wiki/sources first for summary pages, then falls back to raw/sources for the original asset. The returned path is treated as read-only; any write operations triggered by the LLM target the generated wiki page, never the source file itself.

How Source Files Are Ingested Without Mutation

When you add a document to the project, LLM Wiki immediately creates a protected copy while generating editable derivatives elsewhere.

# File system layout

/raw/sources/2024/paper.pdf     ←  User-added original (immutable)
/wiki/sources/paper.md          ←  LLM-generated summary (editable)

The ingestion pipeline in src/lib/source-lifecycle.ts handles this automatically:

// src/lib/source-lifecycle.ts
await copyFile(sourcePath, `${projectRoot}/raw/sources/${relativePath}`);
await preprocessFile(`${projectRoot}/raw/sources/${relativePath}`);
// Generates editable summary: wiki/sources/paper.md
// Original /raw/sources/2024/paper.pdf remains untouched

Resolving Front-Matter References to Immutable Sources

Wiki pages reference immutable sources through YAML front matter, which the resolver maps to read-only filesystem locations.

# wiki/notes.md

sources:
  - paper.pdf          # Resolves to raw/sources/2024/paper.pdf

During rendering, the resolution logic enforces the read-only constraint:

// src/lib/wiki-page-resolver.ts
const path = resolveSourceName(index, "paper.pdf", sourcesRoot);
// Returns: "/project/raw/sources/2024/paper.pdf"
// Writes are redirected to wiki/sources/paper.md instead

Guarding Against Disallowed File Types

The system prevents ingestion of potentially problematic files that could violate immutability assumptions. In src/lib/source-watch-config.ts, the validation layer rejects unsupported formats before they enter the pipeline:

// src/lib/source-watch-config.ts
if (!isPathAllowedBySourceWatch("/project/raw/sources/video.mp4", config)) {
  // video.mp4 is ignored – never ingested or stored in raw/sources
}

This filtering ensures only appropriate document types participate in the immutable storage system, while src/lib/source-identity.ts normalizes all valid source references to maintain canonical paths throughout the application.

Summary

• Read-only guarantee: Files in raw/sources are never modified by the LLM or ingestion pipeline after initial copy.
• Derivative workflow: All editing occurs in wiki/sources/*.md files generated from the immutable originals.
• Path enforcement: resolveSourceName in src/lib/wiki-page-resolver.ts treats raw source paths as read-only while routing mutations to wiki summaries.
• Entry validation: isPathAllowedBySourceWatch blocks binaries and hidden files from entering the immutable storage layer.

Frequently Asked Questions

Can I edit files directly in the raw/sources directory?

No. The architecture explicitly forbids modifications to files in raw/sources. If you need to update content, delete the original file and re-import it, or edit the generated markdown summary in wiki/sources that corresponds to your document.

What happens if I add a binary file to raw/sources?

The isPathAllowedBySourceWatch function in src/lib/source-watch-config.ts filters out binaries, videos, and executables during the source-watching phase. These files are ignored by the ingestion pipeline and never copied into the immutable storage area, preventing corruption of the knowledge base with non-textual assets.

How does the system prevent accidental overwrites of source documents?

Three mechanisms prevent overwrites: the source watcher only grants read permissions, the lifecycle functions in src/lib/source-lifecycle.ts copy rather than move files, and the wiki resolver in src/lib/wiki-page-resolver.ts routes all write operations to separate wiki pages. This creates a physical separation between storage (raw/sources) and working memory (wiki/sources).

Where are the editable versions of my sources stored?

Editable summaries reside in wiki/sources/ as markdown files with .md extensions. When you reference paper.pdf in your front matter, the system displays the content from wiki/sources/paper.md while preserving the original PDF in raw/sources/ as an immutable backup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}