How LLM Wiki Preserves Directory Structures During Folder Imports

LLM Wiki preserves directory structures by calculating each file's path relative to the imported folder root, then recreating that exact hierarchy inside the project's raw/sources directory using atomic file operations.

When importing external folders into a knowledge base, maintaining the original organizational structure is critical for context preservation. The nashsu/llm_wiki project implements a robust folder import system that mirrors external hierarchies exactly inside the project's raw/sources directory. Understanding how LLM Wiki preserves directory structures during folder imports reveals a pattern of relative path resolution combined with recursive directory reconstruction.

The Import Pipeline in source-lifecycle.ts

The core logic resides in src/lib/source-lifecycle.ts, specifically within the importSourceFolder function. This implementation uses a four-stage pipeline to ensure faithful replication of complex folder trees.

Step 1: Recursive File Discovery

First, the system collects all files including those nested in subdirectories and hidden folders. The flattenFiles utility combines with listDirectory to produce a flat array of absolute paths while traversing the entire tree:

const sourceFiles = flattenFiles(await listDirectory(selectedFolder, true));

This operation, found at line 394, ensures that no content is missed regardless of nesting depth. The true parameter enables recursive descent into subdirectories.

Step 2: Relative Path Calculation

For each discovered file, the system computes its position relative to the import root using getRelativePath from src/lib/path-utils.ts:

const relativeSourcePath = getRelativePath(file.path, sourceRoot);

Located at line 397, this calculation strips the external folder's base path while preserving the internal hierarchy. This relative path becomes the blueprint for destination placement.

Step 3: Directory Recreation and File Copying

The destination path is constructed by appending the relative path to the project's raw/sources directory:

const destPath = `${destDir}/${relativeSourcePath}`;

Before copying, the system ensures parent directories exist. At lines 420-423, parentPath extracts the directory component, createDirectory builds missing intermediate folders, and copyFile performs the atomic transfer:

const parent = parentPath(destPath);
if (parent) await createDirectory(parent);
await copyFile(file.path, destPath);

This sequence guarantees that deep hierarchies like notes/2024/january/ are recreated faithfully even if multiple parent levels are missing.

Step 4: Deterministic Ordering

After successful imports, the file list is sorted using naturalCompare at lines 430-433. This alphabetic ordering ensures consistent processing sequences across different operating systems and filesystems.

Safety Mechanisms and Edge Cases

Beyond basic copying, the implementation includes safeguards for production environments.

Handling Hidden Files and Dot-Folders

The system explicitly honors hidden directories such as .claude or .git because flattenFiles includes them in its traversal (lines 90-94). However, it filters out tool-configuration files that might contain secrets, balancing completeness with security.

Preventing Self-Referential Imports

To avoid infinite recursion or circular copies, the code rejects attempts to import the project folder itself or any of its subdirectories. Lines 81-83 validate the source path against the project root before processing begins.

Practical Implementation Examples

Developers can trigger imports programmatically using the exposed API.

Standard programmatic usage:

import { importSourceFolder } from "@/lib/source-lifecycle";

await importSourceFolder(
  { id: "proj1", name: "MyWiki", path: "/home/user/mywiki" },
  "/home/user/external-notes",
  llmConfig,
  sourceWatchConfig
);

This execution transforms an external structure like:

/home/user/external-notes/
├─ notes/
│  └─ meeting.md
└─ .claude/
   └─ research.md

Into:

/home/user/mywiki/raw/sources/imported/
├─ notes/
│  └─ meeting.md
└─ .claude/
   └─ research.md

Filtered import with configuration:

await importSourceFolder(
  { id: "p1", name: "Project", path: "/project" },
  "/external/imported",
  fakeLlmConfig,
  {
    enabled: true,
    includeExtensions: ["md"],
    excludeExtensions: ["json"],
    excludeDirs: ["drafts"],
    maxFileSizeMb: 100,
  },
);

This configuration selectively imports only Markdown files while excluding JSON configurations and draft directories.

Summary

• Relative path resolution via getRelativePath ensures the original folder hierarchy is mathematically preserved during transit from source to destination.
• Atomic directory creation using parentPath and createDirectory guarantees that deep nesting levels are reconstructed exactly, even with missing intermediate folders.
• Recursive traversal through flattenFiles and listDirectory captures all content including hidden dot-folders while excluding sensitive configuration files.
• Self-reference protection prevents catastrophic circular copying by validating the import target against the project root.
• Deterministic ordering with naturalCompare maintains consistent file sequences across different platforms and filesystem implementations.

Frequently Asked Questions

Does LLM Wiki maintain the exact folder hierarchy when importing nested directories?

Yes. The system calculates each file's path relative to the imported folder root using getRelativePath, then reconstructs that identical path inside raw/sources. This approach preserves unlimited nesting depths, ensuring that a file located at external/docs/teams/engineering/specs.md appears at project/raw/sources/imported/docs/teams/engineering/specs.md.

How does the import system handle hidden folders like .git or .claude?

The flattenFiles utility explicitly includes dot-folders in its traversal (lines 90-94), meaning hidden directories are preserved in the imported structure. However, the system filters out specific tool-configuration files that might contain API keys or secrets, striking a balance between completeness and security.

What prevents importing the project folder into itself?

Lines 81-83 in src/lib/source-lifecycle.ts implement a validation check that compares the import source path against the project root. If the system detects a self-referential import attempt—such as importing a parent directory containing the wiki project—it rejects the operation to prevent infinite recursion and circular file copying.

Can I filter which files get imported while still preserving directory structure?

Yes. The importSourceFolder function accepts a configuration object allowing includeExtensions, excludeExtensions, and excludeDirs parameters. These filters apply during the file discovery phase, meaning only matched files are copied, but those files maintain their original relative paths within the preserved hierarchy.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}