How Litho Handles File Filtering for Documentation Generation: A Deep Dive into the StructureExtractor

Litho filters files and directories through a configurable multi-layer pipeline in StructureExtractor that applies user-defined rules for tests, hidden files, extensions, and binary detection before any AI processing occurs.

Litho, the documentation generation engine powering the sopaco/deepwiki-rs repository, employs a deterministic filtering system to ensure only relevant source material reaches the LLM. Understanding how Litho handles file filtering for documentation generation is essential for optimizing documentation output and reducing processing overhead. The filtering logic resides primarily in the StructureExtractor component, which consults the user-configurable Config object to prune the project structure before scoring and summarization.

Configuration-Driven Filtering in Litho

All file filtering behavior in Litho originates from the Config struct defined in src/config.rs【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/config.rs#L97-L130】. These fields provide granular control over what enters the documentation pipeline:

• include_tests – Controls whether test files (*_test.*) and test directories are retained. Defaults to false.
• include_hidden – Determines if hidden files and directories (.-prefixed) are processed. Defaults to false.
• excluded_dirs – Directory names unconditionally skipped during traversal. Defaults include .litho, litho.docs, target, node_modules.
• excluded_files – Specific filenames or simple wildcards (*) to ignore, such as litho.toml, *.log, or *.md.
• excluded_extensions – File extensions permanently excluded (e.g., jpg, png, pdf, zip).
• included_extensions – Whitelist of extensions; if non-empty, only these extensions pass the filter.
• max_file_size – Size limit in bytes (default 64 KB). Files exceeding this threshold are skipped.
• Binary detection – Files detected as binary via is_binary_file_path are automatically excluded.

Directory Filtering Logic

Before descending into subdirectories, Litho invokes should_ignore_directory in src/generator/preprocess/extractors/structure_extractor.rs【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/generator/preprocess/extractors/structure_extractor.rs#L25-L45】. This method evaluates directories in strict priority order:

1. User-defined excluded directories – Matches against config.excluded_dirs.
2. Test directories – Identified by is_test_directory; skipped if include_tests is false.
3. Hidden directories – Names beginning with . are skipped if include_hidden is false.

If any check matches, the directory is pruned entirely, preventing recursive traversal and eliminating all nested files from documentation consideration.

File Filtering Implementation

Individual files undergo rigorous validation through should_ignore_file in src/generator/preprocess/extractors/structure_extractor.rs【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/generator/preprocess/extractors/structure_extractor.rs#L49-L84】. The function applies the following sequential checks:

Step	Validation	Implementation Detail
Excluded file patterns	Matches config.excluded_files (supports * wildcards)	Drops generated files, lockfiles, and documentation sources
Excluded extensions	Case-insensitive comparison against config.excluded_extensions	Filters images, archives, and binaries by extension
Included extensions whitelist	If config.included_extensions is non-empty, only matching extensions pass	Enables strict source-only documentation
Test file detection	is_test_file heuristic; skipped when include_tests is false	Excludes unit and integration tests
Hidden file detection	Names prefixed with . skipped when include_hidden is false	Removes configuration and cache files
Size limit	metadata.len() > config.max_file_size	Prevents processing of oversized files (default 64 KB)
Binary content detection	is_binary_file_path checks for null bytes	Avoids LLM processing of non-textual data

Files surviving all eight stages are admitted to the ProjectStructure model, scored for importance, and marked as is_core if their relevance exceeds 0.5. Only these core files proceed to LLM summarization.

Practical Example: Customizing Litho File Filters

Configure Litho via a litho.toml file in your project root to customize file filtering behavior:

[default]

# Include test files and hidden files for comprehensive documentation

include_tests = true
include_hidden = true

# Exclude specific generated directories

excluded_dirs = ["generated", "docs", "dist"]

# Document only Rust and TypeScript source files

included_extensions = ["rs", "ts", "tsx"]

# Increase file size limit to 100 KB

max_file_size = 102400

Execute Litho with your custom configuration:

deepwiki-rs -p ./my-project --profile default

During execution, StructureExtractor will traverse ./my-project, skip directories named generated, docs, or dist, ignore all files except those ending in .rs, .ts, or .tsx, and process hidden files and tests due to the boolean flags. Files exceeding 100 KB or detected as binary will still be excluded regardless of extension.

Summary

• Litho file filtering for documentation generation operates through a hierarchical pipeline in StructureExtractor that evaluates directories before files.
• Configuration options in src/config.rs control inclusion of tests, hidden files, specific extensions, file sizes, and binary detection.
• Directory traversal uses should_ignore_directory to prune excluded, test, and hidden directories early, preventing unnecessary file system access.
• File validation uses should_ignore_file to apply eight sequential checks including pattern matching, extension whitelisting, size limits, and binary heuristics.
• Only files marked as is_core (importance score > 0.5) after filtering proceed to LLM-driven documentation generation.

Frequently Asked Questions

How do I exclude specific directories from Litho documentation generation?

Add the directory names to the excluded_dirs array in your litho.toml configuration file. These directories are checked in should_ignore_directory within src/generator/preprocess/extractors/structure_extractor.rs, causing Litho to skip them entirely during the initial project scan.

Can Litho process test files and hidden files?

Yes, by setting include_tests = true and include_hidden = true in your configuration. By default, both are set to false, causing should_ignore_file and should_ignore_directory to skip files matching test patterns (*_test.*) or hidden names (. prefix). Enabling these flags allows test suites and configuration files to be documented.

What is the default file size limit in Litho, and how do I change it?

The default max_file_size is 64 KB (65536 bytes). Files exceeding this limit are skipped by should_ignore_file in src/generator/preprocess/extractors/structure_extractor.rs. To increase the limit, set max_file_size in your litho.toml to your desired byte value (e.g., max_file_size = 102400 for 100 KB).

How does Litho detect and exclude binary files?

Litho uses the is_binary_file_path helper function defined in src/utils/file_utils.rs to inspect file content for null bytes and other binary indicators. This check runs as the final step in should_ignore_file after extension and size filtering, ensuring that images, executables, and archives never reach the LLM processing stage regardless of their file extension.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}