# How Litho Handles File Filtering for Documentation Generation: A Deep Dive into the StructureExtractor

> Discover how Litho filters files for documentation generation using its StructureExtractor pipeline. Learn about custom rules for tests, hidden files, extensions, and binary detection before AI processing.

- Repository: [Sopaco/deepwiki-rs](https://github.com/sopaco/deepwiki-rs)
- Tags: deep-dive
- Published: 2026-02-16

---

**Litho filters files and directories through a configurable multi-layer pipeline in `StructureExtractor` that applies user-defined rules for tests, hidden files, extensions, and binary detection before any AI processing occurs.**

Litho, the documentation generation engine powering the `sopaco/deepwiki-rs` repository, employs a deterministic filtering system to ensure only relevant source material reaches the LLM. Understanding how Litho handles file filtering for documentation generation is essential for optimizing documentation output and reducing processing overhead. The filtering logic resides primarily in the `StructureExtractor` component, which consults the user-configurable `Config` object to prune the project structure before scoring and summarization.

## Configuration-Driven Filtering in Litho

All file filtering behavior in Litho originates from the `Config` struct defined in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs)【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/config.rs#L97-L130】. These fields provide granular control over what enters the documentation pipeline:

- **`include_tests`** – Controls whether test files (`*_test.*`) and test directories are retained. Defaults to `false`.
- **`include_hidden`** – Determines if hidden files and directories (`.`-prefixed) are processed. Defaults to `false`.
- **`excluded_dirs`** – Directory names unconditionally skipped during traversal. Defaults include `.litho`, `litho.docs`, `target`, `node_modules`.
- **`excluded_files`** – Specific filenames or simple wildcards (`*`) to ignore, such as [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml), `*.log`, or `*.md`.
- **`excluded_extensions`** – File extensions permanently excluded (e.g., `jpg`, `png`, `pdf`, `zip`).
- **`included_extensions`** – Whitelist of extensions; if non-empty, only these extensions pass the filter.
- **`max_file_size`** – Size limit in bytes (default `64 KB`). Files exceeding this threshold are skipped.
- **Binary detection** – Files detected as binary via `is_binary_file_path` are automatically excluded.

## Directory Filtering Logic

Before descending into subdirectories, Litho invokes `should_ignore_directory` in [`src/generator/preprocess/extractors/structure_extractor.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/extractors/structure_extractor.rs)【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/generator/preprocess/extractors/structure_extractor.rs#L25-L45】. This method evaluates directories in strict priority order:

1. **User-defined excluded directories** – Matches against `config.excluded_dirs`.
2. **Test directories** – Identified by `is_test_directory`; skipped if `include_tests` is `false`.
3. **Hidden directories** – Names beginning with `.` are skipped if `include_hidden` is `false`.

If any check matches, the directory is pruned entirely, preventing recursive traversal and eliminating all nested files from documentation consideration.

## File Filtering Implementation

Individual files undergo rigorous validation through `should_ignore_file` in [`src/generator/preprocess/extractors/structure_extractor.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/extractors/structure_extractor.rs)【/cache/repos/github.com/sopaco/deepwiki-rs/main/src/generator/preprocess/extractors/structure_extractor.rs#L49-L84】. The function applies the following sequential checks:

| Step | Validation | Implementation Detail |
|------|------------|----------------------|
| **Excluded file patterns** | Matches `config.excluded_files` (supports `*` wildcards) | Drops generated files, lockfiles, and documentation sources |
| **Excluded extensions** | Case-insensitive comparison against `config.excluded_extensions` | Filters images, archives, and binaries by extension |
| **Included extensions whitelist** | If `config.included_extensions` is non-empty, only matching extensions pass | Enables strict source-only documentation |
| **Test file detection** | `is_test_file` heuristic; skipped when `include_tests` is `false` | Excludes unit and integration tests |
| **Hidden file detection** | Names prefixed with `.` skipped when `include_hidden` is `false` | Removes configuration and cache files |
| **Size limit** | `metadata.len() > config.max_file_size` | Prevents processing of oversized files (default 64 KB) |
| **Binary content detection** | `is_binary_file_path` checks for null bytes | Avoids LLM processing of non-textual data |

Files surviving all eight stages are admitted to the `ProjectStructure` model, scored for importance, and marked as `is_core` if their relevance exceeds `0.5`. Only these core files proceed to LLM summarization.

## Practical Example: Customizing Litho File Filters

Configure Litho via a [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml) file in your project root to customize file filtering behavior:

```toml
[default]

# Include test files and hidden files for comprehensive documentation

include_tests = true
include_hidden = true

# Exclude specific generated directories

excluded_dirs = ["generated", "docs", "dist"]

# Document only Rust and TypeScript source files

included_extensions = ["rs", "ts", "tsx"]

# Increase file size limit to 100 KB

max_file_size = 102400

```

Execute Litho with your custom configuration:

```bash
deepwiki-rs -p ./my-project --profile default

```

During execution, `StructureExtractor` will traverse `./my-project`, skip directories named `generated`, `docs`, or `dist`, ignore all files except those ending in `.rs`, `.ts`, or `.tsx`, and process hidden files and tests due to the boolean flags. Files exceeding 100 KB or detected as binary will still be excluded regardless of extension.

## Summary

- **Litho file filtering for documentation generation** operates through a hierarchical pipeline in `StructureExtractor` that evaluates directories before files.
- Configuration options in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs) control inclusion of tests, hidden files, specific extensions, file sizes, and binary detection.
- Directory traversal uses `should_ignore_directory` to prune excluded, test, and hidden directories early, preventing unnecessary file system access.
- File validation uses `should_ignore_file` to apply eight sequential checks including pattern matching, extension whitelisting, size limits, and binary heuristics.
- Only files marked as `is_core` (importance score > 0.5) after filtering proceed to LLM-driven documentation generation.

## Frequently Asked Questions

### How do I exclude specific directories from Litho documentation generation?

Add the directory names to the `excluded_dirs` array in your [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml) configuration file. These directories are checked in `should_ignore_directory` within [`src/generator/preprocess/extractors/structure_extractor.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/extractors/structure_extractor.rs), causing Litho to skip them entirely during the initial project scan.

### Can Litho process test files and hidden files?

Yes, by setting `include_tests = true` and `include_hidden = true` in your configuration. By default, both are set to `false`, causing `should_ignore_file` and `should_ignore_directory` to skip files matching test patterns (`*_test.*`) or hidden names (`.` prefix). Enabling these flags allows test suites and configuration files to be documented.

### What is the default file size limit in Litho, and how do I change it?

The default `max_file_size` is 64 KB (65536 bytes). Files exceeding this limit are skipped by `should_ignore_file` in [`src/generator/preprocess/extractors/structure_extractor.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/extractors/structure_extractor.rs). To increase the limit, set `max_file_size` in your [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml) to your desired byte value (e.g., `max_file_size = 102400` for 100 KB).

### How does Litho detect and exclude binary files?

Litho uses the `is_binary_file_path` helper function defined in [`src/utils/file_utils.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/utils/file_utils.rs) to inspect file content for null bytes and other binary indicators. This check runs as the final step in `should_ignore_file` after extension and size filtering, ensuring that images, executables, and archives never reach the LLM processing stage regardless of their file extension.