# How to Configure Local Documentation Integration in Litho

> Configure local documentation integration in Litho by enabling knowledge local docs and syncing files like PDFs and Markdown. Learn how to ingest custom data into your LLM knowledge base.

- Repository: [Sopaco/deepwiki-rs](https://github.com/sopaco/deepwiki-rs)
- Tags: how-to-guide
- Published: 2026-02-16

---

**Configure local documentation integration in Litho by enabling the `knowledge.local_docs` section in [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml), defining document categories with glob patterns, and running `deepwiki-rs sync-knowledge` to ingest PDFs, Markdown, SQL schemas, and other files into the LLM knowledge base.**

Litho, the documentation engine powering the *deepwiki‑rs* project, can ingest arbitrary local documentation files and feed them into its LLM‑driven knowledge base. This integration is controlled entirely through the `knowledge.local_docs` configuration section in [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml), allowing you to specify which files to index, how to chunk them, and which research agents should consume them.

## Understanding the Local Docs Configuration Structure

The configuration is deserialized into the `LocalDocsConfig` struct defined in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs) (lines 269‑292). This struct controls whether the integration is active, where cached metadata is stored, and how documents are organized into categories.

### Core Configuration Fields

| Field | Purpose | Default |
|-------|---------|---------|
| `enabled` | Master switch for the integration | `false` |
| `cache_dir` | Directory for processed metadata and chunks | `.litho/cache/knowledge/local_docs` |
| `watch_for_changes` | Re‑process files when modification timestamps change | `true` |
| `default_chunking` | Global chunking rules applied unless a category overrides them | `None` |

The `default_chunking` field uses the `ChunkingConfig` struct (lines 217‑236 in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs)), which supports strategies like `semantic`, `paragraph`, or `fixed_size`.

### Document Category Definitions

Categories allow you to group related files and apply specific processing rules. Each category is defined by the `DocumentCategory` struct (lines 194‑215 in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs)):

- **`name`**: Identifier used in metadata and for agent filtering.
- **`description`**: Human‑readable explanation shown during sync.
- **`paths`**: Array of glob patterns (relative to project root) selecting files.
- **`target_agents`**: Optional whitelist of agents that may consume these docs. If empty, all agents receive the content.
- **`chunking`**: Optional per‑category `ChunkingConfig` overriding global defaults.

## How Litho Processes Local Documentation

The ingestion pipeline is orchestrated by the `KnowledgeSync` component in [`src/integrations/knowledge_sync.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/knowledge_sync.rs) (lines 56‑87). When you run a sync, Litho executes the following stages:

### Configuration Parsing and Validation

On startup, Litho loads the `Config` struct from [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml). The `knowledge.local_docs` section is deserialized into `LocalDocsConfig`. If `enabled` is `false`, the entire local docs subsystem is skipped.

### File Discovery and Glob Expansion

The `sync_local_docs` method calls `LocalDocsProcessor::expand_glob_patterns` in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs) (lines 64‑78). This resolves each glob pattern in `paths` to concrete file paths on disk. The processor respects the project root and handles recursive wildcards (`**`).

### Content Extraction and Chunking

Each discovered file is processed by `LocalDocsProcessor::process_file_with_chunking` (lines 400‑460 in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs)):

1. **Type Detection**: The file extension is mapped to a handler (PDF, Markdown, plain text, SQL, YAML, JSON).
2. **Content Extraction**: PDFs are parsed via `pdf_extract`, Markdown is read as UTF‑8 text, and structured files (SQL, YAML) are normalized.
3. **Chunking**: If the document exceeds `min_size_for_chunking` bytes, a `DocumentChunker` splits it according to the configured strategy (`semantic`, `paragraph`, or `fixed_size`).
4. **Metadata Generation**: Each chunk (or the whole file if unchunked) produces a `LocalDocMetadata` record containing the source path, file type, modification timestamp, category name, target agents, and optional `ChunkInfo`.

### Caching and Metadata Storage

All `LocalDocMetadata` objects are serialized to [`_metadata.json`](https://github.com/sopaco/deepwiki-rs/blob/main/_metadata.json) inside the `cache_dir`. If `watch_for_changes` is enabled, subsequent syncs compare file timestamps against the cache and only re‑process modified or new files. This incremental approach keeps sync times low for large documentation sets.

## Step-by-Step Configuration Examples

### Minimal Configuration for Markdown Files

To enable local docs and index all Markdown files in a `docs/` directory:

```toml
[knowledge.local_docs]
enabled = true
watch_for_changes = true

[[knowledge.local_docs.categories]]
name = "general_docs"
description = "General project documentation"
paths = ["docs/**/*.md"]

```

Place this in your [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml) and run `deepwiki-rs sync-knowledge`. The engine will crawl `docs/`, extract text from every `.md` file, and make it available to all agents.

### Advanced Multi-Category Setup with Chunking

For larger projects, split documentation by domain and apply specific chunking strategies:

```toml
[knowledge.local_docs]
enabled = true
cache_dir = ".litho/cache/knowledge"
watch_for_changes = true

# Global default: semantic chunking for large files

[knowledge.local_docs.default_chunking]
enabled = true
strategy = "semantic"
max_chunk_size = 8000
chunk_overlap = 200
min_size_for_chunking = 10000

# Architecture documents (Markdown + PDFs)

[[knowledge.local_docs.categories]]
name = "architecture"
description = "System architecture diagrams and design docs"
paths = ["docs/architecture/**/*.md", "docs/architecture/**/*.pdf"]
target_agents = ["architecture_researcher"]

# Database schemas (SQL files) - smaller chunks for precise lookup

[[knowledge.local_docs.categories]]
name = "database"
description = "SQL schema definitions"
paths = ["db/schema/**/*.sql"]
target_agents = ["database_overview_analyzer"]

[knowledge.local_docs.categories.chunking]
enabled = true
strategy = "paragraph"
max_chunk_size = 4000
chunk_overlap = 100
min_size_for_chunking = 2000

```

In this setup:
- **Architecture** docs use semantic chunking (8000‑token chunks) and are restricted to the `architecture_researcher` agent.
- **Database** schemas use paragraph‑level chunking (4000‑token chunks) for finer granularity, accessible only to `database_overview_analyzer`.

### Running the Knowledge Sync

After editing [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml), execute:

```bash
deepwiki-rs sync-knowledge

```

The CLI prints per‑category progress, indicating which files were processed and whether they were chunked (e.g., “✓ [architecture] docs/arch/api.md (chunked into 3 parts)”).

## Programmatic Access to Local Docs

You can trigger a sync programmatically using the Rust API:

```rust
use deepwiki_rs::integrations::knowledge_sync::KnowledgeSync;
use deepwiki_rs::config::Config;

// Load configuration from litho.toml
let cfg = Config::default(); // or Config::from_file("path/to/litho.toml")

// Initialize the sync orchestrator
let sync = KnowledgeSync::new(cfg);

// Execute full sync (includes local docs)
sync.sync_all().await?;

```

The `sync_all` method internally invokes `sync_local_docs`, which populates the [`_metadata.json`](https://github.com/sopaco/deepwiki-rs/blob/main/_metadata.json) cache. Agents later query this metadata to retrieve relevant documentation chunks during research phases.

## Summary

- **Enable** the integration by setting `knowledge.local_docs.enabled = true` in [`litho.toml`](https://github.com/sopaco/deepwiki-rs/blob/main/litho.toml).
- **Organize** documentation into categories using glob patterns (`paths`) and restrict access via `target_agents`.
- **Optimize** retrieval by configuring `chunking` strategies (semantic, paragraph, or fixed‑size) globally or per‑category.
- **Sync** using `deepwiki-rs sync-knowledge` or programmatically via `KnowledgeSync::sync_all()`.
- **Cache** is stored in [`.litho/cache/knowledge/local_docs/_metadata.json`](https://github.com/sopaco/deepwiki-rs/blob/main/.litho/cache/knowledge/local_docs/_metadata.json) and respects `watch_for_changes` for incremental updates.

## Frequently Asked Questions

### What file types does Litho support for local documentation?

Litho supports **PDFs**, **Markdown** (`.md`), **plain text**, **SQL schemas** (`.sql`), and structured data files like **YAML** and **JSON**. The file type is automatically detected in `LocalDocsProcessor::process_file_with_chunking` (lines 400‑460 in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs)), and appropriate extractors are applied (e.g., `pdf_extract` for PDFs).

### How do I prevent certain agents from accessing specific documentation?

Use the **`target_agents`** field within a document category. If you specify `target_agents = ["architecture_researcher"]`, only that agent will receive the documents; other agents will not see them. If `target_agents` is empty or omitted, the documents are made available to **all** agents. This filtering occurs during the agent consumption phase when agents query the knowledge store.

### Can I use different chunking strategies for different types of documentation?

Yes. Define a **`chunking`** block inside a specific category to override the global `default_chunking` settings. For example, you can use **semantic** chunking with 8000‑token chunks for architecture PDFs, while applying **paragraph** chunking with 4000‑token chunks for SQL schema files. The `DocumentChunker` (configured via `ChunkingConfig` in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs)) handles the splitting logic based on your per‑category or global configuration.

### Where does Litho store the processed documentation metadata?

Processed metadata is stored in **[`_metadata.json`](https://github.com/sopaco/deepwiki-rs/blob/main/_metadata.json)** inside the directory specified by `cache_dir` (default: `.litho/cache/knowledge/local_docs`). This cache contains `LocalDocMetadata` records for every file and chunk, including source paths, timestamps, categories, and target agents. If `watch_for_changes` is enabled, Litho compares file modification times against this cache during subsequent syncs to process only new or modified files.