How to Configure Local Documentation Integration in Litho

Configure local documentation integration in Litho by enabling the knowledge.local_docs section in litho.toml, defining document categories with glob patterns, and running deepwiki-rs sync-knowledge to ingest PDFs, Markdown, SQL schemas, and other files into the LLM knowledge base.

Litho, the documentation engine powering the deepwiki‑rs project, can ingest arbitrary local documentation files and feed them into its LLM‑driven knowledge base. This integration is controlled entirely through the knowledge.local_docs configuration section in litho.toml, allowing you to specify which files to index, how to chunk them, and which research agents should consume them.

Understanding the Local Docs Configuration Structure

The configuration is deserialized into the LocalDocsConfig struct defined in src/config.rs (lines 269‑292). This struct controls whether the integration is active, where cached metadata is stored, and how documents are organized into categories.

Core Configuration Fields

Field	Purpose	Default
enabled	Master switch for the integration	false
cache_dir	Directory for processed metadata and chunks	.litho/cache/knowledge/local_docs
watch_for_changes	Re‑process files when modification timestamps change	true
default_chunking	Global chunking rules applied unless a category overrides them	None

The default_chunking field uses the ChunkingConfig struct (lines 217‑236 in src/config.rs), which supports strategies like semantic, paragraph, or fixed_size.

Document Category Definitions

Categories allow you to group related files and apply specific processing rules. Each category is defined by the DocumentCategory struct (lines 194‑215 in src/config.rs):

• name: Identifier used in metadata and for agent filtering.
• description: Human‑readable explanation shown during sync.
• paths: Array of glob patterns (relative to project root) selecting files.
• target_agents: Optional whitelist of agents that may consume these docs. If empty, all agents receive the content.
• chunking: Optional per‑category ChunkingConfig overriding global defaults.

How Litho Processes Local Documentation

The ingestion pipeline is orchestrated by the KnowledgeSync component in src/integrations/knowledge_sync.rs (lines 56‑87). When you run a sync, Litho executes the following stages:

Configuration Parsing and Validation

On startup, Litho loads the Config struct from litho.toml. The knowledge.local_docs section is deserialized into LocalDocsConfig. If enabled is false, the entire local docs subsystem is skipped.

File Discovery and Glob Expansion

The sync_local_docs method calls LocalDocsProcessor::expand_glob_patterns in src/integrations/local_docs.rs (lines 64‑78). This resolves each glob pattern in paths to concrete file paths on disk. The processor respects the project root and handles recursive wildcards (**).

Content Extraction and Chunking

Each discovered file is processed by LocalDocsProcessor::process_file_with_chunking (lines 400‑460 in src/integrations/local_docs.rs):

1. Type Detection: The file extension is mapped to a handler (PDF, Markdown, plain text, SQL, YAML, JSON).
2. Content Extraction: PDFs are parsed via pdf_extract, Markdown is read as UTF‑8 text, and structured files (SQL, YAML) are normalized.
3. Chunking: If the document exceeds min_size_for_chunking bytes, a DocumentChunker splits it according to the configured strategy (semantic, paragraph, or fixed_size).
4. Metadata Generation: Each chunk (or the whole file if unchunked) produces a LocalDocMetadata record containing the source path, file type, modification timestamp, category name, target agents, and optional ChunkInfo.

Caching and Metadata Storage

All LocalDocMetadata objects are serialized to _metadata.json inside the cache_dir. If watch_for_changes is enabled, subsequent syncs compare file timestamps against the cache and only re‑process modified or new files. This incremental approach keeps sync times low for large documentation sets.

Step-by-Step Configuration Examples

Minimal Configuration for Markdown Files

To enable local docs and index all Markdown files in a docs/ directory:

[knowledge.local_docs]
enabled = true
watch_for_changes = true

[[knowledge.local_docs.categories]]
name = "general_docs"
description = "General project documentation"
paths = ["docs/**/*.md"]

Place this in your litho.toml and run deepwiki-rs sync-knowledge. The engine will crawl docs/, extract text from every .md file, and make it available to all agents.

Advanced Multi-Category Setup with Chunking

For larger projects, split documentation by domain and apply specific chunking strategies:

[knowledge.local_docs]
enabled = true
cache_dir = ".litho/cache/knowledge"
watch_for_changes = true

# Global default: semantic chunking for large files

[knowledge.local_docs.default_chunking]
enabled = true
strategy = "semantic"
max_chunk_size = 8000
chunk_overlap = 200
min_size_for_chunking = 10000

# Architecture documents (Markdown + PDFs)

[[knowledge.local_docs.categories]]
name = "architecture"
description = "System architecture diagrams and design docs"
paths = ["docs/architecture/**/*.md", "docs/architecture/**/*.pdf"]
target_agents = ["architecture_researcher"]

# Database schemas (SQL files) - smaller chunks for precise lookup

[[knowledge.local_docs.categories]]
name = "database"
description = "SQL schema definitions"
paths = ["db/schema/**/*.sql"]
target_agents = ["database_overview_analyzer"]

[knowledge.local_docs.categories.chunking]
enabled = true
strategy = "paragraph"
max_chunk_size = 4000
chunk_overlap = 100
min_size_for_chunking = 2000

In this setup:

• Architecture docs use semantic chunking (8000‑token chunks) and are restricted to the architecture_researcher agent.
• Database schemas use paragraph‑level chunking (4000‑token chunks) for finer granularity, accessible only to database_overview_analyzer.

Running the Knowledge Sync

After editing litho.toml, execute:

deepwiki-rs sync-knowledge

The CLI prints per‑category progress, indicating which files were processed and whether they were chunked (e.g., “✓ [architecture] docs/arch/api.md (chunked into 3 parts)”).

Programmatic Access to Local Docs

You can trigger a sync programmatically using the Rust API:

use deepwiki_rs::integrations::knowledge_sync::KnowledgeSync;
use deepwiki_rs::config::Config;

// Load configuration from litho.toml
let cfg = Config::default(); // or Config::from_file("path/to/litho.toml")

// Initialize the sync orchestrator
let sync = KnowledgeSync::new(cfg);

// Execute full sync (includes local docs)
sync.sync_all().await?;

The sync_all method internally invokes sync_local_docs, which populates the _metadata.json cache. Agents later query this metadata to retrieve relevant documentation chunks during research phases.

Summary

• Enable the integration by setting knowledge.local_docs.enabled = true in litho.toml.
• Organize documentation into categories using glob patterns (paths) and restrict access via target_agents.
• Optimize retrieval by configuring chunking strategies (semantic, paragraph, or fixed‑size) globally or per‑category.
• Sync using deepwiki-rs sync-knowledge or programmatically via KnowledgeSync::sync_all().
• Cache is stored in .litho/cache/knowledge/local_docs/_metadata.json and respects watch_for_changes for incremental updates.

Frequently Asked Questions

What file types does Litho support for local documentation?

Litho supports PDFs, Markdown (.md), plain text, SQL schemas (.sql), and structured data files like YAML and JSON. The file type is automatically detected in LocalDocsProcessor::process_file_with_chunking (lines 400‑460 in src/integrations/local_docs.rs), and appropriate extractors are applied (e.g., pdf_extract for PDFs).

How do I prevent certain agents from accessing specific documentation?

Use the target_agents field within a document category. If you specify target_agents = ["architecture_researcher"], only that agent will receive the documents; other agents will not see them. If target_agents is empty or omitted, the documents are made available to all agents. This filtering occurs during the agent consumption phase when agents query the knowledge store.

Can I use different chunking strategies for different types of documentation?

Yes. Define a chunking block inside a specific category to override the global default_chunking settings. For example, you can use semantic chunking with 8000‑token chunks for architecture PDFs, while applying paragraph chunking with 4000‑token chunks for SQL schema files. The DocumentChunker (configured via ChunkingConfig in src/config.rs) handles the splitting logic based on your per‑category or global configuration.

Where does Litho store the processed documentation metadata?

Processed metadata is stored in _metadata.json inside the directory specified by cache_dir (default: .litho/cache/knowledge/local_docs). This cache contains LocalDocMetadata records for every file and chunk, including source paths, timestamps, categories, and target agents. If watch_for_changes is enabled, Litho compares file modification times against this cache during subsequent syncs to process only new or modified files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}