# How Litho Integrates External Documentation as Knowledge Sources

> Integrate external documentation into Litho. Discover how Litho ingests Markdown PDF JSON YAML SQL and text files for agent consumption and efficient documentation generation.

- Repository: [Sopaco/deepwiki-rs](https://github.com/sopaco/deepwiki-rs)
- Tags: how-to-guide
- Published: 2026-02-16

---

**Yes, Litho can ingest Markdown, PDF, JSON, YAML, SQL, and plain-text files as external knowledge sources, automatically chunking and caching them for agent consumption during documentation generation.**

The `sopaco/deepwiki-rs` engine, codenamed Litho, provides a robust pipeline to integrate external documentation as knowledge sources within its multi-agent generation workflow. By configuring the `knowledge.local_docs` section in your TOML configuration, you can mount existing technical documentation, API specifications, architectural decisions, or any structured text directly into the context available to research agents.

## Architectural Overview of External Knowledge Integration

Litho separates knowledge ingestion from generation through three core layers: configuration, processing, and runtime access.

### Configuration Layer

The integration starts with `KnowledgeConfig` defined in `src/config.rs#L26-L30`, which holds the optional `local_docs` section. When enabled, `LocalDocsConfig` (`src/config.rs#L69-L79`) captures:

- **Enablement flags** and cache directories
- **Document categories** via `DocumentCategory` (`src/config.rs#L94-L111`)
- **Target agent routing** to control which agents receive specific knowledge
- **Chunking strategies** (semantic, paragraph, or fixed-size)

### Ingestion and Processing Pipeline

The `LocalDocs` integration in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs) handles the heavy lifting:

1. **File Discovery**: Expands glob patterns defined in each `DocumentCategory` to locate Markdown, PDF, YAML, JSON, SQL, or plain-text files
2. **Metadata Capture**: Creates `LocalDocMetadata` entries tracking file paths, modification times, and categories
3. **Content Chunking**: Uses `DocumentChunker` (`src/integrations/local_docs.rs#L9-L35`) to split large documents based on `ChunkingConfig::min_size_for_chunking`, applying semantic (section-aware) splitting for Markdown when specified
4. **Caching**: Persists processed chunks to `LocalDocsConfig.cache_dir` to avoid re-processing unchanged files when `watch_for_changes` is enabled

### Optional Remote Sync

For documentation stored in remote repositories, the `KnowledgeSyncer` in `src/integrations/knowledge_sync.rs#L21-L32` can pull Git repositories or HTTP endpoints into the local cache before the LocalDocs pipeline runs.

### Runtime Access

During generation, agents access prepared knowledge through `Generator::load_external_knowledge_by_categories` in `src/generator/context.rs#L64-L66`. This method returns `DocumentChunk` objects filtered by category, allowing agents to reference business context, API contracts, or architectural decisions during documentation generation.

## Configuring External Knowledge Sources

Enable external documentation integration by adding a `knowledge.local_docs` section to your Litho configuration file:

```toml
[knowledge.local_docs]
enabled = true
cache_dir = "./.litho_cache"
watch_for_changes = true

[[knowledge.local_docs.categories]]
name = "architecture"
description = "High-level system architecture diagrams and rationale"
paths = ["./docs/architecture/**/*.md", "./external/arch/*.pdf"]
target_agents = ["architectural_researcher"]
chunking = { strategy = "semantic", max_chunk_size = 10000 }

[[knowledge.local_docs.categories]]
name = "api"
description = "OpenAPI specs and API reference docs"
paths = ["./api/**/*.yaml", "./api/**/*.json"]
target_agents = ["boundary_analyzer"]

```

This configuration instructs Litho to scan the specified glob patterns, process Markdown and PDF files for the architecture category, and YAML/JSON files for the API category, routing each to their respective target agents.

## Running Litho with External Documentation

Execute the generator with your configuration:

```bash
deepwiki-rs --config litho-example.toml generate --output ./docs/generated

```

Litho performs the following sequence:

1. Syncs remote knowledge sources if `knowledge_sync` is configured
2. Discovers and chunks local documents according to your category definitions
3. Caches processed chunks to minimize subsequent startup times
4. Injects the prepared knowledge into the multi-agent generation pipeline

## Accessing Knowledge in Custom Agents

Agents retrieve external knowledge using the generator context:

```rust
use crate::generator::context::GeneratorContext;

async fn my_agent(context: &mut GeneratorContext) -> anyhow::Result<()> {
    // Load all external knowledge for the "api" category
    let api_chunks = context
        .load_external_knowledge_by_categories(&["api".to_string()], None)
        .await?;

    // Process the chunks
    let summary = summarize_chunks(&api_chunks);
    println!("API Summary:\n{}", summary);
    Ok(())
}

```

The `load_external_knowledge_by_categories` method defined in `src/generator/context.rs#L64-L66` returns a vector of `DocumentChunk` objects prepared by the `LocalDocs` pipeline, enabling agents to cite or reason over external material during generation.

## Summary

- **Litho supports multiple formats**: Markdown, PDF, YAML, JSON, SQL, and plain-text files can all serve as external knowledge sources.
- **Configuration-driven**: The `knowledge.local_docs` section in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs) controls enablement, caching, and category definitions.
- **Intelligent processing**: The `LocalDocs` integration ([`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs)) handles file discovery, metadata extraction, and configurable chunking strategies.
- **Agent-specific routing**: `DocumentCategory` configurations route specific knowledge to designated agents via `target_agents` lists.
- **Runtime integration**: Agents access prepared knowledge through `load_external_knowledge_by_categories` in [`src/generator/context.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/context.rs).

## Frequently Asked Questions

### What file formats does Litho support for external documentation?

Litho accepts Markdown, PDF, plain-text, SQL, YAML, and JSON files. The `LocalDocs` pipeline in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs) discovers files based on glob patterns defined in your `DocumentCategory` configurations, regardless of format, and processes them through the appropriate chunking strategies.

### How does Litho handle large documentation files?

Large documents are automatically split into manageable chunks according to `ChunkingConfig` settings. The `DocumentChunker` in [`src/integrations/local_docs.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/local_docs.rs) supports semantic chunking (section-aware for Markdown), paragraph-based splitting, or fixed-size chunking. Documents exceeding `min_size_for_chunking` are processed accordingly, with each chunk inheriting the category's `target_agents` routing.

### Can I sync external documentation from remote repositories?

Yes. The optional `KnowledgeSyncer` in [`src/integrations/knowledge_sync.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/integrations/knowledge_sync.rs) can pull documentation from remote Git repositories or HTTP endpoints into a local cache before the `LocalDocs` pipeline processes them. Configure this in your TOML file to ensure remote documentation stays synchronized with your generation workflow.

### Which agents can access the external knowledge sources?

Access is controlled per category via the `target_agents` field in `DocumentCategory` configurations defined in [`src/config.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/config.rs). When agents execute, they call `load_external_knowledge_by_categories` from [`src/generator/context.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/context.rs) to retrieve only the chunks relevant to their assigned categories, ensuring agents receive context specific to their responsibilities.