How Litho Integrates External Documentation as Knowledge Sources

Yes, Litho can ingest Markdown, PDF, JSON, YAML, SQL, and plain-text files as external knowledge sources, automatically chunking and caching them for agent consumption during documentation generation.

The sopaco/deepwiki-rs engine, codenamed Litho, provides a robust pipeline to integrate external documentation as knowledge sources within its multi-agent generation workflow. By configuring the knowledge.local_docs section in your TOML configuration, you can mount existing technical documentation, API specifications, architectural decisions, or any structured text directly into the context available to research agents.

Architectural Overview of External Knowledge Integration

Litho separates knowledge ingestion from generation through three core layers: configuration, processing, and runtime access.

Configuration Layer

The integration starts with KnowledgeConfig defined in src/config.rs#L26-L30, which holds the optional local_docs section. When enabled, LocalDocsConfig (src/config.rs#L69-L79) captures:

• Enablement flags and cache directories
• Document categories via DocumentCategory (src/config.rs#L94-L111)
• Target agent routing to control which agents receive specific knowledge
• Chunking strategies (semantic, paragraph, or fixed-size)

Ingestion and Processing Pipeline

The LocalDocs integration in src/integrations/local_docs.rs handles the heavy lifting:

1. File Discovery: Expands glob patterns defined in each DocumentCategory to locate Markdown, PDF, YAML, JSON, SQL, or plain-text files
2. Metadata Capture: Creates LocalDocMetadata entries tracking file paths, modification times, and categories
3. Content Chunking: Uses DocumentChunker (src/integrations/local_docs.rs#L9-L35) to split large documents based on ChunkingConfig::min_size_for_chunking, applying semantic (section-aware) splitting for Markdown when specified
4. Caching: Persists processed chunks to LocalDocsConfig.cache_dir to avoid re-processing unchanged files when watch_for_changes is enabled

Optional Remote Sync

For documentation stored in remote repositories, the KnowledgeSyncer in src/integrations/knowledge_sync.rs#L21-L32 can pull Git repositories or HTTP endpoints into the local cache before the LocalDocs pipeline runs.

Runtime Access

During generation, agents access prepared knowledge through Generator::load_external_knowledge_by_categories in src/generator/context.rs#L64-L66. This method returns DocumentChunk objects filtered by category, allowing agents to reference business context, API contracts, or architectural decisions during documentation generation.

Configuring External Knowledge Sources

Enable external documentation integration by adding a knowledge.local_docs section to your Litho configuration file:

[knowledge.local_docs]
enabled = true
cache_dir = "./.litho_cache"
watch_for_changes = true

[[knowledge.local_docs.categories]]
name = "architecture"
description = "High-level system architecture diagrams and rationale"
paths = ["./docs/architecture/**/*.md", "./external/arch/*.pdf"]
target_agents = ["architectural_researcher"]
chunking = { strategy = "semantic", max_chunk_size = 10000 }

[[knowledge.local_docs.categories]]
name = "api"
description = "OpenAPI specs and API reference docs"
paths = ["./api/**/*.yaml", "./api/**/*.json"]
target_agents = ["boundary_analyzer"]

This configuration instructs Litho to scan the specified glob patterns, process Markdown and PDF files for the architecture category, and YAML/JSON files for the API category, routing each to their respective target agents.

Running Litho with External Documentation

Execute the generator with your configuration:

deepwiki-rs --config litho-example.toml generate --output ./docs/generated

Litho performs the following sequence:

1. Syncs remote knowledge sources if knowledge_sync is configured
2. Discovers and chunks local documents according to your category definitions
3. Caches processed chunks to minimize subsequent startup times
4. Injects the prepared knowledge into the multi-agent generation pipeline

Accessing Knowledge in Custom Agents

Agents retrieve external knowledge using the generator context:

use crate::generator::context::GeneratorContext;

async fn my_agent(context: &mut GeneratorContext) -> anyhow::Result<()> {
    // Load all external knowledge for the "api" category
    let api_chunks = context
        .load_external_knowledge_by_categories(&["api".to_string()], None)
        .await?;

    // Process the chunks
    let summary = summarize_chunks(&api_chunks);
    println!("API Summary:\n{}", summary);
    Ok(())
}

The load_external_knowledge_by_categories method defined in src/generator/context.rs#L64-L66 returns a vector of DocumentChunk objects prepared by the LocalDocs pipeline, enabling agents to cite or reason over external material during generation.

Summary

• Litho supports multiple formats: Markdown, PDF, YAML, JSON, SQL, and plain-text files can all serve as external knowledge sources.
• Configuration-driven: The knowledge.local_docs section in src/config.rs controls enablement, caching, and category definitions.
• Intelligent processing: The LocalDocs integration (src/integrations/local_docs.rs) handles file discovery, metadata extraction, and configurable chunking strategies.
• Agent-specific routing: DocumentCategory configurations route specific knowledge to designated agents via target_agents lists.
• Runtime integration: Agents access prepared knowledge through load_external_knowledge_by_categories in src/generator/context.rs.

Frequently Asked Questions

What file formats does Litho support for external documentation?

Litho accepts Markdown, PDF, plain-text, SQL, YAML, and JSON files. The LocalDocs pipeline in src/integrations/local_docs.rs discovers files based on glob patterns defined in your DocumentCategory configurations, regardless of format, and processes them through the appropriate chunking strategies.

How does Litho handle large documentation files?

Large documents are automatically split into manageable chunks according to ChunkingConfig settings. The DocumentChunker in src/integrations/local_docs.rs supports semantic chunking (section-aware for Markdown), paragraph-based splitting, or fixed-size chunking. Documents exceeding min_size_for_chunking are processed accordingly, with each chunk inheriting the category's target_agents routing.

Can I sync external documentation from remote repositories?

Yes. The optional KnowledgeSyncer in src/integrations/knowledge_sync.rs can pull documentation from remote Git repositories or HTTP endpoints into a local cache before the LocalDocs pipeline processes them. Configure this in your TOML file to ensure remote documentation stays synchronized with your generation workflow.

Which agents can access the external knowledge sources?

Access is controlled per category via the target_agents field in DocumentCategory configurations defined in src/config.rs. When agents execute, they call load_external_knowledge_by_categories from src/generator/context.rs to retrieve only the chunks relevant to their assigned categories, ensuring agents receive context specific to their responsibilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}