The Four Stages of Litho's Documentation Generation Pipeline: A Technical Deep Dive

Litho's documentation generation pipeline consists of four sequential stages—Preprocessing, Intelligent Research & Analysis, Documentation Generation, and Verification & Enhancement—that transform raw source code into structured C4-model technical documentation.

The sopaco/deepwiki-rs repository, commonly referred to as Litho, implements a Rust-based documentation generation engine that executes a four-stage documentation generation pipeline. This pipeline progressively refines repository data into comprehensive architectural documentation. Each stage operates as a dedicated module with distinct responsibilities for data transformation, AI-powered analysis, and quality assurance.

Stage 1: Preprocessing – Repository Discovery and Initial Extraction

The Preprocessing stage performs the initial discovery and extraction of raw codebase metadata. This phase scans the target repository to identify file structures, extract code comments, and map module relationships.

The PreProcessAgent struct in src/generator/preprocess/mod.rs orchestrates this discovery phase. It populates an internal memory store with raw insights about the codebase architecture, creating the foundational data layer that subsequent stages rely upon for context-aware analysis. This extraction captures syntax structures, import dependencies, and documentation comments without applying semantic interpretation.

Stage 2: Intelligent Research and Analysis – AI-Powered Architectural Reasoning

During the Intelligent Research & Analysis stage, the pipeline employs multiple AI agents to reason over the preprocessed data. This phase transforms raw code facts into structured architectural knowledge by inferring system intent and detecting domain boundaries.

The ResearchOrchestrator in src/generator/research/orchestrator.rs coordinates specialized research agents including the SystemContextResearcher, DomainModulesDetector, and ArchitectureResearcher. These agents analyze the preprocessed memory state to identify architectural patterns, system boundaries, and workflow relationships. The orchestrator aggregates these insights into a comprehensive architectural model that informs the subsequent documentation composition.

Stage 3: Documentation Generation – C4 Model Composition

The Documentation Generation stage synthesizes researched insights into polished technical documentation following the C4 model (Context, Containers, Components, Code). This phase converts abstract architectural knowledge into readable, structured markdown documents.

The DocumentationComposer in src/generator/compose/mod.rs drives composition agents such as the OverviewEditor, ArchitectureEditor, and WorkflowEditor. These components assemble overview documents, architecture diagrams, workflow descriptions, and module-level specifications into a cohesive documentation tree. The composer structures the output to reflect hierarchical architectural views, ensuring that generated documents maintain consistency with the C4 modeling standards.

Stage 4: Verification and Enhancement – Quality Assurance and Output

The final Verification & Enhancement stage validates the generated documentation for correctness, completeness, and syntactic accuracy. This quality assurance phase ensures that the output is ready for production use.

The SummaryOutlet in src/generator/outlet/summary_outlet.rs performs Mermaid diagram syntax verification, integrity checks, and quality assessments. This stage repairs any detected diagram syntax issues and produces the final output files along with a comprehensive quality summary report. The verification process ensures that all generated architectural diagrams render correctly and that the documentation accurately reflects the analyzed codebase.

Executing the Litho Pipeline

The complete four-stage documentation generation pipeline is orchestrated by the launch function in src/generator/workflow.rs. This coordinator sequentially invokes each stage while recording timing metrics and managing the shared context state between phases.

You can execute this pipeline via command line or integrate it programmatically into Rust applications.

Command Line Execution

Run the complete pipeline against a target repository:

# Generate documentation for the current project

deepwiki-rs -p ./my-project -o ./docs

# The command executes all four stages sequentially:

# 1. Preprocessing (repository scanning)

# 2. Research & Analysis (AI architectural reasoning)

# 3. Documentation Generation (C4 model composition)

# 4. Verification (quality assurance)

Programmatic Integration

Invoke the pipeline from within another Rust binary:

use deepwiki_rs::config::Config;
use deepwiki_rs::generator::workflow::launch;

// Load configuration from TOML or use defaults
let config = Config::load_from_path("Litho.toml")?;

// Execute the complete four-stage pipeline
launch(&config).await?;

Debugging Individual Stages

For custom workflows or debugging, you can execute individual stages independently:

use deepwiki_rs::generator::preprocess::PreProcessAgent;
use deepwiki_rs::generator::research::orchestrator::ResearchOrchestrator;
use deepwiki_rs::generator::compose::DocumentationComposer;
use deepwiki_rs::generator::outlet::SummaryOutlet;

// Stage 1: Preprocessing
let preprocess = PreProcessAgent::new();
preprocess.execute(context.clone()).await?;

// Stage 2: Research & Analysis
let research = ResearchOrchestrator::default();
research.execute_research_pipeline(&context).await?;

// Stage 3: Documentation Generation
let mut doc_tree = DocumentTree::new(&context.config.target_language);
DocumentationComposer::default()
    .execute(&context, &mut doc_tree)
    .await?;

// Stage 4: Verification
SummaryOutlet::new().save(&context).await?;

Summary

Litho's documentation generation pipeline processes raw source code through four distinct transformational stages:

• Preprocessing extracts repository structure and metadata using PreProcessAgent in src/generator/preprocess/mod.rs
• Intelligent Research & Analysis applies AI agents via ResearchOrchestrator in src/generator/research/orchestrator.rs to infer architectural patterns
• Documentation Generation composes C4-model documentation using DocumentationComposer in src/generator/compose/mod.rs
• Verification & Enhancement validates output quality through SummaryOutlet in src/generator/outlet/summary_outlet.rs

The launch function in src/generator/workflow.rs coordinates these stages into a cohesive workflow that transforms codebases into production-ready technical documentation.

Frequently Asked Questions

What is Litho and how does it relate to deepwiki-rs?

Litho is the documentation generation engine implemented in the sopaco/deepwiki-rs repository. Written in Rust, Litho analyzes source code repositories and automatically generates C4-model architectural documentation. The tool processes code through its four-stage pipeline to produce structured markdown files, architecture diagrams, and quality reports without manual intervention.

How does the Preprocessing stage handle different programming languages?

The PreProcessAgent in src/generator/preprocess/mod.rs performs syntax-agnostic repository discovery combined with language-specific parsing. It extracts universal structures such as file hierarchies, import dependencies, and documentation comments across multiple languages. The stage stores these raw insights in an internal memory context that standardizes the representation, allowing subsequent AI research agents to reason over the codebase regardless of the original programming language.

Can I run individual stages of the pipeline independently?

Yes, while the launch function in src/generator/workflow.rs executes all four stages sequentially, you can invoke individual stages programmatically for debugging or custom workflows. Instantiate PreProcessAgent, ResearchOrchestrator, DocumentationComposer, or SummaryOutlet directly and call their respective execution methods (such as execute(), execute_research_pipeline(), or save()) while passing the shared context object between stages.

What output formats does Litho generate?

Litho produces Markdown-based technical documentation structured according to the C4 model, including Context, Container, Component, and Code-level diagrams. The DocumentationComposer generates Mermaid diagram syntax for architectural visualizations, while the SummaryOutlet produces quality assurance reports and final output files. All documentation is emitted as standard markdown files suitable for publishing to wikis, GitHub repositories, or static site generators.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}