# The Four Stages of Litho's Documentation Generation Pipeline: A Technical Deep Dive > Explore the four stages of Litho's documentation generation pipeline: Preprocessing, Research, Generation, and Verification. Transform code into C4 model documentation. - Repository: [Sopaco/deepwiki-rs](https://github.com/sopaco/deepwiki-rs) - Tags: deep-dive - Published: 2026-02-16 --- **Litho's documentation generation pipeline consists of four sequential stages—Preprocessing, Intelligent Research & Analysis, Documentation Generation, and Verification & Enhancement—that transform raw source code into structured C4-model technical documentation.** The `sopaco/deepwiki-rs` repository, commonly referred to as **Litho**, implements a Rust-based documentation generation engine that executes a **four-stage documentation generation pipeline**. This pipeline progressively refines repository data into comprehensive architectural documentation. Each stage operates as a dedicated module with distinct responsibilities for data transformation, AI-powered analysis, and quality assurance. ## Stage 1: Preprocessing – Repository Discovery and Initial Extraction The **Preprocessing** stage performs the initial discovery and extraction of raw codebase metadata. This phase scans the target repository to identify file structures, extract code comments, and map module relationships. The `PreProcessAgent` struct in **[`src/generator/preprocess/mod.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/mod.rs)** orchestrates this discovery phase. It populates an internal memory store with raw insights about the codebase architecture, creating the foundational data layer that subsequent stages rely upon for context-aware analysis. This extraction captures syntax structures, import dependencies, and documentation comments without applying semantic interpretation. ## Stage 2: Intelligent Research and Analysis – AI-Powered Architectural Reasoning During the **Intelligent Research & Analysis** stage, the pipeline employs multiple AI agents to reason over the preprocessed data. This phase transforms raw code facts into structured architectural knowledge by inferring system intent and detecting domain boundaries. The `ResearchOrchestrator` in **[`src/generator/research/orchestrator.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/research/orchestrator.rs)** coordinates specialized research agents including the `SystemContextResearcher`, `DomainModulesDetector`, and `ArchitectureResearcher`. These agents analyze the preprocessed memory state to identify architectural patterns, system boundaries, and workflow relationships. The orchestrator aggregates these insights into a comprehensive architectural model that informs the subsequent documentation composition. ## Stage 3: Documentation Generation – C4 Model Composition The **Documentation Generation** stage synthesizes researched insights into polished technical documentation following the **C4 model** (Context, Containers, Components, Code). This phase converts abstract architectural knowledge into readable, structured markdown documents. The `DocumentationComposer` in **[`src/generator/compose/mod.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/compose/mod.rs)** drives composition agents such as the `OverviewEditor`, `ArchitectureEditor`, and `WorkflowEditor`. These components assemble overview documents, architecture diagrams, workflow descriptions, and module-level specifications into a cohesive documentation tree. The composer structures the output to reflect hierarchical architectural views, ensuring that generated documents maintain consistency with the C4 modeling standards. ## Stage 4: Verification and Enhancement – Quality Assurance and Output The final **Verification & Enhancement** stage validates the generated documentation for correctness, completeness, and syntactic accuracy. This quality assurance phase ensures that the output is ready for production use. The `SummaryOutlet` in **[`src/generator/outlet/summary_outlet.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/outlet/summary_outlet.rs)** performs **Mermaid diagram syntax verification**, integrity checks, and quality assessments. This stage repairs any detected diagram syntax issues and produces the final output files along with a comprehensive quality summary report. The verification process ensures that all generated architectural diagrams render correctly and that the documentation accurately reflects the analyzed codebase. ## Executing the Litho Pipeline The complete **four-stage documentation generation pipeline** is orchestrated by the `launch` function in **[`src/generator/workflow.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/workflow.rs)**. This coordinator sequentially invokes each stage while recording timing metrics and managing the shared context state between phases. You can execute this pipeline via command line or integrate it programmatically into Rust applications. ### Command Line Execution Run the complete pipeline against a target repository: ```bash # Generate documentation for the current project deepwiki-rs -p ./my-project -o ./docs # The command executes all four stages sequentially: # 1. Preprocessing (repository scanning) # 2. Research & Analysis (AI architectural reasoning) # 3. Documentation Generation (C4 model composition) # 4. Verification (quality assurance) ``` ### Programmatic Integration Invoke the pipeline from within another Rust binary: ```rust use deepwiki_rs::config::Config; use deepwiki_rs::generator::workflow::launch; // Load configuration from TOML or use defaults let config = Config::load_from_path("Litho.toml")?; // Execute the complete four-stage pipeline launch(&config).await?; ``` ### Debugging Individual Stages For custom workflows or debugging, you can execute individual stages independently: ```rust use deepwiki_rs::generator::preprocess::PreProcessAgent; use deepwiki_rs::generator::research::orchestrator::ResearchOrchestrator; use deepwiki_rs::generator::compose::DocumentationComposer; use deepwiki_rs::generator::outlet::SummaryOutlet; // Stage 1: Preprocessing let preprocess = PreProcessAgent::new(); preprocess.execute(context.clone()).await?; // Stage 2: Research & Analysis let research = ResearchOrchestrator::default(); research.execute_research_pipeline(&context).await?; // Stage 3: Documentation Generation let mut doc_tree = DocumentTree::new(&context.config.target_language); DocumentationComposer::default() .execute(&context, &mut doc_tree) .await?; // Stage 4: Verification SummaryOutlet::new().save(&context).await?; ``` ## Summary Litho's documentation generation pipeline processes raw source code through four distinct transformational stages: - **Preprocessing** extracts repository structure and metadata using `PreProcessAgent` in [`src/generator/preprocess/mod.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/mod.rs) - **Intelligent Research & Analysis** applies AI agents via `ResearchOrchestrator` in [`src/generator/research/orchestrator.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/research/orchestrator.rs) to infer architectural patterns - **Documentation Generation** composes C4-model documentation using `DocumentationComposer` in [`src/generator/compose/mod.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/compose/mod.rs) - **Verification & Enhancement** validates output quality through `SummaryOutlet` in [`src/generator/outlet/summary_outlet.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/outlet/summary_outlet.rs) The `launch` function in [`src/generator/workflow.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/workflow.rs) coordinates these stages into a cohesive workflow that transforms codebases into production-ready technical documentation. ## Frequently Asked Questions ### What is Litho and how does it relate to deepwiki-rs? **Litho** is the documentation generation engine implemented in the `sopaco/deepwiki-rs` repository. Written in Rust, Litho analyzes source code repositories and automatically generates C4-model architectural documentation. The tool processes code through its four-stage pipeline to produce structured markdown files, architecture diagrams, and quality reports without manual intervention. ### How does the Preprocessing stage handle different programming languages? The `PreProcessAgent` in [`src/generator/preprocess/mod.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/preprocess/mod.rs) performs **syntax-agnostic repository discovery** combined with language-specific parsing. It extracts universal structures such as file hierarchies, import dependencies, and documentation comments across multiple languages. The stage stores these raw insights in an internal memory context that standardizes the representation, allowing subsequent AI research agents to reason over the codebase regardless of the original programming language. ### Can I run individual stages of the pipeline independently? Yes, while the `launch` function in [`src/generator/workflow.rs`](https://github.com/sopaco/deepwiki-rs/blob/main/src/generator/workflow.rs) executes all four stages sequentially, you can invoke individual stages programmatically for debugging or custom workflows. Instantiate `PreProcessAgent`, `ResearchOrchestrator`, `DocumentationComposer`, or `SummaryOutlet` directly and call their respective execution methods (such as `execute()`, `execute_research_pipeline()`, or `save()`) while passing the shared context object between stages. ### What output formats does Litho generate? Litho produces **Markdown-based technical documentation** structured according to the C4 model, including Context, Container, Component, and Code-level diagrams. The `DocumentationComposer` generates Mermaid diagram syntax for architectural visualizations, while the `SummaryOutlet` produces quality assurance reports and final output files. All documentation is emitted as standard markdown files suitable for publishing to wikis, GitHub repositories, or static site generators.