What Agents Are Involved in Litho's Preprocessing Stage? A Deep Dive into DeepWiki-RS

Litho's preprocessing stage uses two specialized agents—CodeAnalyze and RelationshipsAnalyze—that run sequentially inside the PreProcessAgent workflow to extract code insights and build architectural dependency graphs.

The sopaco/deepwiki-rs repository implements Litho, a Rust-based documentation generator that leverages LLMs to analyze codebases. Understanding the agents involved in Litho's preprocessing stage is essential for anyone extending the pipeline or debugging generation issues, as these agents determine how the system interprets your project's structure and dependencies.

The Two Agents Driving Litho's Preprocessing Stage

The preprocessing pipeline relies on two distinct agent implementations located in src/generator/preprocess/agents/. Each agent handles a specific layer of analysis, from individual file inspection to project-wide architecture mapping.

CodeAnalyze Agent (Core Source Analysis)

The CodeAnalyze agent, defined in src/generator/preprocess/agents/code_analyze.rs, performs granular analysis of individual source files identified as "core" to the project.

Its primary responsibilities include:

• Extracting language-specific interfaces, dependencies, and complexity metrics for each core file
• Enriching raw code data with LLM-generated insights about purpose and functionality
• Producing a collection of CodeInsight objects that feed downstream agents

The agent executes parallel LLM calls for each core file, making it the most computationally intensive step of the preprocessing stage.

RelationshipsAnalyze Agent (Project Architecture)

The RelationshipsAnalyze agent, implemented in src/generator/preprocess/agents/relationships_analyze.rs, operates on the aggregated output from CodeAnalyze.

This agent's responsibilities include:

• Consuming the collection of CodeInsight objects produced by the previous stage
• Compressing data when necessary to fit token limits
• Prompting the LLM to generate an architectural dependency graph representing the entire project's structure

Unlike CodeAnalyze, which runs parallel operations per-file, RelationshipsAnalyze typically executes a single optimized prompt that synthesizes all core insights into a unified relationship map.

How the Preprocessing Pipeline Orchestrates These Agents

The central PreProcessAgent defined in src/generator/preprocess/mod.rs coordinates the execution flow. It instantiates and runs the agents sequentially, handling context passing between stages.

The complete preprocessing workflow follows this structure:

PreProcessAgent
 ├─ original_document_extractor (extractor, not an agent)
 ├─ StructureExtractor (extractor, not an agent)
 ├─ CodeAnalyze                ← first agent
 └─ RelationshipsAnalyze       ← second agent

Note that extractors (such as original_document_extractor and StructureExtractor) handle data ingestion and file discovery, while agents perform LLM-powered analysis.

Unused Agent Components

The source tree contains a third agent implementation—code_purpose_analyze in src/generator/preprocess/agents/code_purpose_analyze.rs—but this component is not invoked by the current preprocessing pipeline. It exists as potential extension infrastructure but remains dormant in the standard execution path.

Running the Preprocessing Stage

You can trigger the preprocessing pipeline through the built-in CLI or programmatically via the Rust API.

CLI Execution

The preprocess sub-command initiates the full pipeline:

$ litho preprocess --project /path/to/your/repo

This command creates a GeneratorContext and executes PreProcessAgent::execute(), running both CodeAnalyze and RelationshipsAnalyze automatically.

Manual Rust Invocation

For testing or custom workflows, instantiate the agents directly:

use deepwiki_rs::generator::preprocess::PreProcessAgent;
use deepwiki_rs::generator::context::GeneratorContext;

// Assume `context` is already configured
let agent = PreProcessAgent::new();
let result = tokio::runtime::Runtime::new()
    .unwrap()
    .block_on(agent.execute(context))
    .expect("preprocessing failed");

// Access individual agent outputs
for insight in result.core_code_insights.iter() {
    println!("Analyzed: {}", insight.code_dossier.name);
}

println!("Dependencies: {:#?}", result.relationships);

Summary

• Two active agents drive Litho's preprocessing: CodeAnalyze and RelationshipsAnalyze.
• CodeAnalyze (in src/generator/preprocess/agents/code_analyze.rs) extracts file-level insights via parallel LLM calls.
• RelationshipsAnalyze (in src/generator/preprocess/agents/relationships_analyze.rs) synthesizes these into a project-wide dependency graph.
• The PreProcessAgent (in src/generator/preprocess/mod.rs) orchestrates execution, while code_purpose_analyze exists but remains unused.

Frequently Asked Questions

What is the difference between extractors and agents in Litho?

Extractors such as original_document_extractor and StructureExtractor handle mechanical data ingestion—reading files, parsing directory structures, and identifying core source files without LLM involvement. Agents like CodeAnalyze and RelationshipsAnalyze perform cognitive analysis by calling LLMs to interpret code meaning, extract dependencies, and generate architectural insights.

Why is the CodePurposeAnalyze agent not used in the current pipeline?

The code_purpose_analyze agent exists in the source tree at src/generator/preprocess/agents/code_purpose_analyze.rs but is not instantiated or executed by the PreProcessAgent. It appears to be legacy infrastructure or a planned extension for future functionality that would analyze code purpose separately from the general analysis performed by CodeAnalyze.

How does CodeAnalyze handle multiple core files?

CodeAnalyze processes core files in parallel by spawning concurrent LLM calls for each identified source file. This approach maximizes throughput when analyzing large codebases, as each file's interface extraction, dependency mapping, and complexity analysis occurs independently before being aggregated into the final CodeInsight collection passed to RelationshipsAnalyze.

Can I run the preprocessing agents independently without the full pipeline?

Yes, you can instantiate and execute individual agents manually for testing or custom workflows. Import CodeAnalyze or RelationshipsAnalyze directly from their respective modules in src/generator/preprocess/agents/, construct them with ::new(), and call .execute() with the appropriate context. However, RelationshipsAnalyze requires the CodeInsight output from CodeAnalyze to function correctly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}