OpenDeepWiki Incremental Repository Update Architecture: A Deep Dive into the 4-Layer Pipeline

OpenDeepWiki implements a four-layer pipeline (API Endpoint, Service, Background Worker, and Data Layer) to detect Git changes and regenerate wiki content only for modified files, minimizing processing time and resource usage.

OpenDeepWiki is an open-source documentation generator that transforms code repositories into searchable wikis. Its architecture for incremental repository updates ensures that when a Git branch changes, only the affected files trigger wiki regeneration rather than rebuilding the entire documentation set. This article examines the complete technical implementation across four architectural layers, referencing actual source files from the AIDotNet/OpenDeepWiki repository.

The Four-Layer Architecture

The incremental update system is organized into four distinct layers, each with specific responsibilities and core types:

Layer	Responsibility	Core Types
API Endpoint	Receives manual trigger requests and exposes task status endpoints.	IncrementalUpdateEndpoints
Service	Checks whether a branch has new commits, prepares a workspace, calculates changed files, and delegates to the wiki generator.	IncrementalUpdateService
Background Worker	Continuously polls pending tasks, schedules periodic checks, and updates task status.	IncrementalUpdateWorker
Data Layer	Persists tasks, repository metadata, and tracks commit IDs.	EF entities IncrementalUpdateTask, Repository, RepositoryBranch

Key files:

• src/OpenDeepWiki/Endpoints/IncrementalUpdateEndpoints.cs
• src/OpenDeepWiki/Services/Repositories/IncrementalUpdateService.cs
• src/OpenDeepWiki/Services/Repositories/IncrementalUpdateWorker.cs
• src/OpenDeepWiki/EFCore/

Triggering Manual Updates

Manual updates are initiated through the API layer. The POST /api/v1/repositories/{repoId}/branches/{branchId}/incremental-update endpoint validates the repository and branch, then creates a high-priority task.

// From IncrementalUpdateEndpoints.cs
var taskId = await updateService.TriggerManualUpdateAsync(repositoryId, branchId, cancellationToken);
return Results.Ok(new TriggerIncrementalUpdateResponse 
{ 
    TaskId = taskId, 
    Status = "Pending" 
});

The TriggerManualUpdateAsync method assigns a priority value from IncrementalUpdateOptions.ManualTriggerPriority (default: 100), ensuring manual requests process before scheduled background tasks.

Detecting Repository Changes

The core detection logic resides in IncrementalUpdateService.CheckForUpdatesAsync. This method performs four critical operations:

1. Load metadata – Retrieves the repository configuration and branch record containing LastCommitId.
2. Prepare workspace – Clones or pulls the branch into a temporary workspace using PrepareWorkspaceWithRetryAsync, which implements exponential back-off and corruption detection (IsWorkspaceCorrupted, CleanupCorruptedWorkspaceAsync).
3. Compare commits – Validates the current workspace commit against branch.LastCommitId.
4. Calculate deltas – If commits differ, invokes IRepositoryAnalyzer.GetChangedFilesAsync to enumerate modified files.

// From IncrementalUpdateService.cs
var workspace = await PrepareWorkspaceWithRetryAsync(
    repository, 
    branch.BranchName, 
    previousCommitId, 
    cancellationToken);

var currentCommitId = workspace.CommitId;

if (previousCommitId == currentCommitId) 
    return new UpdateCheckResult { NeedsUpdate = false };

var changedFiles = await _repositoryAnalyzer.GetChangedFilesAsync(
    workspace, 
    previousCommitId, 
    currentCommitId, 
    cancellationToken);

Executing Incremental Wiki Generation

When changes are detected, ProcessIncrementalUpdateAsync orchestrates the wiki regeneration:

1. Re-prepares the workspace at the previous commit point to establish a baseline.
2. Retrieves all configured languages for the branch (BranchLanguages).
3. For each language, calls IWikiGenerator.IncrementalUpdateAsync passing the changed file list.
4. Updates branch metadata (LastCommitId, LastProcessedAt) and repository metadata (LastUpdateCheckAt).
5. Notifies subscribers via INotificationService (fails silently to ensure pipeline continuity).

// From IncrementalUpdateService.cs
await _wikiGenerator.IncrementalUpdateAsync(
    workspace, 
    branchLanguage, 
    checkResult.ChangedFiles ?? Array.Empty<string>(), 
    cancellationToken);

Background Processing and Scheduling

The IncrementalUpdateWorker implements IHostedService and runs continuously while the application is active. It performs three functions on each polling cycle:

• Task polling – Every PollingIntervalSeconds (default: 30), fetches pending tasks ordered by Priority descending.
• Task execution – Processes each task through ProcessSingleTaskAsync, handling status transitions (Pending → Processing → Completed/Failed).
• Scheduled checks – Queries repositories where LastUpdateCheckAt exceeds UpdateIntervalMinutes (default: 1440 minutes/1 day) and creates normal-priority tasks for each branch.

// From IncrementalUpdateWorker.cs
var pendingTasks = await GetPendingTasksAsync(context, stoppingToken);
foreach (var task in pendingTasks) 
    await ProcessSingleTaskAsync(context, updateService, task, stoppingToken);
await CheckScheduledUpdatesAsync(context, stoppingToken);

Configuration and Resilience

The system behavior is controlled through IncrementalUpdateOptions, bound to configuration sections:

Option	Description	Default
PollingIntervalSeconds	Database polling frequency for the background worker.	30
ManualTriggerPriority	Priority value for manually triggered tasks (higher executes first).	100
MaxRetryAttempts	Maximum retries for workspace preparation failures.	3
RetryBaseDelayMs	Base delay in milliseconds for exponential back-off.	2000
DefaultUpdateIntervalMinutes	Default interval between automatic update checks.	1440 (1 day)

The workspace preparation logic implements resilience patterns including exponential back-off, corruption detection via IsWorkspaceCorrupted, and automatic cleanup through CleanupCorruptedWorkspaceAsync.

Data Model and Task Persistence

Tasks are persisted using Entity Framework Core with the following key entities:

• IncrementalUpdateTask – Represents a single update operation with fields for RepositoryId, BranchId, PreviousCommitId, TargetCommitId, Status, Priority, and IsManualTrigger.
• Repository – Contains repository metadata including LastUpdateCheckAt.
• RepositoryBranch – Tracks branch-specific state via LastCommitId and LastProcessedAt.

The IContext interface abstracts database operations, allowing the worker and service layers to query and update task states atomically.

Summary

OpenDeepWiki's incremental repository update architecture delivers efficient, resilient documentation generation through:

• Layered separation of concerns across API, Service, Worker, and Data layers.
• Manual and scheduled triggers with priority-based task queuing.
• Git-aware change detection comparing LastCommitId against current workspace state.
• Per-language incremental generation that processes only changed files.
• Resilient workspace management with exponential back-off and corruption handling.
• Configurable polling intervals and retry policies via IncrementalUpdateOptions.

Frequently Asked Questions

How does OpenDeepWiki determine which files have changed?

OpenDeepWiki compares the LastCommitId stored in the RepositoryBranch entity against the current commit in the workspace. If they differ, the IRepositoryAnalyzer.GetChangedFilesAsync method enumerates the delta between these two commits, returning only the modified file paths to the wiki generator.

What happens if the Git workspace becomes corrupted during an update?

The PrepareWorkspaceWithRetryAsync method in IncrementalUpdateService includes corruption detection via IsWorkspaceCorrupted and automatic cleanup via CleanupCorruptedWorkspaceAsync. If corruption is detected, the system removes the damaged workspace and re-clones the repository, retrying up to MaxRetryAttempts times with exponential back-off.

Can I trigger incremental updates manually instead of waiting for the scheduler?

Yes. The POST /api/v1/repositories/{repoId}/branches/{branchId}/incremental-update endpoint in IncrementalUpdateEndpoints creates a high-priority task (priority 100 by default). The IncrementalUpdateWorker processes manual tasks before scheduled ones due to the priority-based ordering in GetPendingTasksAsync.

How does the background worker know when to check a repository for changes?

The IncrementalUpdateWorker polls the database every PollingIntervalSeconds (default 30 seconds). During each cycle, CheckScheduledUpdatesAsync queries repositories where LastUpdateCheckAt exceeds UpdateIntervalMinutes (default 1440 minutes/1 day), creating automatic update tasks for those branches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}