# OpenDeepWiki Incremental Repository Update Architecture: A Deep Dive into the 4-Layer Pipeline

> Explore OpenDeepWiki's incremental repository update architecture. Learn how its 4-layer pipeline efficiently detects Git changes and regenerates wiki content, saving time and resources.

- Repository: [AIDotNet/OpenDeepWiki](https://github.com/aidotnet/opendeepwiki)
- Tags: architecture
- Published: 2026-02-16

---

**OpenDeepWiki implements a four-layer pipeline (API Endpoint, Service, Background Worker, and Data Layer) to detect Git changes and regenerate wiki content only for modified files, minimizing processing time and resource usage.**

OpenDeepWiki is an open-source documentation generator that transforms code repositories into searchable wikis. Its architecture for **incremental repository updates** ensures that when a Git branch changes, only the affected files trigger wiki regeneration rather than rebuilding the entire documentation set. This article examines the complete technical implementation across four architectural layers, referencing actual source files from the `AIDotNet/OpenDeepWiki` repository.

## The Four-Layer Architecture

The incremental update system is organized into four distinct layers, each with specific responsibilities and core types:

| Layer | Responsibility | Core Types |
|-------|----------------|------------|
| **API Endpoint** | Receives manual trigger requests and exposes task status endpoints. | `IncrementalUpdateEndpoints` |
| **Service** | Checks whether a branch has new commits, prepares a workspace, calculates changed files, and delegates to the wiki generator. | `IncrementalUpdateService` |
| **Background Worker** | Continuously polls pending tasks, schedules periodic checks, and updates task status. | `IncrementalUpdateWorker` |
| **Data Layer** | Persists tasks, repository metadata, and tracks commit IDs. | EF entities `IncrementalUpdateTask`, `Repository`, `RepositoryBranch` |

**Key files:**
- [`src/OpenDeepWiki/Endpoints/IncrementalUpdateEndpoints.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Endpoints/IncrementalUpdateEndpoints.cs)
- [`src/OpenDeepWiki/Services/Repositories/IncrementalUpdateService.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Repositories/IncrementalUpdateService.cs)
- [`src/OpenDeepWiki/Services/Repositories/IncrementalUpdateWorker.cs`](https://github.com/AIDotNet/OpenDeepWiki/blob/main/src/OpenDeepWiki/Services/Repositories/IncrementalUpdateWorker.cs)
- `src/OpenDeepWiki/EFCore/`

## Triggering Manual Updates

Manual updates are initiated through the API layer. The `POST /api/v1/repositories/{repoId}/branches/{branchId}/incremental-update` endpoint validates the repository and branch, then creates a high-priority task.

```csharp
// From IncrementalUpdateEndpoints.cs
var taskId = await updateService.TriggerManualUpdateAsync(repositoryId, branchId, cancellationToken);
return Results.Ok(new TriggerIncrementalUpdateResponse 
{ 
    TaskId = taskId, 
    Status = "Pending" 
});

```

The `TriggerManualUpdateAsync` method assigns a priority value from `IncrementalUpdateOptions.ManualTriggerPriority` (default: 100), ensuring manual requests process before scheduled background tasks.

## Detecting Repository Changes

The core detection logic resides in `IncrementalUpdateService.CheckForUpdatesAsync`. This method performs four critical operations:

1. **Load metadata** – Retrieves the repository configuration and branch record containing `LastCommitId`.
2. **Prepare workspace** – Clones or pulls the branch into a temporary workspace using `PrepareWorkspaceWithRetryAsync`, which implements exponential back-off and corruption detection (`IsWorkspaceCorrupted`, `CleanupCorruptedWorkspaceAsync`).
3. **Compare commits** – Validates the current workspace commit against `branch.LastCommitId`.
4. **Calculate deltas** – If commits differ, invokes `IRepositoryAnalyzer.GetChangedFilesAsync` to enumerate modified files.

```csharp
// From IncrementalUpdateService.cs
var workspace = await PrepareWorkspaceWithRetryAsync(
    repository, 
    branch.BranchName, 
    previousCommitId, 
    cancellationToken);

var currentCommitId = workspace.CommitId;

if (previousCommitId == currentCommitId) 
    return new UpdateCheckResult { NeedsUpdate = false };

var changedFiles = await _repositoryAnalyzer.GetChangedFilesAsync(
    workspace, 
    previousCommitId, 
    currentCommitId, 
    cancellationToken);

```

## Executing Incremental Wiki Generation

When changes are detected, `ProcessIncrementalUpdateAsync` orchestrates the wiki regeneration:

1. Re-prepares the workspace at the previous commit point to establish a baseline.
2. Retrieves all configured languages for the branch (`BranchLanguages`).
3. For each language, calls `IWikiGenerator.IncrementalUpdateAsync` passing the changed file list.
4. Updates branch metadata (`LastCommitId`, `LastProcessedAt`) and repository metadata (`LastUpdateCheckAt`).
5. Notifies subscribers via `INotificationService` (fails silently to ensure pipeline continuity).

```csharp
// From IncrementalUpdateService.cs
await _wikiGenerator.IncrementalUpdateAsync(
    workspace, 
    branchLanguage, 
    checkResult.ChangedFiles ?? Array.Empty<string>(), 
    cancellationToken);

```

## Background Processing and Scheduling

The `IncrementalUpdateWorker` implements `IHostedService` and runs continuously while the application is active. It performs three functions on each polling cycle:

- **Task polling** – Every `PollingIntervalSeconds` (default: 30), fetches pending tasks ordered by `Priority` descending.
- **Task execution** – Processes each task through `ProcessSingleTaskAsync`, handling status transitions (`Pending → Processing → Completed/Failed`).
- **Scheduled checks** – Queries repositories where `LastUpdateCheckAt` exceeds `UpdateIntervalMinutes` (default: 1440 minutes/1 day) and creates normal-priority tasks for each branch.

```csharp
// From IncrementalUpdateWorker.cs
var pendingTasks = await GetPendingTasksAsync(context, stoppingToken);
foreach (var task in pendingTasks) 
    await ProcessSingleTaskAsync(context, updateService, task, stoppingToken);
await CheckScheduledUpdatesAsync(context, stoppingToken);

```

## Configuration and Resilience

The system behavior is controlled through `IncrementalUpdateOptions`, bound to configuration sections:

| Option | Description | Default |
|--------|-------------|---------|
| `PollingIntervalSeconds` | Database polling frequency for the background worker. | 30 |
| `ManualTriggerPriority` | Priority value for manually triggered tasks (higher executes first). | 100 |
| `MaxRetryAttempts` | Maximum retries for workspace preparation failures. | 3 |
| `RetryBaseDelayMs` | Base delay in milliseconds for exponential back-off. | 2000 |
| `DefaultUpdateIntervalMinutes` | Default interval between automatic update checks. | 1440 (1 day) |

The workspace preparation logic implements resilience patterns including exponential back-off, corruption detection via `IsWorkspaceCorrupted`, and automatic cleanup through `CleanupCorruptedWorkspaceAsync`.

## Data Model and Task Persistence

Tasks are persisted using Entity Framework Core with the following key entities:

- **`IncrementalUpdateTask`** – Represents a single update operation with fields for `RepositoryId`, `BranchId`, `PreviousCommitId`, `TargetCommitId`, `Status`, `Priority`, and `IsManualTrigger`.
- **`Repository`** – Contains repository metadata including `LastUpdateCheckAt`.
- **`RepositoryBranch`** – Tracks branch-specific state via `LastCommitId` and `LastProcessedAt`.

The `IContext` interface abstracts database operations, allowing the worker and service layers to query and update task states atomically.

## Summary

OpenDeepWiki's incremental repository update architecture delivers efficient, resilient documentation generation through:

- **Layered separation** of concerns across API, Service, Worker, and Data layers.
- **Manual and scheduled triggers** with priority-based task queuing.
- **Git-aware change detection** comparing `LastCommitId` against current workspace state.
- **Per-language incremental generation** that processes only changed files.
- **Resilient workspace management** with exponential back-off and corruption handling.
- **Configurable polling intervals** and retry policies via `IncrementalUpdateOptions`.

## Frequently Asked Questions

### How does OpenDeepWiki determine which files have changed?

OpenDeepWiki compares the `LastCommitId` stored in the `RepositoryBranch` entity against the current commit in the workspace. If they differ, the `IRepositoryAnalyzer.GetChangedFilesAsync` method enumerates the delta between these two commits, returning only the modified file paths to the wiki generator.

### What happens if the Git workspace becomes corrupted during an update?

The `PrepareWorkspaceWithRetryAsync` method in `IncrementalUpdateService` includes corruption detection via `IsWorkspaceCorrupted` and automatic cleanup via `CleanupCorruptedWorkspaceAsync`. If corruption is detected, the system removes the damaged workspace and re-clones the repository, retrying up to `MaxRetryAttempts` times with exponential back-off.

### Can I trigger incremental updates manually instead of waiting for the scheduler?

Yes. The `POST /api/v1/repositories/{repoId}/branches/{branchId}/incremental-update` endpoint in `IncrementalUpdateEndpoints` creates a high-priority task (priority 100 by default). The `IncrementalUpdateWorker` processes manual tasks before scheduled ones due to the priority-based ordering in `GetPendingTasksAsync`.

### How does the background worker know when to check a repository for changes?

The `IncrementalUpdateWorker` polls the database every `PollingIntervalSeconds` (default 30 seconds). During each cycle, `CheckScheduledUpdatesAsync` queries repositories where `LastUpdateCheckAt` exceeds `UpdateIntervalMinutes` (default 1440 minutes/1 day), creating automatic update tasks for those branches.