# How the Persistent Ingest Queue in LLM Wiki Handles Restarts and Retries

> Discover how the LLM Wiki persistent ingest queue handles restarts and retries. Learn how it serializes task state to localStorage for robust recovery.

- Repository: [nash_su/llm_wiki](https://github.com/nashsu/llm_wiki)
- Tags: internals
- Published: 2026-09-12

---

**The LLM Wiki persistent ingest queue survives application restarts by serializing task state to browser `localStorage` and provides granular retry functions for failed or stopped operations.**

The ingest pipeline in the `nashsu/llm_wiki` repository processes markdown sources through a durable task queue designed to withstand browser crashes, page refreshes, and manual shutdowns. By persisting task metadata and status to `localStorage` under the key `"ingest-queue"`, this system ensures that pending ingests resume automatically and failed tasks remain available for retry without manual intervention.

## Queue Persistence Mechanism

The queue durability relies on explicit serialization hooks tied to the browser lifecycle. All task state—including IDs, sources, and status flags—is maintained in `localStorage` to guarantee recovery across sessions.

### Graceful Shutdown Serialization

When the application closes or the page unloads, the system triggers `saveQueueState()` to serialize the current queue. This function is registered to the `beforeunload` event in [`src/lib/ingest-queue.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/ingest-queue.ts), ensuring that even in-flight tasks are captured and written to `localStorage` before the process terminates.

### Startup Recovery Process

During application initialization in [`src/main.tsx`](https://github.com/nashsu/llm_wiki/blob/main/src/main.tsx), the `restoreQueue()` function reads the stored JSON from `localStorage`, reconstructs the `IngestTask` objects, and repopulates the in-memory queue. If the restored data contains tasks with `"queued"` or `"failed"` status, the system immediately invokes `processQueue()` to resume ingestion without user action.

## Task Lifecycle and Retry Strategies

Each `IngestTask` tracks execution through explicit status values, allowing the system to distinguish between active, pending, and error states when determining retry eligibility.

### Task Status States

The queue recognizes five distinct statuses defined in the `IngestTask` interface within [`src/lib/ingest-queue.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/ingest-queue.ts):

- **`"queued"`** – Pending execution
- **`"running"`** – Actively processing
- **`"failed"`** – Error occurred during execution
- **`"stopped"`** – Manually halted by user
- **`"completed"`** – Successfully finished

The `nextTask` selector only considers items marked `"queued"` or `"failed"` for dequeuing, ensuring that completed or stopped tasks do not block the pipeline.

### Automatic Retry on Restart

Upon restoration, the queue automatically evaluates all non-terminal tasks. The `retryAllFailedTasks()` function iterates through every task with `"failed"` status, invokes `retryTask(taskId)` to reset the status to `"queued"` and clear error metadata, then triggers `processQueue()` to begin immediate reprocessing. Similarly, `retryAllStoppedTasks()` converts `"stopped"` tasks back to `"queued"` and resumes the pipeline.

### Manual Retry Operations

For targeted recovery, individual tasks can be re-queued using `retryTask(taskId)`. This utility updates the task status, wipes error information, and pushes the item back into the processing loop. Bulk retry operations are also supported for administrative maintenance after system failures.

## Pause, Resume, and Cleanup Operations

The queue supports operational flow control and storage maintenance to manage long-running ingest sessions.

### Pausing and Resuming Processing

The `pauseProcessing()` and `resumeProcessing()` functions in [`src/lib/ingest-queue.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/ingest-queue.ts) control task dequeuing without purging queue data. Importantly, the pause flag persists to `localStorage`, meaning a paused queue remains halted after a browser restart until `resumeProcessing()` is explicitly called.

### Clearing Completed Tasks

To prevent unbounded `localStorage` growth, `clearCompletedTasks()` removes all entries with `"completed"` status from the persisted state. This housekeeping function maintains queue performance by eliminating unnecessary historical data while preserving failed or pending tasks for retry.

## Integration with the Ingest Pipeline

The persistent queue integrates with the broader ingestion architecture through specific lifecycle hooks in [`src/lib/source-lifecycle.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/source-lifecycle.ts). This module calls `enqueueIngest()` to add new sources to the queue and manages cache cleanup upon completion.

Once the queue drains—emitting a completion event—[`src/lib/sweep-reviews.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/sweep-reviews.ts) triggers review sweeps, ensuring that downstream processes only execute after all ingest tasks have finished processing or retrying.

## Code Examples

### Enqueue a new ingest job

```typescript
import { enqueueIngest } from "@/lib/ingest-queue";

await enqueueIngest({
  id: crypto.randomUUID(),
  source: "/wiki/sources/my-article.md",
  status: "queued",
  // …other IngestTask fields…
});

```

### Retry a specific failed task

```typescript
import { retryTask } from "@/lib/ingest-queue";

await retryTask("c3f7a1b2-d4e5-6789-abcd-ef0123456789");

```

### Bulk-retry all failed tasks after restart

```typescript
import { retryAllFailedTasks } from "@/lib/ingest-queue";

const retried = await retryAllFailedTasks();
console.log(`${retried} failed tasks have been re-queued`);

```

### Pause and resume processing

```typescript
import { pauseProcessing, resumeProcessing } from "@/lib/ingest-queue";

pauseProcessing();      // stops dequeuing new tasks
// …perform other work…
resumeProcessing();     // continues ingest processing

```

## Summary

- The queue persists to `localStorage` under the key `"ingest-queue"` via `saveQueueState()` during the `beforeunload` event.
- `restoreQueue()` in [`src/main.tsx`](https://github.com/nashsu/llm_wiki/blob/main/src/main.tsx) rebuilds the queue state on application startup, automatically resuming eligible tasks.
- Tasks use explicit status fields (`"queued"`, `"failed"`, `"stopped"`, `"completed"`) to determine retry eligibility and pipeline flow.
- `retryTask()` and `retryAllFailedTasks()` provide granular and bulk retry capabilities for failed operations.
- `pauseProcessing()` and `resumeProcessing()` allow operational control with state persistence across browser restarts.
- `clearCompletedTasks()` prevents storage bloat by purging finished entries while retaining failed tasks for recovery.

## Frequently Asked Questions

### Where is the ingest queue state stored in LLM Wiki?

The queue state is serialized to the browser's `localStorage` under the key `"ingest-queue"`. This write occurs during the `beforeunload` event via `saveQueueState()` in [`src/lib/ingest-queue.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/ingest-queue.ts), and restoration happens on startup in [`src/main.tsx`](https://github.com/nashsu/llm_wiki/blob/main/src/main.tsx) through `restoreQueue()`.

### How does the queue handle tasks that were running during a browser crash?

Tasks holding `"running"` status at shutdown are captured in the `localStorage` snapshot by `saveQueueState()`. Upon restart, `restoreQueue()` reconstructs these tasks with their original metadata; the `processQueue()` logic then re-evaluates them for execution based on current queue rules and availability.

### Can I manually retry specific failed ingest tasks?

Yes. The `retryTask(taskId)` function in [`src/lib/ingest-queue.ts`](https://github.com/nashsu/llm_wiki/blob/main/src/lib/ingest-queue.ts) accepts a unique task identifier, resets that specific task's status from `"failed"` to `"queued"`, clears associated error metadata, and immediately re-adds it to the processing pipeline.

### Does the pause state survive browser restarts?

Yes. When `pauseProcessing()` is invoked, the pause flag is persisted to `localStorage` alongside the queue data. After a restart, `restoreQueue()` maintains this paused state, preventing automatic task execution until `resumeProcessing()` is explicitly called to clear the flag.