How the Persistent Ingest Queue in LLM Wiki Handles Restarts and Retries

The LLM Wiki persistent ingest queue survives application restarts by serializing task state to browser localStorage and provides granular retry functions for failed or stopped operations.

The ingest pipeline in the nashsu/llm_wiki repository processes markdown sources through a durable task queue designed to withstand browser crashes, page refreshes, and manual shutdowns. By persisting task metadata and status to localStorage under the key "ingest-queue", this system ensures that pending ingests resume automatically and failed tasks remain available for retry without manual intervention.

Queue Persistence Mechanism

The queue durability relies on explicit serialization hooks tied to the browser lifecycle. All task state—including IDs, sources, and status flags—is maintained in localStorage to guarantee recovery across sessions.

Graceful Shutdown Serialization

When the application closes or the page unloads, the system triggers saveQueueState() to serialize the current queue. This function is registered to the beforeunload event in src/lib/ingest-queue.ts, ensuring that even in-flight tasks are captured and written to localStorage before the process terminates.

Startup Recovery Process

During application initialization in src/main.tsx, the restoreQueue() function reads the stored JSON from localStorage, reconstructs the IngestTask objects, and repopulates the in-memory queue. If the restored data contains tasks with "queued" or "failed" status, the system immediately invokes processQueue() to resume ingestion without user action.

Task Lifecycle and Retry Strategies

Each IngestTask tracks execution through explicit status values, allowing the system to distinguish between active, pending, and error states when determining retry eligibility.

Task Status States

The queue recognizes five distinct statuses defined in the IngestTask interface within src/lib/ingest-queue.ts:

• "queued" – Pending execution
• "running" – Actively processing
• "failed" – Error occurred during execution
• "stopped" – Manually halted by user
• "completed" – Successfully finished

The nextTask selector only considers items marked "queued" or "failed" for dequeuing, ensuring that completed or stopped tasks do not block the pipeline.

Automatic Retry on Restart

Upon restoration, the queue automatically evaluates all non-terminal tasks. The retryAllFailedTasks() function iterates through every task with "failed" status, invokes retryTask(taskId) to reset the status to "queued" and clear error metadata, then triggers processQueue() to begin immediate reprocessing. Similarly, retryAllStoppedTasks() converts "stopped" tasks back to "queued" and resumes the pipeline.

Manual Retry Operations

For targeted recovery, individual tasks can be re-queued using retryTask(taskId). This utility updates the task status, wipes error information, and pushes the item back into the processing loop. Bulk retry operations are also supported for administrative maintenance after system failures.

Pause, Resume, and Cleanup Operations

The queue supports operational flow control and storage maintenance to manage long-running ingest sessions.

Pausing and Resuming Processing

The pauseProcessing() and resumeProcessing() functions in src/lib/ingest-queue.ts control task dequeuing without purging queue data. Importantly, the pause flag persists to localStorage, meaning a paused queue remains halted after a browser restart until resumeProcessing() is explicitly called.

Clearing Completed Tasks

To prevent unbounded localStorage growth, clearCompletedTasks() removes all entries with "completed" status from the persisted state. This housekeeping function maintains queue performance by eliminating unnecessary historical data while preserving failed or pending tasks for retry.

Integration with the Ingest Pipeline

The persistent queue integrates with the broader ingestion architecture through specific lifecycle hooks in src/lib/source-lifecycle.ts. This module calls enqueueIngest() to add new sources to the queue and manages cache cleanup upon completion.

Once the queue drains—emitting a completion event—src/lib/sweep-reviews.ts triggers review sweeps, ensuring that downstream processes only execute after all ingest tasks have finished processing or retrying.

Code Examples

Enqueue a new ingest job

import { enqueueIngest } from "@/lib/ingest-queue";

await enqueueIngest({
  id: crypto.randomUUID(),
  source: "/wiki/sources/my-article.md",
  status: "queued",
  // …other IngestTask fields…
});

Retry a specific failed task

import { retryTask } from "@/lib/ingest-queue";

await retryTask("c3f7a1b2-d4e5-6789-abcd-ef0123456789");

Bulk-retry all failed tasks after restart

import { retryAllFailedTasks } from "@/lib/ingest-queue";

const retried = await retryAllFailedTasks();
console.log(`${retried} failed tasks have been re-queued`);

Pause and resume processing

import { pauseProcessing, resumeProcessing } from "@/lib/ingest-queue";

pauseProcessing();      // stops dequeuing new tasks
// …perform other work…
resumeProcessing();     // continues ingest processing

Summary

• The queue persists to localStorage under the key "ingest-queue" via saveQueueState() during the beforeunload event.
• restoreQueue() in src/main.tsx rebuilds the queue state on application startup, automatically resuming eligible tasks.
• Tasks use explicit status fields ("queued", "failed", "stopped", "completed") to determine retry eligibility and pipeline flow.
• retryTask() and retryAllFailedTasks() provide granular and bulk retry capabilities for failed operations.
• pauseProcessing() and resumeProcessing() allow operational control with state persistence across browser restarts.
• clearCompletedTasks() prevents storage bloat by purging finished entries while retaining failed tasks for recovery.

Frequently Asked Questions

Where is the ingest queue state stored in LLM Wiki?

The queue state is serialized to the browser's localStorage under the key "ingest-queue". This write occurs during the beforeunload event via saveQueueState() in src/lib/ingest-queue.ts, and restoration happens on startup in src/main.tsx through restoreQueue().

How does the queue handle tasks that were running during a browser crash?

Tasks holding "running" status at shutdown are captured in the localStorage snapshot by saveQueueState(). Upon restart, restoreQueue() reconstructs these tasks with their original metadata; the processQueue() logic then re-evaluates them for execution based on current queue rules and availability.

Can I manually retry specific failed ingest tasks?

Yes. The retryTask(taskId) function in src/lib/ingest-queue.ts accepts a unique task identifier, resets that specific task's status from "failed" to "queued", clears associated error metadata, and immediately re-adds it to the processing pipeline.

Does the pause state survive browser restarts?

Yes. When pauseProcessing() is invoked, the pause flag is persisted to localStorage alongside the queue data. After a restart, restoreQueue() maintains this paused state, preventing automatic task execution until resumeProcessing() is explicitly called to clear the flag.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}