How EmbedFileJob Powers Project N.O.M.A.D.'s RAG Knowledge Base

The EmbedFileJob is the Bull-MQ background worker that converts user-uploaded files into searchable vector embeddings by orchestrating service readiness checks, text extraction, Ollama-based embedding generation, and Qdrant storage, forming the backbone of Project N.O.M.A.D.'s retrieval-augmented generation capabilities.

Project N.O.M.A.D. implements a retrieval-augmented generation (RAG) system that transforms raw documents into queryable knowledge. The EmbedFileJob, defined in admin/app/jobs/embed_file_job.ts, serves as the critical background processor that manages this conversion pipeline, handling everything from PDFs and images to massive ZIM archives while maintaining real-time progress feedback.

Core Architecture and Queue Registration

The job operates as a Bull-MQ worker with intentionally low concurrency due to the CPU-intensive nature of embedding generation. In admin/commands/queue/work.ts at lines 97-108, the job is registered with a concurrency limit of 2, ensuring system stability during heavy processing workloads.

Service Readiness Validation

Before processing any file, the job verifies that downstream dependencies are available. At lines 45-56 of embed_file_job.ts, the handler checks Ollama (for embedding models) and Qdrant (for vector storage) connectivity, throwing explicit errors that trigger Bull-MQ's retry mechanism if services are unavailable.

The Embedding Pipeline Execution

The actual transformation logic delegates to the RagService, but the job manages the orchestration and state persistence.

Progress Tracking and Status Updates

The job provides real-time feedback by updating Bull-MQ progress events from 5% to 95% as processing advances. Lines 70-73 of embed_file_job.ts implement a progress callback that streams completion percentages to the UI, while lines 84-89 and 152-162 persist final status, timestamps, and chunk counts to the job payload for API retrieval.

File Processing Delegation

At line 78 of embed_file_job.ts, the job invokes ragService.processAndEmbedFile(), implemented in admin/app/services/rag_service.ts at lines 104-119. This method handles text extraction (including OCR for images), token-based chunking, vector generation via Ollama, and upsertion into Qdrant.

ZIM Archive Batch Processing

For large ZIM archives that exceed memory constraints, the job implements sophisticated batching logic at lines 90-106. Instead of loading entire archives, it processes configurable batches and automatically re-queues subsequent chunks by dispatching new EmbedFileJob instances with updated offsets.

The underlying ZIM processing in rag_service.ts (lines 661-685) extracts article batches using configurable offsets and sizes, embedding each chunk with rich metadata including article titles and content types. Only after the final batch completes does the job optionally delete the source file, ensuring data integrity throughout the multi-stage process.

Integration Points and Dispatch Patterns

The EmbedFileJob is dispatched from multiple entry points across the codebase:

• RAG Controller: admin/app/controllers/rag_controller.ts at line 31 handles user uploads by dispatching jobs with file paths and display names.
• Download Pipeline: admin/app/jobs/run_download_job.ts at line 87 automatically queues embedding jobs after completing remote file downloads.
• Storage Sync: The storage synchronization scanner and document discovery routines also leverage this job for bulk indexing operations.

Dispatching an Embed File Job

// From the RAG controller (admin/app/controllers/rag_controller.ts)
await EmbedFileJob.dispatch({
  filePath: fullPath,          // absolute path on the server
  fileName: fileInfo.name,     // friendly name shown to the user
});

Core Job Handler Logic

async handle(job: Job) {
  const { filePath, fileName, batchOffset } = job.data as EmbedFileJobParams;

  // 1️⃣ Verify Ollama & Qdrant are up
  const models = await ollamaService.getModels();
  if (!models) throw new Error('Ollama service not ready yet');

  const qdrantUrl = await dockerService.getServiceURL('nomad_qdrant');
  if (!qdrantUrl) throw new Error('Qdrant service not ready yet');

  // 2️⃣ Progress starts
  await job.updateProgress(5);
  await job.updateData({ status: 'processing', startedAt: Date.now() });

  // 3️⃣ Run the heavy-lifting RAG pipeline
  const result = await ragService.processAndEmbedFile(
    filePath,
    /* deleteAfterEmbedding */ job.data.isFinalBatch === true,
    batchOffset,
    async (pct) => await job.updateProgress(Math.min(95, Math.round(5 + pct * 0.9)))
  );

  // 4️⃣ Batch-or-final handling
  if (result.hasMoreBatches) {
    await EmbedFileJob.dispatch({
      filePath,
      fileName,
      batchOffset: result.nextOffset,
      totalArticles: result.totalArticles,
      isFinalBatch: false,
    });
  }
  // …final status update omitted for brevity
}

ZIM Batch Processing Implementation

const zimChunks = await zimExtractionService.extractZIMContent(filepath, {
  startOffset: batchOffset ?? 0,
  batchSize: ZIM_BATCH_SIZE,
});

for (const zimChunk of zimChunks) {
  await this.embedAndStoreText(zimChunk.text, {
    source: filepath,
    content_type: 'zim_article',
    article_title: zimChunk.articleTitle,
    // …rich metadata for later search
  });
}

Summary

• The EmbedFileJob serves as the primary background worker for Project N.O.M.A.D.'s RAG pipeline, converting raw files into searchable vector embeddings stored in Qdrant.
• It enforces service readiness checks before processing, verifying Ollama and Qdrant availability to prevent failed partial embeddings.
• Progress tracking from 5% to 95% provides real-time UI feedback, while detailed status persistence enables API-driven job monitoring.
• ZIM batch processing handles massive archives through automatic chunking and job re-queuing, preventing memory exhaustion while maintaining processing state.
• The job integrates with upload controllers, download workers, and storage scanners, making it the central orchestration point for knowledge-base creation.

Frequently Asked Questions

What is the concurrency limit for EmbedFileJob and why?

The job is registered with a concurrency limit of 2 in admin/commands/queue/work.ts because embedding operations are CPU-intensive and can overwhelm the Ollama service if too many run simultaneously. This low concurrency ensures system stability while processing large files or handling multiple concurrent uploads.

How does EmbedFileJob handle service outages during processing?

Before executing the embedding pipeline, the job validates that both Ollama and Qdrant are reachable (lines 45-56 of embed_file_job.ts). If either service is unavailable, it throws an error that triggers Bull-MQ's built-in retry mechanism with exponential backoff, automatically reprocessing the file once services recover.

Can EmbedFileJob process files larger than available memory?

Yes, specifically for ZIM archives. The job implements batch processing logic (lines 90-106) that splits large archives into manageable chunks using configurable offsets. After processing each batch, it dispatches a new job instance for the next segment, allowing terabyte-scale archives to be embedded without loading the entire file into memory at once.

Where does EmbedFileJob store processing results and metadata?

The job persists status updates, chunk counts, timestamps, and error details directly to the Bull-MQ job payload (lines 84-89 and 152-162 of embed_file_job.ts). This data is accessible via the API, allowing the UI to display real-time progress and historical processing statistics for each uploaded document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}