RagService Capabilities for RAG Implementation in Project Nomad

The RagService in Crosstalk-Solutions/project-nomad provides a full-stack Retrieval-Augmented Generation engine with semantic embedding, hybrid search, multi-format document processing, and knowledge-base management.

The RagService (located in admin/app/services/rag_service.ts) serves as the core RAG engine for Project Nomad's knowledge-base features. It orchestrates document ingestion, vector embedding, and intelligent retrieval using Ollama for local embeddings and Qdrant for vector storage. Understanding these RagService capabilities for RAG implementation enables developers to build robust retrieval systems that handle diverse document formats while maintaining search accuracy.

Document Ingestion and Processing

The service automatically detects file types and routes them through specialized extraction pipelines.

PDF Processing with OCR Fallback

For PDF documents, the processPDFFile method (lines 41-57) first attempts extraction using pdf-parse. When extracted content falls below length thresholds—indicating scanned documents—the service converts each page to PNG and runs Tesseract OCR to recover text.

Image Text Extraction

The processImageFile method (lines 32-35) preprocesses images through grayscale conversion, normalization, and resizing before Tesseract extracts text content.

ZIM Archive Batch Processing

Large ZIM files stream in configurable batches using ZIM_BATCH_SIZE. The processZIMFile method (lines 66-111) extracts rich metadata for every article and section, embeds each chunk individually, and optionally deletes the archive after the final batch completes.

Semantic Embedding and Storage

Intelligent Chunking

The embedAndStoreText method (lines 35-84) segments documents using TokenChunker with overlapping windows based on a 3 characters-per-token estimation ratio. This ensures context preservation across chunk boundaries.

Token Budget Management

Before embedding, the service estimates token counts via estimateTokenCount and truncates content using truncateToTokenLimit (lines 126-160) to enforce MAX_SAFE_TOKENS limits. Model-specific prefixes are applied to chunks without overflowing context windows.

Vector Upsertion

Chunks are embedded using Ollama's nomic-embed-text model and upserted into Qdrant. The service verifies model availability before operations and triggers lazy downloads via OllamaService if the model is missing locally.

Hybrid Search and Retrieval

Query Preprocessing

The preprocessQuery method (lines 66-78) normalizes user queries, expands domain-specific abbreviations (e.g., converting "bob" to "bug out bag"), and extracts stop-word-free keywords for hybrid matching.

Semantic Search with Reranking

The searchSimilarDocuments method (lines 70-102) generates query embeddings using SEARCH_QUERY_PREFIX, performs initial semantic lookup in Qdrant, then applies multi-factor reranking via rerankResults (lines 118-147):

• Keyword overlap: Up to 10% score boost
• Direct term matches: Up to 7.5% boost
• Semantic gate: Boosts apply only when base semantic score exceeds 0.35

Source Diversity Enforcement

The applySourceDiversity method (lines 152-176) penalizes multiple results sharing the same document_id or source, preventing result redundancy and ensuring broad knowledge coverage.

Knowledge Base Management

The service provides comprehensive lifecycle management through dedicated methods:

• getStoredFiles (lines 184-215): Lists all unique files currently stored in the vector database
• deleteFileBySource (lines 224-254): Removes all vectors for a given source and safely deletes the physical file
• discoverNomadDocs (lines 261-314): Auto-discovers Nomad documentation and queues embedding jobs
• scanAndSyncStorage (lines 319-401): Scans storage directories, detects missing embeddings, and dispatches synchronization jobs

Integration and Progress Tracking

Real-time Progress Callbacks

All long-running pipelines—including processAndEmbedFile, embedAndStoreText, and processZIMFile—accept an optional onProgress callback receiving percentage values (0-100), enabling real-time UI updates during document processing.

Model Verification

Before any embedding or search operation, the service checks for local model presence and triggers on-the-fly downloads if needed, ensuring zero-configuration deployments.

Implementation Examples

Embedding a PDF with Progress Tracking

import RagService from '#services/rag_service'

async function embedPdf(filePath: string) {
  const ragService = new RagService(/* dockerService, ollamaService */)
  
  const result = await ragService.processAndEmbedFile(
    filePath,
    true,                 // delete file after embedding
    undefined,            // no batch offset
    (pct) => console.log(`Progress: ${pct.toFixed(0)}%`)
  )
  
  console.log('Embedding complete:', result)
}

Performing Hybrid Search

async function searchKnowledgeBase(query: string) {
  const ragService = new RagService(/* dependencies */)
  
  const hits = await ragService.searchSimilarDocuments(query, 5)
  hits.forEach((hit, i) => {
    console.log(`#${i + 1} Score: ${hit.score.toFixed(3)}`)
    console.log(`Source: ${hit.metadata?.source}`)
    console.log(`Text: ${hit.text.slice(0, 120)}...`)
  })
}

Synchronizing Storage

async function syncKnowledgeBase() {
  const ragService = new RagService(/* dependencies */)
  const report = await ragService.scanAndSyncStorage()
  console.log('Sync report:', report)
}

Summary

• RagService in admin/app/services/rag_service.ts provides end-to-end RAG functionality for Project Nomad
• Supports PDF, image, text, and ZIM archives with automatic OCR fallback for scanned documents
• Uses Ollama's nomic-embed-text model with Qdrant for vector storage and semantic search
• Implements hybrid search combining semantic similarity with keyword reranking (up to 10% boost for overlaps) and source diversity penalties
• Enforces token budgets via MAX_SAFE_TOKENS and intelligent chunking with TokenChunker
• Provides knowledge-base management tools including auto-discovery, synchronization, and safe deletion workflows
• Supports progress callbacks and lazy model loading for production deployments

Frequently Asked Questions

What embedding model does RagService use?

The service uses Ollama's nomic-embed-text model for generating vector embeddings. Before any operation, it verifies the model is present locally and triggers an automatic download via OllamaService if missing.

How does RagService handle scanned PDFs that contain no extractable text?

When processPDFFile detects insufficient text extraction via pdf-parse, it automatically converts each PDF page to PNG format and processes them through Tesseract OCR. This ensures scanned documents and image-based PDFs remain fully searchable.

What is the hybrid search approach implemented in RagService?

The searchSimilarDocuments method combines semantic vector search in Qdrant with keyword-based reranking. Results receive conservative score boosts—approximately 10% for keyword overlap and 7.5% for direct term matches—but only when the base semantic score exceeds 0.35. The applySourceDiversity method then penalizes redundant sources to ensure varied results.

How does the service process large ZIM archives without memory issues?

The processZIMFile method streams ZIM content in configurable batches using ZIM_BATCH_SIZE, processing articles incrementally rather than loading the entire archive into memory. It extracts rich metadata for each article, embeds chunks individually, and can automatically delete the source archive after the final batch completes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}