# RagService Capabilities for RAG Implementation in Project Nomad

> Discover the RagService capabilities for RAG implementation in Project Nomad Explore semantic embedding, hybrid search, multi-format document processing, and knowledge-base management.

- Repository: [Crosstalk Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
- Tags: capabilities
- Published: 2026-03-16

---

**The RagService in Crosstalk-Solutions/project-nomad provides a full-stack Retrieval-Augmented Generation engine with semantic embedding, hybrid search, multi-format document processing, and knowledge-base management.**

The **RagService** (located in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts)) serves as the core RAG engine for Project Nomad's knowledge-base features. It orchestrates document ingestion, vector embedding, and intelligent retrieval using Ollama for local embeddings and Qdrant for vector storage. Understanding these RagService capabilities for RAG implementation enables developers to build robust retrieval systems that handle diverse document formats while maintaining search accuracy.

## Document Ingestion and Processing

The service automatically detects file types and routes them through specialized extraction pipelines.

### PDF Processing with OCR Fallback

For PDF documents, the `processPDFFile` method (lines 41-57) first attempts extraction using `pdf-parse`. When extracted content falls below length thresholds—indicating scanned documents—the service converts each page to PNG and runs Tesseract OCR to recover text.

### Image Text Extraction

The `processImageFile` method (lines 32-35) preprocesses images through grayscale conversion, normalization, and resizing before Tesseract extracts text content.

### ZIM Archive Batch Processing

Large ZIM files stream in configurable batches using `ZIM_BATCH_SIZE`. The `processZIMFile` method (lines 66-111) extracts rich metadata for every article and section, embeds each chunk individually, and optionally deletes the archive after the final batch completes.

## Semantic Embedding and Storage

### Intelligent Chunking

The `embedAndStoreText` method (lines 35-84) segments documents using `TokenChunker` with overlapping windows based on a 3 characters-per-token estimation ratio. This ensures context preservation across chunk boundaries.

### Token Budget Management

Before embedding, the service estimates token counts via `estimateTokenCount` and truncates content using `truncateToTokenLimit` (lines 126-160) to enforce `MAX_SAFE_TOKENS` limits. Model-specific prefixes are applied to chunks without overflowing context windows.

### Vector Upsertion

Chunks are embedded using Ollama's `nomic-embed-text` model and upserted into Qdrant. The service verifies model availability before operations and triggers lazy downloads via `OllamaService` if the model is missing locally.

## Hybrid Search and Retrieval

### Query Preprocessing

The `preprocessQuery` method (lines 66-78) normalizes user queries, expands domain-specific abbreviations (e.g., converting "bob" to "bug out bag"), and extracts stop-word-free keywords for hybrid matching.

### Semantic Search with Reranking

The `searchSimilarDocuments` method (lines 70-102) generates query embeddings using `SEARCH_QUERY_PREFIX`, performs initial semantic lookup in Qdrant, then applies multi-factor reranking via `rerankResults` (lines 118-147):

- **Keyword overlap**: Up to 10% score boost
- **Direct term matches**: Up to 7.5% boost  
- **Semantic gate**: Boosts apply only when base semantic score exceeds 0.35

### Source Diversity Enforcement

The `applySourceDiversity` method (lines 152-176) penalizes multiple results sharing the same `document_id` or `source`, preventing result redundancy and ensuring broad knowledge coverage.

## Knowledge Base Management

The service provides comprehensive lifecycle management through dedicated methods:

- **`getStoredFiles`** (lines 184-215): Lists all unique files currently stored in the vector database
- **`deleteFileBySource`** (lines 224-254): Removes all vectors for a given source and safely deletes the physical file
- **`discoverNomadDocs`** (lines 261-314): Auto-discovers Nomad documentation and queues embedding jobs
- **`scanAndSyncStorage`** (lines 319-401): Scans storage directories, detects missing embeddings, and dispatches synchronization jobs

## Integration and Progress Tracking

### Real-time Progress Callbacks

All long-running pipelines—including `processAndEmbedFile`, `embedAndStoreText`, and `processZIMFile`—accept an optional `onProgress` callback receiving percentage values (0-100), enabling real-time UI updates during document processing.

### Model Verification

Before any embedding or search operation, the service checks for local model presence and triggers on-the-fly downloads if needed, ensuring zero-configuration deployments.

## Implementation Examples

### Embedding a PDF with Progress Tracking

```typescript
import RagService from '#services/rag_service'

async function embedPdf(filePath: string) {
  const ragService = new RagService(/* dockerService, ollamaService */)
  
  const result = await ragService.processAndEmbedFile(
    filePath,
    true,                 // delete file after embedding
    undefined,            // no batch offset
    (pct) => console.log(`Progress: ${pct.toFixed(0)}%`)
  )
  
  console.log('Embedding complete:', result)
}

```

### Performing Hybrid Search

```typescript
async function searchKnowledgeBase(query: string) {
  const ragService = new RagService(/* dependencies */)
  
  const hits = await ragService.searchSimilarDocuments(query, 5)
  hits.forEach((hit, i) => {
    console.log(`#${i + 1} Score: ${hit.score.toFixed(3)}`)
    console.log(`Source: ${hit.metadata?.source}`)
    console.log(`Text: ${hit.text.slice(0, 120)}...`)
  })
}

```

### Synchronizing Storage

```typescript
async function syncKnowledgeBase() {
  const ragService = new RagService(/* dependencies */)
  const report = await ragService.scanAndSyncStorage()
  console.log('Sync report:', report)
}

```

## Summary

- **RagService** in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) provides end-to-end RAG functionality for Project Nomad
- Supports **PDF, image, text, and ZIM archives** with automatic OCR fallback for scanned documents
- Uses **Ollama's nomic-embed-text** model with **Qdrant** for vector storage and semantic search
- Implements **hybrid search** combining semantic similarity with keyword reranking (up to 10% boost for overlaps) and source diversity penalties
- Enforces **token budgets** via `MAX_SAFE_TOKENS` and intelligent chunking with `TokenChunker`
- Provides **knowledge-base management** tools including auto-discovery, synchronization, and safe deletion workflows
- Supports **progress callbacks** and **lazy model loading** for production deployments

## Frequently Asked Questions

### What embedding model does RagService use?

The service uses Ollama's `nomic-embed-text` model for generating vector embeddings. Before any operation, it verifies the model is present locally and triggers an automatic download via `OllamaService` if missing.

### How does RagService handle scanned PDFs that contain no extractable text?

When `processPDFFile` detects insufficient text extraction via `pdf-parse`, it automatically converts each PDF page to PNG format and processes them through Tesseract OCR. This ensures scanned documents and image-based PDFs remain fully searchable.

### What is the hybrid search approach implemented in RagService?

The `searchSimilarDocuments` method combines semantic vector search in Qdrant with keyword-based reranking. Results receive conservative score boosts—approximately 10% for keyword overlap and 7.5% for direct term matches—but only when the base semantic score exceeds 0.35. The `applySourceDiversity` method then penalizes redundant sources to ensure varied results.

### How does the service process large ZIM archives without memory issues?

The `processZIMFile` method streams ZIM content in configurable batches using `ZIM_BATCH_SIZE`, processing articles incrementally rather than loading the entire archive into memory. It extracts rich metadata for each article, embeds chunks individually, and can automatically delete the source archive after the final batch completes.