How to Embed and Store Text Data for Semantic Search in Project N.O.M.A.D.

Project N.O.M.A.D. provides a complete pipeline that converts arbitrary text into 768-dimensional vector embeddings and stores them in a Qdrant vector database for semantic similarity retrieval.

Project N.O.M.A.D. (Nomad) is an open-source knowledge base developed by Crosstalk-Solutions that enables self-hosted Retrieval-Augmented Generation (RAG). To embed and store text data for semantic search, the system implements a three-layer architecture spanning API controllers, background job queues, and a dedicated embedding service that handles chunking, vectorization, and persistence.

The Three-Layer Embedding Pipeline

The ingestion flow follows a strict separation of concerns across the API, job queue, and service layers.

API Layer: File Upload and Job Dispatch

The RagController.upload() method in admin/app/controllers/rag_controller.ts (lines 14-33) handles incoming HTTP requests. It sanitizes filenames, writes uploaded files to storage/kb_uploads, and immediately dispatches a background job to process the content asynchronously.

// Endpoint: POST /rag/upload
// Controller: admin/app/controllers/rag_controller.ts

Background Job: EmbedFileJob

The EmbedFileJob.handle() method in admin/app/jobs/embed_file_job.ts (lines 34-82) validates that Ollama (for embeddings) and Qdrant (vector storage) are reachable before invoking RagService.processAndEmbedFile(). For large ZIM archives, this job supports batch processing to prevent timeouts.

// Job: admin/app/jobs/embed_file_job.ts
// Validates dependencies then triggers embedding

Core Service: RagService

RagService in admin/app/services/rag_service.ts performs the heavy lifting. It coordinates text chunking, embedding generation, and vector storage. The service initializes the Qdrant collection via _ensureCollection() (lines 64-81), which creates the nomad_knowledge_base collection with 768-dimensional vectors if it does not exist.

Chunking and Preprocessing Strategy

Before embedding, raw text undergoes intelligent segmentation to balance context preservation with vector database efficiency.

Token-based chunking uses TokenChunker with a target of 1700 tokens per chunk (approximately 5100 characters, using a 3:1 character-to-token ratio). An overlap of roughly 150 tokens ensures semantic continuity between adjacent chunks. This logic resides in admin/app/services/rag_service.ts (lines 68-78).

Each chunk then passes through:

• Sanitization (sanitizeText): Removes problematic characters that could interfere with storage or retrieval
• Keyword extraction (extractKeywords): Generates searchable metadata for hybrid search capabilities

These preprocessing steps are implemented in admin/app/services/rag_service.ts (lines 104-115).

Generating Vector Embeddings with Ollama

Project N.O.M.A.D. uses the nomic-embed-text:v1.5 model via Ollama to generate embeddings. This model produces 768-dimensional vectors optimized for semantic similarity tasks.

The embedding process follows these specific rules as implemented in admin/app/services/rag_service.ts (lines 90-115 and 122-140):

1. Prefixing: Each chunk is prefixed with search_document: to align with the model's training for asymmetric search
2. Truncation: Text is truncated to a safe token budget before being sent to Ollama
3. Batching: Chunks are processed in batches of 8 (RagService.EMBEDDING_BATCH_SIZE = 8) to reduce HTTP overhead while respecting hardware constraints

// Constants from admin/app/services/rag_service.ts
const EMBEDDING_MODEL = 'nomic-embed-text:v1.5'
const EMBEDDING_DIMENSION = 768
const EMBEDDING_BATCH_SIZE = 8

Storing Embeddings in Qdrant

Once generated, embeddings are upserted into the Qdrant collection nomad_knowledge_base using cosine similarity for distance calculation.

The payload stored with each vector (lines 336-380 in admin/app/services/rag_service.ts) includes:

• text: The sanitized original content
• chunk_index and total_chunks: Position metadata
• keywords: Extracted search terms
• source: File path or origin identifier
• content_type: MIME type classification
• ZIM-specific metadata (when applicable): article_title, section_title, document_id

Practical Implementation Examples

Directly Embed a Raw String

Use RagService.embedAndStoreText() (lines 235-250) to programmatically embed text without file upload:

import { RagService } from '#services/rag_service'
import { DockerService } from '#services/docker_service'
import { OllamaService } from '#services/ollama_service'

const rag = new RagService(new DockerService(), new OllamaService())

async function embedSample() {
  const text = `Project Nomad is a self-hosted knowledge-base that enables semantic search across your documents.`
  const result = await rag.embedAndStoreText(text, { 
    source: 'sample.txt',
    content_type: 'text/plain'
  })
  console.log('Embedded chunks:', result?.chunks)
}

embedSample()

Upload via HTTP API

Send files to the upload endpoint for background processing:

curl -F "file=@my-doc.pdf" \
  -X POST http://localhost:3333/rag/upload \
  -H "Accept: application/json"

The endpoint returns a job ID. Query the status via GET /rag/job-status?filePath=my-doc.pdf or programmatically:

import { EmbedFileJob } from '#jobs/embed_file_job'

async function checkStatus(filePath: string) {
  const job = await EmbedFileJob.getByFilePath(filePath) // lines 80-85
  if (!job) return console.log('Job not found')
  const state = await job.getState()
  console.log('Progress:', state.progress, 'Status:', state.data?.status)
}

Execute Semantic Search

Retrieve relevant documents using searchSimilarDocuments() (lines 670-680), which automatically prefixes queries with search_query: for optimal model performance:

import { RagService } from '#services/rag_service'
import { DockerService } from '#services/docker_service'
import { OllamaService } from '#services/ollama_service'

const rag = new RagService(new DockerService(), new OllamaService())

async function search() {
  const results = await rag.searchSimilarDocuments(
    'how to backup a ZIM file', 
    5 // top-k results
  )
  
  results.forEach(r => {
    console.log(`Score: ${r.score.toFixed(3)} – ${r.text.slice(0, 120)}…`)
  })
}

search()

Summary

• Pipeline: Uploads flow through RagController → EmbedFileJob → RagService before reaching Qdrant
• Chunking: Target 1700 tokens per chunk with ~150 token overlap using TokenChunker
• Model: nomic-embed-text:v1.5 generates 768-dimensional vectors via Ollama
• Storage: Embeddings reside in the nomad_knowledge_base Qdrant collection with rich metadata payloads
• Batching: Default batch size of 8 chunks optimizes throughput without overwhelming local hardware
• Search: Use searchSimilarDocuments() with query prefixing for asymmetric semantic search

Frequently Asked Questions

What embedding model does Project N.O.M.A.D. use?

Project N.O.M.A.D. uses nomic-embed-text:v1.5 via Ollama, producing 768-dimensional vectors. This model is specifically instructed with search_document: prefixes during ingestion and search_query: prefixes during retrieval to optimize for asymmetric semantic search, as defined in RagService.EMBEDDING_MODEL and RagService.SEARCH_QUERY_PREFIX.

How does Nomad handle large files like ZIM archives?

The EmbedFileJob in admin/app/jobs/embed_file_job.ts detects ZIM archives and splits processing into batches to prevent job timeouts. The ZimExtractionService extracts individual articles with hierarchical metadata (article titles, section paths) that enriches the vector payload for better retrieval context.

What is the default chunk size and why?

The default chunk size targets 1700 tokens (approximately 5100 characters), calculated using a 3:1 character-to-token ratio. This balances context preservation with vector database limitations. An overlap of roughly 150 tokens ensures semantic continuity between chunks, preventing information loss at boundaries.

How do I verify my text was successfully embedded?

Check the job status via EmbedFileJob.getByFilePath() (lines 80-85) which returns the BullMQ job state including progress and error details. For direct API uploads, query GET /rag/job-status?filePath=your-file.ext. Successfully embedded chunks appear in the nomad_knowledge_base Qdrant collection with payloads containing the original text, chunk indices, and extracted keywords.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}