# How to Embed and Store Text Data for Semantic Search in Project N.O.M.A.D.

> Learn how to embed and store text data for semantic search in Project N.O.M.A.D. Convert text to vector embeddings and store in Qdrant for efficient similarity retrieval.

- Repository: [Crosstalk Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
- Tags: how-to-guide
- Published: 2026-03-16

---

**Project N.O.M.A.D. provides a complete pipeline that converts arbitrary text into 768-dimensional vector embeddings and stores them in a Qdrant vector database for semantic similarity retrieval.**

Project N.O.M.A.D. (Nomad) is an open-source knowledge base developed by Crosstalk-Solutions that enables self-hosted Retrieval-Augmented Generation (RAG). To embed and store text data for semantic search, the system implements a three-layer architecture spanning API controllers, background job queues, and a dedicated embedding service that handles chunking, vectorization, and persistence.

## The Three-Layer Embedding Pipeline

The ingestion flow follows a strict separation of concerns across the API, job queue, and service layers.

### API Layer: File Upload and Job Dispatch

The `RagController.upload()` method in [`admin/app/controllers/rag_controller.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/controllers/rag_controller.ts) (lines 14-33) handles incoming HTTP requests. It sanitizes filenames, writes uploaded files to `storage/kb_uploads`, and immediately dispatches a background job to process the content asynchronously.

```typescript
// Endpoint: POST /rag/upload
// Controller: admin/app/controllers/rag_controller.ts

```

### Background Job: EmbedFileJob

The `EmbedFileJob.handle()` method in [`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts) (lines 34-82) validates that Ollama (for embeddings) and Qdrant (vector storage) are reachable before invoking `RagService.processAndEmbedFile()`. For large ZIM archives, this job supports batch processing to prevent timeouts.

```typescript
// Job: admin/app/jobs/embed_file_job.ts
// Validates dependencies then triggers embedding

```

### Core Service: RagService

`RagService` in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) performs the heavy lifting. It coordinates text chunking, embedding generation, and vector storage. The service initializes the Qdrant collection via `_ensureCollection()` (lines 64-81), which creates the `nomad_knowledge_base` collection with 768-dimensional vectors if it does not exist.

## Chunking and Preprocessing Strategy

Before embedding, raw text undergoes intelligent segmentation to balance context preservation with vector database efficiency.

**Token-based chunking** uses `TokenChunker` with a target of **1700 tokens** per chunk (approximately 5100 characters, using a 3:1 character-to-token ratio). An overlap of roughly 150 tokens ensures semantic continuity between adjacent chunks. This logic resides in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) (lines 68-78).

Each chunk then passes through:
- **Sanitization** (`sanitizeText`): Removes problematic characters that could interfere with storage or retrieval
- **Keyword extraction** (`extractKeywords`): Generates searchable metadata for hybrid search capabilities

These preprocessing steps are implemented in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) (lines 104-115).

## Generating Vector Embeddings with Ollama

Project N.O.M.A.D. uses the `nomic-embed-text:v1.5` model via Ollama to generate embeddings. This model produces **768-dimensional vectors** optimized for semantic similarity tasks.

The embedding process follows these specific rules as implemented in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) (lines 90-115 and 122-140):

1. **Prefixing**: Each chunk is prefixed with `search_document: ` to align with the model's training for asymmetric search
2. **Truncation**: Text is truncated to a safe token budget before being sent to Ollama
3. **Batching**: Chunks are processed in batches of **8** (`RagService.EMBEDDING_BATCH_SIZE = 8`) to reduce HTTP overhead while respecting hardware constraints

```typescript
// Constants from admin/app/services/rag_service.ts
const EMBEDDING_MODEL = 'nomic-embed-text:v1.5'
const EMBEDDING_DIMENSION = 768
const EMBEDDING_BATCH_SIZE = 8

```

## Storing Embeddings in Qdrant

Once generated, embeddings are upserted into the Qdrant collection `nomad_knowledge_base` using cosine similarity for distance calculation.

The payload stored with each vector (lines 336-380 in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts)) includes:
- `text`: The sanitized original content
- `chunk_index` and `total_chunks`: Position metadata
- `keywords`: Extracted search terms
- `source`: File path or origin identifier
- `content_type`: MIME type classification
- ZIM-specific metadata (when applicable): `article_title`, `section_title`, `document_id`

## Practical Implementation Examples

### Directly Embed a Raw String

Use `RagService.embedAndStoreText()` (lines 235-250) to programmatically embed text without file upload:

```typescript
import { RagService } from '#services/rag_service'
import { DockerService } from '#services/docker_service'
import { OllamaService } from '#services/ollama_service'

const rag = new RagService(new DockerService(), new OllamaService())

async function embedSample() {
  const text = `Project Nomad is a self-hosted knowledge-base that enables semantic search across your documents.`
  const result = await rag.embedAndStoreText(text, { 
    source: 'sample.txt',
    content_type: 'text/plain'
  })
  console.log('Embedded chunks:', result?.chunks)
}

embedSample()

```

### Upload via HTTP API

Send files to the upload endpoint for background processing:

```bash
curl -F "file=@my-doc.pdf" \
  -X POST http://localhost:3333/rag/upload \
  -H "Accept: application/json"

```

The endpoint returns a job ID. Query the status via `GET /rag/job-status?filePath=my-doc.pdf` or programmatically:

```typescript
import { EmbedFileJob } from '#jobs/embed_file_job'

async function checkStatus(filePath: string) {
  const job = await EmbedFileJob.getByFilePath(filePath) // lines 80-85
  if (!job) return console.log('Job not found')
  const state = await job.getState()
  console.log('Progress:', state.progress, 'Status:', state.data?.status)
}

```

### Execute Semantic Search

Retrieve relevant documents using `searchSimilarDocuments()` (lines 670-680), which automatically prefixes queries with `search_query: ` for optimal model performance:

```typescript
import { RagService } from '#services/rag_service'
import { DockerService } from '#services/docker_service'
import { OllamaService } from '#services/ollama_service'

const rag = new RagService(new DockerService(), new OllamaService())

async function search() {
  const results = await rag.searchSimilarDocuments(
    'how to backup a ZIM file', 
    5 // top-k results
  )
  
  results.forEach(r => {
    console.log(`Score: ${r.score.toFixed(3)} – ${r.text.slice(0, 120)}…`)
  })
}

search()

```

## Summary

- **Pipeline**: Uploads flow through `RagController` → `EmbedFileJob` → `RagService` before reaching Qdrant
- **Chunking**: Target 1700 tokens per chunk with ~150 token overlap using `TokenChunker`
- **Model**: `nomic-embed-text:v1.5` generates 768-dimensional vectors via Ollama
- **Storage**: Embeddings reside in the `nomad_knowledge_base` Qdrant collection with rich metadata payloads
- **Batching**: Default batch size of 8 chunks optimizes throughput without overwhelming local hardware
- **Search**: Use `searchSimilarDocuments()` with query prefixing for asymmetric semantic search

## Frequently Asked Questions

### What embedding model does Project N.O.M.A.D. use?

Project N.O.M.A.D. uses `nomic-embed-text:v1.5` via Ollama, producing 768-dimensional vectors. This model is specifically instructed with `search_document: ` prefixes during ingestion and `search_query: ` prefixes during retrieval to optimize for asymmetric semantic search, as defined in `RagService.EMBEDDING_MODEL` and `RagService.SEARCH_QUERY_PREFIX`.

### How does Nomad handle large files like ZIM archives?

The `EmbedFileJob` in [`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts) detects ZIM archives and splits processing into batches to prevent job timeouts. The `ZimExtractionService` extracts individual articles with hierarchical metadata (article titles, section paths) that enriches the vector payload for better retrieval context.

### What is the default chunk size and why?

The default chunk size targets **1700 tokens** (approximately 5100 characters), calculated using a 3:1 character-to-token ratio. This balances context preservation with vector database limitations. An overlap of roughly 150 tokens ensures semantic continuity between chunks, preventing information loss at boundaries.

### How do I verify my text was successfully embedded?

Check the job status via `EmbedFileJob.getByFilePath()` (lines 80-85) which returns the BullMQ job state including progress and error details. For direct API uploads, query `GET /rag/job-status?filePath=your-file.ext`. Successfully embedded chunks appear in the `nomad_knowledge_base` Qdrant collection with payloads containing the original text, chunk indices, and extracted keywords.