# How to Integrate with the RagController API for RAG Operations in Project Nomad

> Integrate with the RagController API in Project Nomad to upload documents, monitor embedding jobs, and manage Qdrant vector stores for efficient RAG operations. Learn how now.

- Repository: [Crosstalk Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
- Tags: how-to-guide
- Published: 2026-03-16

---

**Use the HTTP endpoints exposed by `RagController` in [`admin/app/controllers/rag_controller.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/controllers/rag_controller.ts) to upload documents, monitor embedding jobs, and manage the Qdrant vector store for retrieval-augmented generation.**

The Project Nomad platform provides a complete retrieval-augmented generation (RAG) pipeline through the **RagController API**, allowing you to programmatically ingest documents, generate 768-dimensional embeddings via Ollama, and store vectors in Qdrant for semantic search. This RESTful interface handles file uploads, background processing via BullMQ, and knowledge base synchronization without blocking your application threads. All routes are registered in [`admin/start/routes.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/start/routes.ts) and delegate business logic to the `RagService` class and `EmbedFileJob` worker.

## Architecture Overview

The RagController API follows a layered architecture that separates HTTP handling from vector generation and storage.

**HTTP Routing Layer**: The [`admin/start/routes.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/start/routes.ts) file maps endpoints to controller actions using AdonisJS router conventions:

```typescript
router.post('/upload', [RagController, 'upload'])
router.get('/files', [RagController, 'getStoredFiles'])
router.delete('/files', [RagController, 'deleteFile'])
router.get('/job-status', [RagController, 'getJobStatus'])
router.post('/sync', [RagController, 'scanAndSync'])

```

**Controller Layer**: Located at [`admin/app/controllers/rag_controller.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/controllers/rag_controller.ts), the `RagController` class validates incoming requests using schemas defined in [`admin/app/validators/rag.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/validators/rag.ts), then forwards operations to `RagService`.

**Service Layer**: The [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) file contains the core `RagService` class, which implements `processAndEmbedFile`, `embedAndStoreText`, `searchSimilarDocuments`, and `deleteFileBySource`. This service detects file types (PDF, image, plain-text, or ZIM), extracts text, chunks content, and upserts vectors into Qdrant.

**Background Job Layer**: Time-consuming embedding operations run asynchronously via [`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts). The `EmbedFileJob` class dispatches the work, reports progress percentages (0-100%), and handles batch processing for ZIM archives.

## Uploading Documents for Embedding

To ingest a document into the knowledge base, send a multipart form-data POST request to the upload endpoint. The controller saves files to `storage/kb_uploads/` and immediately dispatches an embedding job.

```bash
curl -X POST http://localhost:3333/api/rag/upload \
  -F "file=@/path/to/document.pdf"

```

**Response (202 Accepted)**:

```json
{
  "message": "Embedding job queued",
  "jobId": "c3f9e7a1-…",
  "fileName": "document-9b3f2a.pdf",
  "filePath": "/storage/kb_uploads/document-9b3f2a.pdf",
  "alreadyProcessing": false
}

```

The `filePath` value is critical for subsequent status checks. If you upload a file that is already processing, the endpoint returns `"alreadyProcessing": true` without duplicating vectors, ensuring idempotent behavior.

## Monitoring Embedding Progress

Poll the job status endpoint using the exact `filePath` returned from the upload call. The controller queries the BullMQ job payload to return current progress and chunk counts.

```bash
curl "http://localhost:3333/api/rag/job-status?filePath=%2Fstorage%2Fkb_uploads%2Fdocument-9b3f2a.pdf"

```

**In-progress response**:

```json
{
  "exists": true,
  "status": "processing",
  "progress": 57,
  "chunks": 12
}

```

**Completed response**:

```json
{
  "exists": true,
  "status": "completed",
  "progress": 100,
  "chunks": 27
}

```

If extraction or embedding fails, the job payload contains an `error` string, and the controller surfaces a 500 JSON response with the failure message.

## Managing the Knowledge Base

### List Stored Files

Retrieve all indexed sources currently stored in the Qdrant vector database:

```bash
curl http://localhost:3333/api/rag/files

```

```json
{
  "files": [
    "/storage/kb_uploads/report-8c2d1e.pdf",
    "/storage/kb_uploads/notes.txt",
    "/storage/zim/wikipedia_en_100.zim"
  ]
}

```

### Delete Files

Remove a file and its associated vector points by sending the source path in the request body. The `RagService.deleteFileBySource` method validates that the path resides within `storage/kb_uploads` or `storage/zim` to prevent path-traversal attacks before deleting physical files and Qdrant points.

```bash
curl -X DELETE http://localhost:3333/api/rag/files \
  -H "Content-Type: application/json" \
  -d '{"source":"/storage/kb_uploads/old-report.pdf"}'

```

### Synchronize Storage

Trigger a filesystem scan to reconcile manually added files with the vector store index:

```bash
curl -X POST http://localhost:3333/api/rag/sync

```

```json
{
  "success": true,
  "message": "Scanned 42 files, queued 3 for embedding",
  "filesScanned": 42,
  "filesQueued": 3
}

```

The `RagService.scanAndSyncStorage` method walks both upload directories, compares filenames against indexed sources via Qdrant scroll operations, and dispatches `EmbedFileJob` instances for any missing entries.

## Querying the Knowledge Base

To perform a RAG query, use the Ollama chat endpoint with the knowledge base flag enabled. Internally, this calls `RagService.searchSimilarDocuments` to retrieve relevant context before generation.

```bash
curl -X POST http://localhost:3333/api/ollama/chat \
  -H "Content-Type: application/json" \
  -d '{
        "prompt":"Explain the benefits of using a ZIM archive for offline knowledge bases.",
        "knowledge_base":true
      }'

```

The embedding model is automatically downloaded from Ollama if missing, as verified by the `embeddingModelVerified` check inside `RagService.embedAndStoreText`.

## Key Source Files and Implementation Details

| File | Responsibility |
|------|----------------|
| [`admin/start/routes.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/start/routes.ts) | Registers all `/api/rag/*` HTTP endpoints. |
| [`admin/app/controllers/rag_controller.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/controllers/rag_controller.ts) | Validates requests and forwards to services; contains `upload`, `getStoredFiles`, `deleteFile`, `getJobStatus`, and `scanAndSync` methods. |
| [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) | Core RAG pipeline including `processAndEmbedFile`, text extraction, chunking, Qdrant client initialization, and sync logic. |
| [`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts) | BullMQ background worker that executes embedding without blocking the request thread. |
| [`admin/app/validators/rag.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/validators/rag.ts) | Request validation schemas for upload, delete, and status operations. |
| [`admin/types/rag.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/types/rag.ts) | TypeScript definitions for RAG results and embedding responses. |

## Summary

- **Upload documents** via `POST /api/rag/upload` to queue background embedding jobs that store 768-dimensional vectors in Qdrant.
- **Monitor progress** by polling `GET /api/rag/job-status` with the file path returned during upload.
- **Manage sources** using list, delete, and sync endpoints to maintain filesystem and vector store consistency.
- **Query data** through the Ollama chat endpoint with `knowledge_base: true` to leverage semantic search.
- **Security** is enforced at the service layer to prevent path-traversal attacks during file deletion.

## Frequently Asked Questions

### What file types does the RagController API support?

The API supports PDF documents (with OCR), images (OCR), plain-text files, and ZIM archives. The `RagService` class in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) automatically detects file types and applies the appropriate extraction strategy before chunking and embedding.

### How does the API handle large ZIM archives?

Large ZIM files are processed in batches by `EmbedFileJob` in [`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts). The job dispatches follow-up jobs automatically until the entire archive is embedded, preventing memory exhaustion while maintaining progress tracking for each batch.

### Is the document upload endpoint idempotent?

Yes. The upload endpoint checks for existing processing jobs and returns `"alreadyProcessing": true` if the file is already being embedded. This prevents duplicate vector entries in Qdrant when the same file is uploaded multiple times.

### How is the Ollama embedding model managed?

The `RagService` verifies the embedding model status via the `embeddingModelVerified` flag before generating vectors. If the model is missing, the service automatically triggers `ollamaService.downloadModel` to fetch the required 768-dimensional embedding model from the Ollama registry.