How to Integrate with the RagController API for RAG Operations in Project Nomad

Use the HTTP endpoints exposed by RagController in admin/app/controllers/rag_controller.ts to upload documents, monitor embedding jobs, and manage the Qdrant vector store for retrieval-augmented generation.

The Project Nomad platform provides a complete retrieval-augmented generation (RAG) pipeline through the RagController API, allowing you to programmatically ingest documents, generate 768-dimensional embeddings via Ollama, and store vectors in Qdrant for semantic search. This RESTful interface handles file uploads, background processing via BullMQ, and knowledge base synchronization without blocking your application threads. All routes are registered in admin/start/routes.ts and delegate business logic to the RagService class and EmbedFileJob worker.

Architecture Overview

The RagController API follows a layered architecture that separates HTTP handling from vector generation and storage.

HTTP Routing Layer: The admin/start/routes.ts file maps endpoints to controller actions using AdonisJS router conventions:

router.post('/upload', [RagController, 'upload'])
router.get('/files', [RagController, 'getStoredFiles'])
router.delete('/files', [RagController, 'deleteFile'])
router.get('/job-status', [RagController, 'getJobStatus'])
router.post('/sync', [RagController, 'scanAndSync'])

Controller Layer: Located at admin/app/controllers/rag_controller.ts, the RagController class validates incoming requests using schemas defined in admin/app/validators/rag.ts, then forwards operations to RagService.

Service Layer: The admin/app/services/rag_service.ts file contains the core RagService class, which implements processAndEmbedFile, embedAndStoreText, searchSimilarDocuments, and deleteFileBySource. This service detects file types (PDF, image, plain-text, or ZIM), extracts text, chunks content, and upserts vectors into Qdrant.

Background Job Layer: Time-consuming embedding operations run asynchronously via admin/app/jobs/embed_file_job.ts. The EmbedFileJob class dispatches the work, reports progress percentages (0-100%), and handles batch processing for ZIM archives.

Uploading Documents for Embedding

To ingest a document into the knowledge base, send a multipart form-data POST request to the upload endpoint. The controller saves files to storage/kb_uploads/ and immediately dispatches an embedding job.

curl -X POST http://localhost:3333/api/rag/upload \
  -F "file=@/path/to/document.pdf"

Response (202 Accepted):

{
  "message": "Embedding job queued",
  "jobId": "c3f9e7a1-…",
  "fileName": "document-9b3f2a.pdf",
  "filePath": "/storage/kb_uploads/document-9b3f2a.pdf",
  "alreadyProcessing": false
}

The filePath value is critical for subsequent status checks. If you upload a file that is already processing, the endpoint returns "alreadyProcessing": true without duplicating vectors, ensuring idempotent behavior.

Monitoring Embedding Progress

Poll the job status endpoint using the exact filePath returned from the upload call. The controller queries the BullMQ job payload to return current progress and chunk counts.

curl "http://localhost:3333/api/rag/job-status?filePath=%2Fstorage%2Fkb_uploads%2Fdocument-9b3f2a.pdf"

In-progress response:

{
  "exists": true,
  "status": "processing",
  "progress": 57,
  "chunks": 12
}

Completed response:

{
  "exists": true,
  "status": "completed",
  "progress": 100,
  "chunks": 27
}

If extraction or embedding fails, the job payload contains an error string, and the controller surfaces a 500 JSON response with the failure message.

Managing the Knowledge Base

List Stored Files

Retrieve all indexed sources currently stored in the Qdrant vector database:

curl http://localhost:3333/api/rag/files

{
  "files": [
    "/storage/kb_uploads/report-8c2d1e.pdf",
    "/storage/kb_uploads/notes.txt",
    "/storage/zim/wikipedia_en_100.zim"
  ]
}

Delete Files

Remove a file and its associated vector points by sending the source path in the request body. The RagService.deleteFileBySource method validates that the path resides within storage/kb_uploads or storage/zim to prevent path-traversal attacks before deleting physical files and Qdrant points.

curl -X DELETE http://localhost:3333/api/rag/files \
  -H "Content-Type: application/json" \
  -d '{"source":"/storage/kb_uploads/old-report.pdf"}'

Synchronize Storage

Trigger a filesystem scan to reconcile manually added files with the vector store index:

curl -X POST http://localhost:3333/api/rag/sync

{
  "success": true,
  "message": "Scanned 42 files, queued 3 for embedding",
  "filesScanned": 42,
  "filesQueued": 3
}

The RagService.scanAndSyncStorage method walks both upload directories, compares filenames against indexed sources via Qdrant scroll operations, and dispatches EmbedFileJob instances for any missing entries.

Querying the Knowledge Base

To perform a RAG query, use the Ollama chat endpoint with the knowledge base flag enabled. Internally, this calls RagService.searchSimilarDocuments to retrieve relevant context before generation.

curl -X POST http://localhost:3333/api/ollama/chat \
  -H "Content-Type: application/json" \
  -d '{
        "prompt":"Explain the benefits of using a ZIM archive for offline knowledge bases.",
        "knowledge_base":true
      }'

The embedding model is automatically downloaded from Ollama if missing, as verified by the embeddingModelVerified check inside RagService.embedAndStoreText.

Key Source Files and Implementation Details

File	Responsibility
admin/start/routes.ts	Registers all /api/rag/* HTTP endpoints.
admin/app/controllers/rag_controller.ts	Validates requests and forwards to services; contains upload, getStoredFiles, deleteFile, getJobStatus, and scanAndSync methods.
admin/app/services/rag_service.ts	Core RAG pipeline including processAndEmbedFile, text extraction, chunking, Qdrant client initialization, and sync logic.
admin/app/jobs/embed_file_job.ts	BullMQ background worker that executes embedding without blocking the request thread.
admin/app/validators/rag.ts	Request validation schemas for upload, delete, and status operations.
admin/types/rag.ts	TypeScript definitions for RAG results and embedding responses.

Summary

• Upload documents via POST /api/rag/upload to queue background embedding jobs that store 768-dimensional vectors in Qdrant.
• Monitor progress by polling GET /api/rag/job-status with the file path returned during upload.
• Manage sources using list, delete, and sync endpoints to maintain filesystem and vector store consistency.
• Query data through the Ollama chat endpoint with knowledge_base: true to leverage semantic search.
• Security is enforced at the service layer to prevent path-traversal attacks during file deletion.

Frequently Asked Questions

What file types does the RagController API support?

The API supports PDF documents (with OCR), images (OCR), plain-text files, and ZIM archives. The RagService class in admin/app/services/rag_service.ts automatically detects file types and applies the appropriate extraction strategy before chunking and embedding.

How does the API handle large ZIM archives?

Large ZIM files are processed in batches by EmbedFileJob in admin/app/jobs/embed_file_job.ts. The job dispatches follow-up jobs automatically until the entire archive is embedded, preventing memory exhaustion while maintaining progress tracking for each batch.

Is the document upload endpoint idempotent?

Yes. The upload endpoint checks for existing processing jobs and returns "alreadyProcessing": true if the file is already being embedded. This prevents duplicate vector entries in Qdrant when the same file is uploaded multiple times.

How is the Ollama embedding model managed?

The RagService verifies the embedding model status via the embeddingModelVerified flag before generating vectors. If the model is missing, the service automatically triggers ollamaService.downloadModel to fetch the required 768-dimensional embedding model from the Ollama registry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}