# What File Formats Are Supported for Embedding and RAG Processing in Project Nomad?

> Discover supported file formats for Project Nomad's RAG processing including images PDFs text Office documents and ZIM archives Learn how Nomad handles diverse data for efficient embedding

- Repository: [Crosstalk Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
- Tags: api-reference
- Published: 2026-03-16

---

**Project Nomad supports four distinct file categories for RAG processing: images (.jpg, .png, .webp, etc.), PDFs, plain-text and Office documents (.txt, .md, .docx, .rtf), and ZIM archives, with each type routed through specialized extraction pipelines before embedding.**

Project Nomad is an open-source field knowledge management system developed by Crosstalk-Solutions. Understanding what file formats are supported for embedding and RAG processing is essential for preparing your knowledge base, as each category undergoes distinct extraction and sanitization steps before vectorization.

## Supported File Categories and Extensions

The ingestion pipeline in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) recognizes four distinct file categories based on extension mapping in [`admin/app/utils/fs.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/fs.ts).

### Images (OCR Processing)

Image files including **.jpg**, **.jpeg**, **.png**, **.gif**, **.bmp**, **.tiff**, and **.webp** are processed through an OCR pipeline using Tesseract. The extracted visible text is then chunked, sanitized, and embedded. This enables searchable indexing of scanned field notes and photographed documents.

### PDF Documents

Files with the **.pdf** extension are parsed using `pdf-parse` and `pdf2pic` (for image-based PDFs). The extracted plain text undergoes sanitization before embedding, making manuals and SOPs available for semantic search.

### Plain-Text and Office Documents

Text-based files including **.txt**, **.md**, **.docx**, and **.rtf** provide the fastest path to raw text. DOCX and RTF files are unpacked via lightweight parsers to retain formatting without heavy dependencies, then forwarded directly to the embedding routine.

### ZIM Archives (Specialized Batch Processing)

**ZIM** archives receive special treatment in the pipeline. Unlike other formats, ZIM files are streamed and unpacked in batches via [`admin/app/services/zim_extraction_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/zim_extraction_service.ts). Each article's HTML is stripped to plain text, and the resulting chunks are embedded incrementally to maintain low memory usage on constrained hardware.

## How File Type Detection Works

The detection logic resides in **[`admin/app/utils/fs.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/fs.ts)**, where the `determineFileType()` function maps file extensions to their respective categories. This utility returns a category string—`image`, `pdf`, `text`, or `zim`—that determines which extraction handler the RAG service will invoke.

## The RAG Processing Pipeline

When a file arrives for ingestion, **`RagService.processAndEmbedFile()`** in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) orchestrates the workflow:

1. Calls `determineFileType()` to identify the category
2. Loads the file (except for ZIM, which streams directly)
3. Dispatches to the appropriate handler:
   - **Image** → `processImageFile()`
   - **PDF** → `processPDFFile()`
   - **Text** → `processTextFile()`
   - **ZIM** → `processZIMFile()`

If the extension is not recognized, the service returns "Unsupported file type" and aborts the embedding step.

## Code Examples

### Embedding a Single File

```typescript
import { RagService } from '@/app/services/rag_service.js';

async function embedExample() {
  const rag = new RagService(dockerService, ollamaService);
  const result = await rag.processAndEmbedFile(
    '/uploads/manual.pdf',   // path on the host
    true,                    // delete after successful embed
    undefined,               // optional batch offset (only for .zim)
    async (percent) => console.log(`Progress: ${percent}%`)
  );

  console.log(result);
}

```

### Determining File Type Programmatically

```typescript
import { determineFileType } from '@/app/utils/fs.js';

const type = determineFileType('report.docx'); // → 'text'

```

### Processing ZIM Archives in Batches

```typescript
await rag.processAndEmbedFile(
  '/uploads/knowledge.zim',
  false,
  0,                       // start at first batch
  (p) => console.log(`ZIM batch progress: ${p}%`)
);

```

## Key Implementation Files

| File | Role |
|------|------|
| [`admin/app/utils/fs.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/fs.ts) | Implements `determineFileType()` and file-system helpers used by the RAG pipeline. |
| [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) | Core RAG service – orchestrates type detection, extraction, chunking, embedding, and cleanup. |
| [`admin/app/services/zim_extraction_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/zim_extraction_service.ts) | Specialized logic for parsing ZIM archives and feeding their content into the embedding flow. |
| [`admin/app/services/ollama_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/ollama_service.ts) | Provides the embedding model (`nomic-embed-text:v1.5`) used by the RAG service. |

## Summary

- Project Nomad supports **four file categories**: images, PDFs, plain-text/Office documents, and ZIM archives.
- **Image files** undergo OCR processing via Tesseract before embedding.
- **PDFs** are parsed using `pdf-parse` and `pdf2pic` to handle both text and image-based content.
- **Text files** (.txt, .md, .docx, .rtf) are processed directly with minimal overhead.
- **ZIM archives** use specialized batch processing to handle large knowledge bases efficiently.
- File type detection occurs in [`admin/app/utils/fs.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/fs.ts), while [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts) manages the embedding pipeline.

## Frequently Asked Questions

### Can Project Nomad process scanned PDFs that contain only images?

Yes. According to the source code in [`admin/app/services/rag_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/rag_service.ts), PDFs are processed using both `pdf-parse` for text extraction and `pdf2pic` to handle image-based pages. If a PDF contains no extractable text, the image conversion pipeline ensures the content is still accessible via OCR before embedding.

### What happens if I upload an unsupported file format?

If `determineFileType()` in [`admin/app/utils/fs.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/fs.ts) cannot map the file extension to a supported category, `RagService.processAndEmbedFile()` returns the message "Unsupported file type" and aborts the embedding process immediately. No partial processing or storage occurs for unrecognized formats.

### Why does Project Nomad support ZIM archives specifically?

ZIM archives are supported because Project Nomad ships with a pre-built knowledge base from the Kiwix project. The specialized handling in [`admin/app/services/zim_extraction_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/zim_extraction_service.ts) unpacks these archives in batches, stripping HTML to plain text while keeping memory usage low—critical for deployment on constrained field hardware.

### Which embedding model does Project Nomad use for these file formats?

Regardless of the input file format, all extracted text is embedded using the `nomic-embed-text:v1.5` model, as implemented in [`admin/app/services/ollama_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/ollama_service.ts). This provides consistent vector representations across images, PDFs, text documents, and ZIM content for unified semantic search.