Document Text Extraction Flow for Different File Formats in the Macro Repository

The Macro platform routes each uploaded file through a three-stage pipeline—ingestion, conversion (for DOCX), and PDF-based text extraction—to produce searchable text and citation metadata.

Understanding how Macro handles document text extraction across file formats requires tracing the data flow from upload through processing. The architecture separates concerns between storage, format-specific conversion, and unified PDF-based extraction, as implemented in the macro-inc/macro repository.

Ingestion: The Document Storage Service

All document uploads begin in the Document Storage Service at services/document_storage_service/src/api/documents/save_document/save.rs. This service:

• Writes raw files to type-specific S3 buckets (e.g., pdf_upload_bucket, docx_upload_bucket)
• Records metadata in MacroDB
• Triggers downstream processing via S3 events

The routing decision occurs at upload time based on file extension, ensuring each format follows its appropriate pipeline.

DOCX Conversion Pipeline

DOCX files require pre-processing before text extraction due to their compressed archive structure. The flow involves three components:

Step 1: Unzip and XML Extraction

The DOCX Unzip Handler (services/docx_unzip_handler/src/service/document/process_document.rs) processes DOCX archives:

// Inside the DOCX Unzip Handler
let docx_key = build_docx_key(owner, document_id);
let docx_bytes = s3_client.get_object(&docx_key).await?;
let xml = unzip_docx_extract_xml(&docx_bytes)?;          // <document.xml>
let plain = docx_parser::parse_docx(&xml)?;               // plain text (optional)
let pdf_key = build_docx_to_pdf_converted_document_key(owner, document_id);
let pdf_bytes = convert_docx_to_pdf(&docx_bytes)?;       // uses LibreOffice/Chromium
s3_client.put_object(&pdf_key, pdf_bytes).await?;

The handler:

• Extracts document.xml from the ZIP archive
• Optionally generates plain text via docx_parser::parse_docx()
• Converts the DOCX to PDF using an external converter
• Stores the result under a build_docx_to_pdf_converted_document_key S3 key

Step 2: PDF Conversion

The Convert Service—invoked by the unzip handler—produces a temporary PDF. This unified format allows the platform to use a single extraction engine regardless of source format.

Unified PDF Text Extraction

Both native PDFs and DOCX-converted PDFs flow through the Document Text Extractor, an AWS Lambda service.

Entry Point: Lambda Handler

The extractor initializes at services/document_text_extractor/src/main.rs, setting up the pdfium-render instance and S3 event handling.

Core Extraction Logic

The handler in services/document_text_extractor/src/handler/extract_text_citations.rs performs the actual work:

// In the Document Text Extractor Lambda
#[tracing::instrument(skip(pdfium, s3_client, db) fields(key = key, bucket = bucket))]
async fn handler(
    db: Db,
    s3_client: S3Client,
    pdfium: Rc<Pdfium>,
    event: S3Event,
) -> Result<(), anyhow::Error> {
    let key = event.object.key;
    // Short-circuit non-PDFs (e.g., images, txt)
    let file_type = if key.ends_with(".pdf") { "pdf" } else { "other" };
    if file_type != "pdf" {
        tracing::info!(file_type=?file_type, "skipping non-pdf file for extraction");
        return Ok(());
    }

    // Load PDF bytes from S3
    let raw_file = s3_client.get_object(&key).await?;
    let pdf = pdfium.load_pdf_from_byte_vec(raw_file, None)?;
    // Extract references and raw text
    let (refs, text) = ref_id_extract_text(&pdf);
    // Persist extracted parts
    document_text_parts::insert_pdf_references(db, refs, document_id).await?;
    Ok(())
}

Key implementation details:

• Early termination for non-PDF files with explicit logging
• pdfium.load_pdf_from_byte_vec() for in-memory PDF processing
• ref_id_extract_text() for page traversal and citation metadata
• document_text_parts::insert_pdf_references() for database persistence

PDF Parsing Engine

The underlying PDF parser lives in services/search_processing_service/src/parsers/pdf.rs. It wraps pdfium-render to iterate pages and extract raw text plus structured citations.

Alternative: Direct DOCX Parsing

For scenarios requiring DOCX text without PDF conversion, the platform provides services/search_processing_service/src/parsers/docx.rs. This parser reads document.xml directly—useful when conversion overhead is unnecessary.

Consuming Extracted Text

Downstream services fetch processed content via the API:

// API endpoint in Search Processing Service
async fn get_document_text(
    State(state): State<AppState>,
    Path(document_id): Path<String>,
) -> Result<Json<DocumentText>, ServiceError> {
    let parts = document_text_parts::fetch_by_document_id(&state.db, &document_id).await?;
    Ok(Json(DocumentText { pages: parts }))
}

The document_text_parts module abstracts database access for extracted content.

Summary

• PDF files proceed directly to the Document Text Extractor using pdfium-render
• DOCX files undergo unzip, XML extraction, and PDF conversion before entering the same extractor
• Other formats are either skipped by the extractor (tracing::info!(file_type=?file_type, "skipping non-pdf file for extraction")) or handled by format-specific parsers
• All extracted text and citations persist through document_text_parts::insert_pdf_references() and become retrievable via document_text_parts::fetch_by_document_id()

Frequently Asked Questions

How does Macro handle password-protected PDFs?

The pdfium.load_pdf_from_byte_vec() call accepts an optional password parameter. For password-protected files, the extractor would need the password provided at upload time or stored as document metadata—otherwise pdfium returns an error that bubbles up through the handler's anyhow::Error result.

Can the extractor process scanned PDFs without embedded text?

pdfium-render extracts embedded text only. For image-based PDFs requiring OCR, the current implementation does not include that capability. Such files would produce empty text output unless pre-processed through an OCR pipeline before reaching the extractor.

Why convert DOCX to PDF instead of extracting text directly?

The conversion approach unifies the extraction codebase—single parser, single citation schema, consistent output format. While docx.rs enables direct XML parsing, the PDF path ensures layout fidelity and handles complex formatting that raw DOCX XML may not represent equivalently.

Where does the extracted citation metadata come from?

The ref_id_extract_text() function traverses PDF pages to identify structural elements—headers, footnotes, numbered lists—that become DocumentReference::Pdf objects. These citations enable precise search result positioning and context retrieval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}