# Document Text Extraction Flow for Different File Formats in the Macro Repository > Discover the document text extraction flow at Macro. Learn how the platform ingests, converts, and extracts text from various file formats for searchable data and metadata. - Repository: [Macro/macro](https://github.com/macro-inc/macro) - Tags: how-to-guide - Published: 2026-08-15 --- **The Macro platform routes each uploaded file through a three-stage pipeline—ingestion, conversion (for DOCX), and PDF-based text extraction—to produce searchable text and citation metadata.** Understanding how Macro handles document text extraction across file formats requires tracing the data flow from upload through processing. The architecture separates concerns between storage, format-specific conversion, and unified PDF-based extraction, as implemented in the `macro-inc/macro` repository. ## Ingestion: The Document Storage Service All document uploads begin in the **Document Storage Service** at [`services/document_storage_service/src/api/documents/save_document/save.rs`](https://github.com/macro-inc/macro/blob/main/services/document_storage_service/src/api/documents/save_document/save.rs). This service: - Writes raw files to **type-specific S3 buckets** (e.g., `pdf_upload_bucket`, `docx_upload_bucket`) - Records metadata in MacroDB - Triggers downstream processing via S3 events The routing decision occurs at upload time based on file extension, ensuring each format follows its appropriate pipeline. ## DOCX Conversion Pipeline DOCX files require **pre-processing before text extraction** due to their compressed archive structure. The flow involves three components: ### Step 1: Unzip and XML Extraction The **DOCX Unzip Handler** ([`services/docx_unzip_handler/src/service/document/process_document.rs`](https://github.com/macro-inc/macro/blob/main/services/docx_unzip_handler/src/service/document/process_document.rs)) processes DOCX archives: ```rust // Inside the DOCX Unzip Handler let docx_key = build_docx_key(owner, document_id); let docx_bytes = s3_client.get_object(&docx_key).await?; let xml = unzip_docx_extract_xml(&docx_bytes)?; // let plain = docx_parser::parse_docx(&xml)?; // plain text (optional) let pdf_key = build_docx_to_pdf_converted_document_key(owner, document_id); let pdf_bytes = convert_docx_to_pdf(&docx_bytes)?; // uses LibreOffice/Chromium s3_client.put_object(&pdf_key, pdf_bytes).await?; ``` The handler: - Extracts [`document.xml`](https://github.com/macro-inc/macro/blob/main/document.xml) from the ZIP archive - Optionally generates plain text via `docx_parser::parse_docx()` - Converts the DOCX to PDF using an external converter - Stores the result under a `build_docx_to_pdf_converted_document_key` S3 key ### Step 2: PDF Conversion The **Convert Service**—invoked by the unzip handler—produces a temporary PDF. This unified format allows the platform to use a single extraction engine regardless of source format. ## Unified PDF Text Extraction Both **native PDFs** and **DOCX-converted PDFs** flow through the **Document Text Extractor**, an AWS Lambda service. ### Entry Point: Lambda Handler The extractor initializes at [`services/document_text_extractor/src/main.rs`](https://github.com/macro-inc/macro/blob/main/services/document_text_extractor/src/main.rs), setting up the **pdfium-render** instance and S3 event handling. ### Core Extraction Logic The handler in [`services/document_text_extractor/src/handler/extract_text_citations.rs`](https://github.com/macro-inc/macro/blob/main/services/document_text_extractor/src/handler/extract_text_citations.rs) performs the actual work: ```rust // In the Document Text Extractor Lambda #[tracing::instrument(skip(pdfium, s3_client, db) fields(key = key, bucket = bucket))] async fn handler( db: Db, s3_client: S3Client, pdfium: Rc, event: S3Event, ) -> Result<(), anyhow::Error> { let key = event.object.key; // Short-circuit non-PDFs (e.g., images, txt) let file_type = if key.ends_with(".pdf") { "pdf" } else { "other" }; if file_type != "pdf" { tracing::info!(file_type=?file_type, "skipping non-pdf file for extraction"); return Ok(()); } // Load PDF bytes from S3 let raw_file = s3_client.get_object(&key).await?; let pdf = pdfium.load_pdf_from_byte_vec(raw_file, None)?; // Extract references and raw text let (refs, text) = ref_id_extract_text(&pdf); // Persist extracted parts document_text_parts::insert_pdf_references(db, refs, document_id).await?; Ok(()) } ``` Key implementation details: - **Early termination** for non-PDF files with explicit logging - **`pdfium.load_pdf_from_byte_vec()`** for in-memory PDF processing - **`ref_id_extract_text()`** for page traversal and citation metadata - **`document_text_parts::insert_pdf_references()`** for database persistence ### PDF Parsing Engine The underlying PDF parser lives in [`services/search_processing_service/src/parsers/pdf.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/parsers/pdf.rs). It wraps **pdfium-render** to iterate pages and extract raw text plus structured citations. ## Alternative: Direct DOCX Parsing For scenarios requiring DOCX text **without PDF conversion**, the platform provides [`services/search_processing_service/src/parsers/docx.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/parsers/docx.rs). This parser reads [`document.xml`](https://github.com/macro-inc/macro/blob/main/document.xml) directly—useful when conversion overhead is unnecessary. ## Consuming Extracted Text Downstream services fetch processed content via the API: ```rust // API endpoint in Search Processing Service async fn get_document_text( State(state): State, Path(document_id): Path, ) -> Result, ServiceError> { let parts = document_text_parts::fetch_by_document_id(&state.db, &document_id).await?; Ok(Json(DocumentText { pages: parts })) } ``` The `document_text_parts` module abstracts database access for extracted content. ## Summary - **PDF files** proceed directly to the Document Text Extractor using pdfium-render - **DOCX files** undergo unzip, XML extraction, and PDF conversion before entering the same extractor - **Other formats** are either skipped by the extractor (`tracing::info!(file_type=?file_type, "skipping non-pdf file for extraction")`) or handled by format-specific parsers - All extracted text and citations persist through `document_text_parts::insert_pdf_references()` and become retrievable via `document_text_parts::fetch_by_document_id()` ## Frequently Asked Questions ### How does Macro handle password-protected PDFs? The `pdfium.load_pdf_from_byte_vec()` call accepts an optional password parameter. For password-protected files, the extractor would need the password provided at upload time or stored as document metadata—otherwise pdfium returns an error that bubbles up through the handler's `anyhow::Error` result. ### Can the extractor process scanned PDFs without embedded text? pdfium-render extracts embedded text only. For image-based PDFs requiring OCR, the current implementation does not include that capability. Such files would produce empty text output unless pre-processed through an OCR pipeline before reaching the extractor. ### Why convert DOCX to PDF instead of extracting text directly? The conversion approach unifies the extraction codebase—single parser, single citation schema, consistent output format. While [`docx.rs`](https://github.com/macro-inc/macro/blob/main/docx.rs) enables direct XML parsing, the PDF path ensures layout fidelity and handles complex formatting that raw DOCX XML may not represent equivalently. ### Where does the extracted citation metadata come from? The `ref_id_extract_text()` function traverses PDF pages to identify structural elements—headers, footnotes, numbered lists—that become `DocumentReference::Pdf` objects. These citations enable precise search result positioning and context retrieval.