How PDF Processing and Translation Works in GPT‑Academic: A Deep Dive into the Source Code

The GPT‑Academic PDF translation pipeline automatically extracts content using DOC2X, GROBID, or a legacy text parser, chunks the text to respect LLM token limits, translates each fragment in parallel via multi‑threaded LLM calls, and compiles the results into markdown and HTML reports.

The binary-husky/gpt_academic repository provides a modular PDF processing and translation system that handles everything from file ingestion to final report generation. This pipeline is orchestrated by the PDF Translate plugin (PDF_Translate.py), which supports three distinct parsing backends and leverages parallel processing to maximize translation throughput. Below is a complete technical breakdown of how the system parses, chunks, translates, and formats academic PDFs.

Architecture and Execution Flow

The pipeline follows a strict eight‑step execution flow defined in crazy_functions/PDF_Translate.py:

1. Input handling – The plugin entry point 批量翻译PDF文档 calls get_files_from_everything (located in crazy_functions/crazy_utils.py) to collect all PDFs from a user‑provided path or drag‑and‑drop operation.
2. Dependency validation – check_packages (from toolbox.py) verifies that fitz (PyMuPDF), tiktoken, and scipdf are installed; missing packages trigger an installation prompt.
3. Backend selection – The system reads plugin_kwargs["pdf_parse_method"]. If the user specifies no method, it automatically attempts DOC2X first, falls back to GROBID, and finally resorts to the Legacy parser.
4. Content extraction – The chosen backend parses the PDF into a structured dictionary containing metadata and sectioned text.
5. Token‑aware chunking – Text is split into fragments that respect the model’s TOKEN_LIMIT_PER_FRAGMENT using the tokenizer specified in model_info.
6. Parallel translation – Each chunk is sent to the configured LLM (OpenAI, Claude, etc.) using request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency.
7. Report assembly – produce_report_markdown generates two markdown files (original + translation and translation‑only) plus a side‑by‑side HTML view.
8. Delivery – Final assets are promoted to the download zone and displayed in the chatbot UI.

The Three PDF Parsing Backends

GPT‑Academic implements three specialized parsers to handle different document types and infrastructure constraints.

DOC2X Integration for High‑Fidelity OCR

The DOC2X backend (crazy_functions/pdf_fns/parse_pdf_via_doc2x.py) provides the highest quality extraction, particularly for scanned or complex academic layouts. The workflow follows four distinct network operations:

1. Pre‑upload – Request a signed URL from the DOC2X API.
2. Upload – PUT the PDF binary to the signed endpoint.
3. Polling – Query /status every few seconds until the server returns success.
4. Conversion – Request /convert/parse with format="md" to receive a ZIP archive containing the extracted markdown.

The resulting markdown is passed to deliver_to_markdown_plugin, which feeds it into the existing Markdown Translate logic. This path requires a valid DOC2X_API_KEY configured via get_conf("DOC2X_API_KEY").

GROBID for Scholarly Metadata

For users hosting their own GROBID service, the GROBID backend (crazy_functions/pdf_fns/parse_pdf_grobid.py) extracts structured scholarly metadata. The function 解析PDF_基于GROBID calls parse_pdf (from pdf_fns/parse_pdf.py), which uses scipdf_parser to contact the GROBID endpoint and returns an article_dict containing:

• title
• authors
• abstract
• sections (list of dictionaries with heading and text)

If GROBID_URLS is defined in the config, get_avail_grobid_url selects the first live endpoint before extraction begins.

Legacy Text Extraction Fallback

When neither DOC2X nor GROBID is available, the system falls back to Legacy mode (crazy_functions/pdf_fns/parse_pdf_legacy.py). The function 解析PDF_简单拆解 uses fitz (PyMuPDF) to read raw text from each page, then immediately applies breakdown_text_to_satisfy_token_limit to split the content into translatable fragments. While this method lacks structural awareness, it requires no external API keys and works entirely offline.

The Translation Engine

Once content is extracted, the core translation logic in pdf_fns/parse_pdf.py handles LLM interaction. The translate_pdf function implements a three‑phase strategy:

Phase 1: Meta‑prompt construction The engine first generates a concise paper summary by sending the title and abstract to request_gpt_model_in_new_thread_with_ui_alive. This establishes context for subsequent section translations.

Phase 2: Token‑limited fragmentation For each section in article_dict['sections'], the system calls break_down (tokenization logic using model_info[llm_kwargs['llm_model']]['tokenizer']) to ensure no fragment exceeds TOKEN_LIMIT_PER_FRAGMENT.

Phase 3: Parallel LLM execution Fragments are translated concurrently via:

gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
    inputs_array=fragments,
    inputs_show_user_array=display_texts,
    llm_kwargs=llm_kwargs,
    chatbot=chatbot,
    history_array=[history] * len(fragments),
    sys_prompt_array=[system_prompt] * len(fragments),
    # Optional additional prompt from plugin_kwargs

    additional_prompt=plugin_kwargs.get('additional_prompt', '')
)

The sys_prompt_array instructs the model to act as an academic translator: "请你作为一个学术翻译，负责把学术论文准确翻译成中文". Parallel execution keeps the UI responsive while maximizing throughput.

Report Generation and Output

After translation completes, produce_report_markdown (in pdf_fns/parse_pdf.py) writes three artifacts:

1. Combined markdown – Original English and Chinese translation interleaved by section.
2. Translation‑only markdown – Clean Chinese text without source material.
3. Side‑by‑side HTML – Generated via construct_html (located in pdf_fns/report_gen_html.py) for browser preview.

Files are timestamped (e.g., 2026-03-02-12-34-56-translated_and_original.md) and promoted to the download zone using toolbox.py utilities.

Usage Example

To programmatically translate a directory of PDFs using the DOC2X backend:

from crazy_functions.PDF_Translate import 批量翻译PDF文档

# Configure the LLM and parsing method

llm_kwargs = {
    "llm_model": "gpt-4o-mini",
    "temperature": 0.2,
    "max_token": 4096
}
plugin_kwargs = {
    "pdf_parse_method": "DOC2X",  # Options: "DOC2X", "GROBID", "Classic"

    "additional_prompt": "Maintain LaTeX formatting in equations."
}

# Execute the translation pipeline

async for status_update in 批量翻译PDF文档(
    txt="/home/user/papers/",
    llm_kwargs=llm_kwargs,
    plugin_kwargs=plugin_kwargs,
    chatbot=chatbot_instance,
    history=[],
    system_prompt="You are a precise academic translator.",
    user_request=None
):
    print(status_update)  # Yields progress messages and final file paths

The coroutine yields status updates throughout the upload, parsing, and translation phases, finally returning links to the generated markdown and HTML files.

Summary

• Three parsing tiers: DOC2X (high‑quality OCR), GROBID (scholarly metadata), and Legacy (offline text split) provide flexibility for different environments and document types.
• Token‑aware chunking: The system uses model‑specific tokenizers via breakdown_text_to_satisfy_token_limit to prevent context window overflow.
• Parallel execution: request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency translates multiple fragments concurrently while keeping the UI alive.
• Structured output: The pipeline produces timestamped markdown files and a side‑by‑side HTML view for easy review.
• Single entry point: All functionality is exposed through 批量翻译PDF文档 in crazy_functions/PDF_Translate.py, making it accessible from the chat interface or programmatically.

Frequently Asked Questions

How does GPT‑Academic choose between DOC2X, GROBID, and the Legacy parser?

The selection logic in PDF_Translate.py checks plugin_kwargs.get("pdf_parse_method"). If the user specifies "DOC2X", "GROBID", or "Classic", that backend is used exclusively. If the parameter is omitted, the system attempts DOC2X first (verifying DOC2X_API_KEY is set), falls back to GROBID (checking get_avail_grobid_url()), and finally defaults to the Legacy parser if neither external service is available.

What is the maximum PDF size or page count supported?

The codebase does not enforce a hard page limit; however, practical constraints arise from the LLM’s token limit and the DOC2X API timeout settings. Large PDFs are handled by breakdown_text_to_satisfy_token_limit, which splits sections into fragments smaller than TOKEN_LIMIT_PER_FRAGMENT (typically 1,024–2,048 tokens depending on the model). DOC2X and GROBID processing times scale with document complexity, but the Legacy parser handles arbitrarily large files as long as they fit in memory.

Can I use a custom LLM model for PDF translation?

Yes. The llm_kwargs dictionary passed to 批量翻译PDF文档 specifies the model via the llm_model key (e.g., "gpt-4o-mini", "claude-3-sonnet"). The system looks up the corresponding tokenizer in model_info to ensure accurate token counting during the chunking phase. Any model supported by the GPT‑Academic core framework can be used for PDF translation.

Where are the translated files saved?

produce_report_markdown writes outputs to a timestamped subdirectory within the project folder (managed by toolbox.py). Files are named using the pattern {timestamp}-translated_and_original.md, {timestamp}-translation.md, and {timestamp}-trans.html. These paths are then promoted to the chatbot’s download zone via the UI helper functions in toolbox.py, allowing users to click and download directly from the interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}