api-reference

Supported Languages for olmocr: HTML-Based Language Detection and Normalization

July 7, 2026 allenai/olmocr ↗

OLMOCR determines the supported language of a document by extracting the ISO-639-1 code from the HTML <html lang="..."> attribute, normalizing it to a two-letter code (e.g., pt-BR → pt), and storing it in the primary_language metadata field, defaulting to English (en) when the attribute is absent.

The Allen AI olmocr pipeline processes multilingual documents by detecting language codes embedded in HTML templates. Understanding the supported languages for olmocr is straightforward: the system extracts the lang attribute from the <html> tag, normalizes regional variants to standard two-letter ISO-639-1 codes, and records this value in the document metadata. This design ensures compatibility with any valid ISO-639-1 language code while maintaining consistency across the processing pipeline.

How olmocr Detects Document Languages

OLMOCR relies on the HTML lang attribute to establish document language. During the synthesis phase, the extract_html_metadata function in olmocr/synth/mine_html_templates.py parses the HTML structure to locate the <html> tag and extract its language property.

The detection process follows these steps:

  1. Parse the HTML to locate the root <html> element
  2. Extract the lang attribute value (e.g., pt-BR, en, es)
  3. Normalize to ISO-639-1 by stripping regional variants to two-letter codes (e.g., pt-BR becomes pt)
  4. Store in metadata within the primary_language field
  5. Default to English (en) if the attribute is missing or unparsable

This mechanism allows olmocr to support any valid ISO-639-1 language code, including ja (Japanese), zh (Chinese), and ru (Russian), provided the HTML source correctly specifies the language.

Verified Language Support and Test Coverage

The test suite in tests/test_mine_html_templates_metadata.py explicitly validates language extraction for several major languages. The following codes are verified in the current codebase:

  • English (en) – Validated in test_extract_metadata_table_heavy_document
  • Portuguese (pt) – Validated in test_extract_metadata_portuguese_document
  • Spanish (es) – Validated in test_extract_metadata_image_heavy_document
  • German (de) – Validated in test_extract_metadata_empty_body
  • French (fr) – Referenced in pipeline tests within tests/test_grpo.py

While these languages have explicit test coverage, the normalization logic accepts any standard two-letter ISO-639-1 code.

Code Implementation and Examples

The language detection logic resides in olmocr/synth/mine_html_templates.py and is exercised by unit tests in tests/test_mine_html_templates_metadata.py.

Detecting Portuguese Documents

from olmocr.synth.mine_html_templates import extract_html_metadata

html_pt = '<html lang="pt"><body><p>Texto em português.</p></body></html>'
metadata_pt = extract_html_metadata(html_pt)
print(metadata_pt["primary_language"])  # Output: "pt"

Normalizing Regional Variants

from olmocr.synth.mine_html_templates import extract_html_metadata

# Brazilian Portuguese normalizes to standard Portuguese

html_pt_br = '<html lang="pt-BR"><body><p>Texto em português brasileiro.</p></body></html>'
metadata_pt_br = extract_html_metadata(html_pt_br)
print(metadata_pt_br["primary_language"])  # Output: "pt"

Handling Missing Language Attributes

from olmocr.synth.mine_html_templates import extract_html_metadata

# Missing lang attribute defaults to English

html_no_lang = '<body><p>No language attribute.</p></body>'
metadata_default = extract_html_metadata(html_no_lang)
print(metadata_default["primary_language"])  # Output: "en"

Key Implementation Files

Language detection is implemented across the following files:

Summary

  • OLMOCR detects document languages from the HTML <html lang="..."> attribute using the extract_html_metadata function.
  • Regional language codes (e.g., pt-BR, en-US) normalize to two-letter ISO-639-1 codes (e.g., pt, en).
  • The system supports any valid ISO-639-1 language code, not just the explicitly tested languages.
  • Missing or unparsable language attributes default to English (en).
  • Language metadata is stored in the primary_language field of the document metadata object.

Frequently Asked Questions

What happens if the HTML lang attribute is missing?

If the <html> tag lacks a lang attribute or the value cannot be parsed, olmocr defaults to English (en) as the primary_language according to the implementation in olmocr/synth/mine_html_templates.py.

Does olmocr support regional language variants like pt-BR or en-US?

Yes. The extract_html_metadata function normalizes regional variants by extracting only the first two letters of the language code. For example, pt-BR becomes pt and en-US becomes en, maintaining compatibility with the ISO-639-1 standard.

How can I verify language detection is working correctly?

Run the unit tests in tests/test_mine_html_templates_metadata.py, which validate extraction for Portuguese, Spanish, German, and English. Additional integration tests in tests/test_grpo.py verify that the primary_language field propagates correctly through the complete pipeline.

Is language detection automatic or does it require configuration?

Language detection is automatic. OLMOCR reads the lang attribute from the HTML <html> tag during document processing without requiring additional configuration or manual language selection.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →