How the i18n Translation System Works for Lessons in AI Engineering From Scratch

The i18n translation system uses a three-stage protect-translate-restore pipeline that shields technical markup while translating lesson prose via NLLB-200 or commercial APIs, with SHA-256 caching to skip unchanged files.

The rohitg00/ai-engineering-from-scratch repository implements a self-contained internationalization pipeline that converts English lesson markdown into dozens of language-specific versions. This system preserves executable code blocks, LaTeX math, and inline formatting while leveraging either free open-source models or paid APIs to generate localized curriculum content.

Language Registry and NLLB-200 Codes

The pipeline begins with languages.json, which serves as the canonical registry of supported languages. Each entry maps a BCP-47 language code to its corresponding FLORES-200 identifier required by the NLLB-200 model.

{
  "code": "fr",
  "name": "French",
  "native": "Français",
  "nllb": "fra_Latn",
  "ci": true
}

In scripts/translate_lessons.py, the _load_registry() function loads this JSON once and caches it in _REG, LANG_NAMES, and NLLB_CODES dictionaries. When the provider is set to nllb, the script looks up the nllb key to retrieve the target language token (e.g., fra_Latn for French) before instantiating the translation pipeline.

The Protect-Translate-Restore Pipeline

The core translation logic in translate_lessons.py follows a strict three-stage process to ensure technical content never reaches the translation model.

Stage 1: Protecting Technical Spans

The protect(text, patterns=PROTECT) function scans the input markdown against a comprehensive regex list. Matches—including fenced code blocks, inline code, LaTeX math, images, bare URLs, and raw HTML—are replaced with invisible sentinel tokens (⁣PROTECT<n>⁣). The original substrings are stored in a list for later restoration.

For prose lines, the system uses a narrower NLLB_INLINE pattern set that specifically targets inline code, inline math, images, markdown links, bold spans, and URLs. This selective protection ensures that structural markdown remains translatable while safeguarding executable syntax.

Stage 2: Translation Providers

After protection, the translate_text() function dispatches the masked content to the configured provider:

• NLLB-200: The default free provider uses a lightweight HuggingFace pipeline (_nllb_pipe). Text is split into sentences via _nllb_sentence() to respect the model's ~512-token limit before being translated individually.
• Anthropic/OpenAI/DeepL: Commercial providers available via --provider flag or the TRANSLATE_PROVIDER environment variable, reading credentials from LLM_API_KEY.

The "echo" provider returns input unchanged for testing purposes.

Stage 3: Restoring Preserved Content

The restore(text, store) function walks the sentinel list in reverse order, substituting each placeholder with its original protected substring. This reverse iteration guarantees that nested spans (such as links containing URLs) are correctly reinstated without collision.

Incremental Translation with SHA-256 Caching

Translation operations are expensive, so the system implements file-level caching under i18n/<lang>/.translate-cache.json or i18n/<lang>/.cache/<phase>.json.

For each lesson file discovered via lesson_docs() (which walks phases/*/*/docs/en.md), the script computes a SHA-256 hash of the source content. If this source_hash matches the cached entry and the translated file exists at i18n/<lang>/…/<lang>.md, the lesson is skipped. Otherwise, the file is translated and save_cache() updates the registry immediately, allowing interrupted CI runs to resume without re-translating completed lessons.

Multi-Provider Architecture

The script supports a plugin-style architecture for translation backends. Provider selection defaults to nllb but can be overridden via CLI:

# Use Anthropic's Claude model

python3 scripts/translate_lessons.py --lang ja --provider anthropic

# Use DeepL API

LLM_API_KEY=xxx python3 scripts/translate_lessons.py --lang de --provider deepl

Each provider implements a specific translation function (_anthropic(), _openai(), _deepl()) that receives the protected text and target language code, returning translated prose that maintains the sentinel markers for the restoration phase.

CLI Usage and CI Integration

The targets() generator identifies canonical lesson files while deliberately excluding the site-wide README, which is hand-translated via scripts/build_readme_i18n.py.

Typical invocations include:

# Translate all lessons to Spanish

python3 scripts/translate_lessons.py --lang es

# Translate only the NLP foundations phase to Turkish

python3 scripts/translate_lessons.py --lang tr --phase 05-nlp-foundations-to-advanced

# Dry run to preview output paths without API calls

python3 scripts/translate_lessons.py --lang fr --only phases/01-foundations/01-hello-world --dry-run

The .github/workflows/translate.yml CI workflow orchestrates these commands per language, pushing results to the i18n/<lang>/ directory tree where the site generator consumes them for localized builds.

Summary

• Protect-Translate-Restore: The pipeline masks technical markup with sentinels before translation, then restores exact substrings to preserve code executability.
• Registry-Driven: languages.json maps BCP-47 codes to NLLB-200 tokens, enabling support for 200+ languages through a single configuration file.
• Provider Agnostic: Supports NLLB-200 (free), Anthropic, OpenAI, and DeepL via uniform interface functions and environment variables.
• Incremental Caching: SHA-256 hashing prevents re-translating unchanged lessons, storing state in JSON cache files per language or phase.
• Selective Processing: The targets() generator excludes the README from machine translation, routing it to a separate hand-translation workflow.

Frequently Asked Questions

What is the protect-translate-restore pattern and why is it necessary?

The protect-translate-restore pattern ensures that Large Language Models or machine translation systems do not alter executable code, mathematical notation, or structural URLs. By replacing these spans with invisible sentinels before translation and substituting them back afterward, the system guarantees that a Python function signature or LaTeX equation remains identical across all 40+ language versions of a lesson.

Which translation providers are supported by the system?

The translate_lessons.py script supports four production providers and one test provider: NLLB-200 (default, free, no API key required), Anthropic (Claude models), OpenAI (GPT models), and DeepL (specialized translation API). An "echo" provider returns input unchanged for debugging. Selection occurs via the --provider CLI flag or TRANSLATE_PROVIDER environment variable.

How does the caching mechanism determine which files to skip?

Before processing, the script calculates a SHA-256 hash of the source markdown file. This source_hash is compared against entries in i18n/<lang>/.translate-cache.json. If the hash matches and the output file exists, the lesson is skipped entirely. This hash-based approach detects any content changes, including minor typo fixes, while ignoring file timestamps.

Why is the README excluded from the automatic lesson translation?

The repository treats the README as site-wide navigation infrastructure rather than pedagogical content. The targets() generator deliberately omits the README from the translation queue because it contains high-level curriculum descriptions and navigation links that require cultural adaptation beyond literal translation. Instead, scripts/build_readme_i18n.py manages hand-authored README variants for each language.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}