# How the i18n Translation System Works for Lessons in AI Engineering From Scratch

> Learn how the i18n translation system works for AI engineering lessons. Discover its protect-translate-restore pipeline, NLLB-200 integration, and SHA-256 caching.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-09-01

---

**The i18n translation system uses a three-stage protect-translate-restore pipeline that shields technical markup while translating lesson prose via NLLB-200 or commercial APIs, with SHA-256 caching to skip unchanged files.**

The `rohitg00/ai-engineering-from-scratch` repository implements a self-contained internationalization pipeline that converts English lesson markdown into dozens of language-specific versions. This system preserves executable code blocks, LaTeX math, and inline formatting while leveraging either free open-source models or paid APIs to generate localized curriculum content.

## Language Registry and NLLB-200 Codes

The pipeline begins with [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json), which serves as the canonical registry of supported languages. Each entry maps a BCP-47 language code to its corresponding FLORES-200 identifier required by the NLLB-200 model.

```json
{
  "code": "fr",
  "name": "French",
  "native": "Français",
  "nllb": "fra_Latn",
  "ci": true
}

```

In [`scripts/translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/translate_lessons.py), the `_load_registry()` function loads this JSON once and caches it in `_REG`, `LANG_NAMES`, and `NLLB_CODES` dictionaries. When the provider is set to `nllb`, the script looks up the `nllb` key to retrieve the target language token (e.g., `fra_Latn` for French) before instantiating the translation pipeline.

## The Protect-Translate-Restore Pipeline

The core translation logic in [`translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/translate_lessons.py) follows a strict three-stage process to ensure technical content never reaches the translation model.

### Stage 1: Protecting Technical Spans

The `protect(text, patterns=PROTECT)` function scans the input markdown against a comprehensive regex list. Matches—including fenced code blocks, inline code, LaTeX math, images, bare URLs, and raw HTML—are replaced with invisible sentinel tokens (`⁣PROTECT<n>⁣`). The original substrings are stored in a list for later restoration.

For prose lines, the system uses a narrower `NLLB_INLINE` pattern set that specifically targets inline code, inline math, images, markdown links, bold spans, and URLs. This selective protection ensures that structural markdown remains translatable while safeguarding executable syntax.

### Stage 2: Translation Providers

After protection, the `translate_text()` function dispatches the masked content to the configured provider:

- **NLLB-200**: The default free provider uses a lightweight HuggingFace pipeline (`_nllb_pipe`). Text is split into sentences via `_nllb_sentence()` to respect the model's ~512-token limit before being translated individually.
- **Anthropic/OpenAI/DeepL**: Commercial providers available via `--provider` flag or the `TRANSLATE_PROVIDER` environment variable, reading credentials from `LLM_API_KEY`.

The `"echo"` provider returns input unchanged for testing purposes.

### Stage 3: Restoring Preserved Content

The `restore(text, store)` function walks the sentinel list in reverse order, substituting each placeholder with its original protected substring. This reverse iteration guarantees that nested spans (such as links containing URLs) are correctly reinstated without collision.

## Incremental Translation with SHA-256 Caching

Translation operations are expensive, so the system implements file-level caching under `i18n/<lang>/.translate-cache.json` or `i18n/<lang>/.cache/<phase>.json`.

For each lesson file discovered via `lesson_docs()` (which walks `phases/*/*/docs/en.md`), the script computes a SHA-256 hash of the source content. If this `source_hash` matches the cached entry and the translated file exists at `i18n/<lang>/…/<lang>.md`, the lesson is skipped. Otherwise, the file is translated and `save_cache()` updates the registry immediately, allowing interrupted CI runs to resume without re-translating completed lessons.

## Multi-Provider Architecture

The script supports a plugin-style architecture for translation backends. Provider selection defaults to `nllb` but can be overridden via CLI:

```bash

# Use Anthropic's Claude model

python3 scripts/translate_lessons.py --lang ja --provider anthropic

# Use DeepL API

LLM_API_KEY=xxx python3 scripts/translate_lessons.py --lang de --provider deepl

```

Each provider implements a specific translation function (`_anthropic()`, `_openai()`, `_deepl()`) that receives the protected text and target language code, returning translated prose that maintains the sentinel markers for the restoration phase.

## CLI Usage and CI Integration

The `targets()` generator identifies canonical lesson files while deliberately excluding the site-wide README, which is hand-translated via [`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py).

Typical invocations include:

```bash

# Translate all lessons to Spanish

python3 scripts/translate_lessons.py --lang es

# Translate only the NLP foundations phase to Turkish

python3 scripts/translate_lessons.py --lang tr --phase 05-nlp-foundations-to-advanced

# Dry run to preview output paths without API calls

python3 scripts/translate_lessons.py --lang fr --only phases/01-foundations/01-hello-world --dry-run

```

The [`.github/workflows/translate.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/translate.yml) CI workflow orchestrates these commands per language, pushing results to the `i18n/<lang>/` directory tree where the site generator consumes them for localized builds.

## Summary

- **Protect-Translate-Restore**: The pipeline masks technical markup with sentinels before translation, then restores exact substrings to preserve code executability.
- **Registry-Driven**: [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json) maps BCP-47 codes to NLLB-200 tokens, enabling support for 200+ languages through a single configuration file.
- **Provider Agnostic**: Supports NLLB-200 (free), Anthropic, OpenAI, and DeepL via uniform interface functions and environment variables.
- **Incremental Caching**: SHA-256 hashing prevents re-translating unchanged lessons, storing state in JSON cache files per language or phase.
- **Selective Processing**: The `targets()` generator excludes the README from machine translation, routing it to a separate hand-translation workflow.

## Frequently Asked Questions

### What is the protect-translate-restore pattern and why is it necessary?

The protect-translate-restore pattern ensures that Large Language Models or machine translation systems do not alter executable code, mathematical notation, or structural URLs. By replacing these spans with invisible sentinels before translation and substituting them back afterward, the system guarantees that a Python function signature or LaTeX equation remains identical across all 40+ language versions of a lesson.

### Which translation providers are supported by the system?

The [`translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/translate_lessons.py) script supports four production providers and one test provider: **NLLB-200** (default, free, no API key required), **Anthropic** (Claude models), **OpenAI** (GPT models), and **DeepL** (specialized translation API). An **"echo"** provider returns input unchanged for debugging. Selection occurs via the `--provider` CLI flag or `TRANSLATE_PROVIDER` environment variable.

### How does the caching mechanism determine which files to skip?

Before processing, the script calculates a SHA-256 hash of the source markdown file. This `source_hash` is compared against entries in `i18n/<lang>/.translate-cache.json`. If the hash matches and the output file exists, the lesson is skipped entirely. This hash-based approach detects any content changes, including minor typo fixes, while ignoring file timestamps.

### Why is the README excluded from the automatic lesson translation?

The repository treats the README as site-wide navigation infrastructure rather than pedagogical content. The `targets()` generator deliberately omits the README from the translation queue because it contains high-level curriculum descriptions and navigation links that require cultural adaptation beyond literal translation. Instead, [`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py) manages hand-authored README variants for each language.