How AI Engineering from Scratch Handles Internationalization and Translations: A Complete Technical Guide

The AI Engineering from Scratch repository treats English as the single source of truth and generates all other language versions automatically through a pipeline that protects technical syntax and supports both free NLLB-200 and commercial LLM providers.

The rohitg00/ai-engineering-from-scratch repository delivers AI engineering curriculum to a global audience through a sophisticated internationalization system. Unlike manual translation workflows, this repository implements a fully automated pipeline that preserves code integrity while scaling to dozens of languages. The architecture separates canonical English source files from generated translations, ensuring technical accuracy across all supported locales.

Core Architecture of the i18n Pipeline

The internationalization system consists of three tightly-coupled components: a centralized language registry, an automated lesson translation engine, and a CI/CD matrix that handles distribution and caching.

Language Registry (languages.json)

All supported locales are defined in [languages.json](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json) at the repository root. This JSON file maps each language to its ISO code, native name, and the corresponding NLLB-200 model identifier required for the free default translator.

Each entry specifies whether the language should be built in CI (ci: true), enabling the GitHub Actions matrix to automatically include it in the translation workflow. Adding a new language requires only appending an entry to this file; no additional configuration changes are necessary.

Lesson Translation Engine (translate_lessons.py)

The [scripts/translate_lessons.py](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/translate_lessons.py) script serves as the core translation worker. It walks every lesson markdown file located in phases/**/docs/en.md, processes the content to protect technical spans, and sends only the prose to the configured translation provider.

Before translation, the script replaces inline code, LaTeX math, fenced code blocks, markdown links, images, and bold technical terms with invisible placeholder tokens (⁣PROTECT<n>⁣). This guarantees that the translation never corrupts syntax or identifiers. After translation completes, the script restores the protected spans unchanged and writes the results to the dedicated translations branch under i18n/<lang>/.

README Internationalization (build_readme_i18n.py)

While lesson content is fully automated, the repository landing page receives special treatment through [scripts/build_readme_i18n.py](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py). This generator copies the English [README.md](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), substitutes only the translatable prose blocks with hand-crafted translations stored in [scripts/readme_translations.py](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py), and rewrites relative links to resolve correctly from the i18n/<lang>/ directory.

CI/CD Matrix (.github/workflows/translate.yml)

The [.github/workflows/translate.yml](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/translate.yml) file defines a GitHub Actions matrix that runs one job per (language, phase) pair. Because a full language run would exceed the 6-hour job limit, the workflow slices work by phase (e.g., 01-math-foundations).

Each job restores the language’s cache from the translations branch, translates only changed lessons using hash-cached entries stored in i18n/<lang>/.cache/<phase>.json, and pushes the slice back to the branch with race-safe work-tree handling. This sharding strategy allows independent retries and prevents job timeouts.

How the Translation Pipeline Protects Technical Content

The system implements zero-cost default translation using the open-source NLLB-200 model (facebook/nllb-200-distilled-600M) that runs inside the CI runner, requiring no API keys or per-token charges. Optional providers including Anthropic, OpenAI, and DeepL can be selected via the --provider flag when higher quality is required.

The protection mechanism ensures that technical content remains identical across all locales. Before sending text to any LLM, the script identifies and masks:

• Inline code spans and fenced code blocks
• LaTeX math expressions
• Markdown links and image references
• Bold technical terms and identifiers

These spans are replaced with indexed placeholders that are restored post-translation, guaranteeing that code syntax, mathematical notation, and file paths remain intact regardless of the target language.

Running Translations Locally and in CI

Translate a single language for all phases using the default NLLB model:

python3 scripts/translate_lessons.py --lang zh

This runs the NLLB-200 model for Simplified Chinese (zh), walks every lesson, writes results to i18n/zh/..., and updates the per-language cache.

Translate only a specific phase for rapid CI testing:

python3 scripts/translate_lessons.py --lang es --phase 05-nlp-foundations-to-advanced

This limits the job to the "05-nlp-foundations-to-advanced" phase, reducing runtime to a few minutes.

Switch to a commercial provider for higher quality:

export LLM_API_KEY=sk-...
python3 scripts/translate_lessons.py --lang fr --provider anthropic

The same placeholder protection guarantees that code blocks remain untouched regardless of the provider.

Add a new language to the pipeline by editing languages.json:

{
  "code": "sw",
  "name": "Swahili",
  "native": "Kiswahili",
  "nllb": "swh_Latn",
  "ci": true
}

Committing this change causes the CI matrix to automatically pick up the new language on the next push.

Rebuild translated README files from hand-authored translations:

python3 scripts/build_readme_i18n.py

Verify that README translations are up-to-date in CI:

python3 scripts/build_readme_i18n.py --check

This exits with a non-zero status if any language’s README is stale, causing the CI job to fail.

Frontend Integration and Language Switching

The repository separates translated lesson files from the main branch to keep the canonical source repository clean. All translated lesson files live exclusively on the translations branch, while the hand-curated README translations reside in i18n/<lang>/ directories on main.

The build script site/build.js reads languages.json entries marked with ci: true and emits site/langs.js. When a user selects a language via URL parameter (?lang=zh), the frontend fetches the corresponding markdown from the translations branch using raw.githubusercontent.com; if the translation is missing, it falls back to the original English file from the main branch.

Summary

• Canonical English source: All lessons are written exclusively in phases/**/docs/en.md on the main branch.
• Automated protection: Technical spans are masked with ⁣PROTECT<n>⁣ placeholders before translation to prevent corruption.
• Zero-cost default: The NLLB-200 model provides free translation within CI runners, with optional API-based providers available.
• Per-phase sharding: The CI matrix processes (language, phase) pairs independently to avoid job timeouts.
• Separate translations branch: Generated lesson content lives on the translations branch, while hand-authored README translations commit to main.
• Registry-driven scaling: Adding a language requires only a JSON entry in languages.json.

Frequently Asked Questions

What translation models does the repository use by default?

The pipeline defaults to the open-source NLLB-200 model (facebook/nllb-200-distilled-600M) which runs locally inside the CI runner. This approach requires no API keys and incurs no per-token charges. Users can optionally specify --provider anthropic, --provider openai, or --provider deepl to use commercial LLMs with higher translation quality.

How does the pipeline prevent code blocks from being translated?

The translate_lessons.py script identifies technical spans—including inline code, fenced code blocks, LaTeX math, markdown links, and bold identifiers—and replaces them with indexed placeholder tokens (⁣PROTECT<n>⁣) before sending text to the translation service. After translation completes, the script restores the original technical content exactly, ensuring that syntax, file paths, and mathematical expressions remain unchanged across all language versions.

Why are translations stored on a separate branch instead of main?

The repository maintains a strict separation where the main branch contains only the canonical English source files (en.md) and hand-crafted README translations. All machine-generated lesson translations are pushed to the translations branch. This design keeps the main repository history clean, enables easy review of translation changes without cluttering source code PRs, and allows the frontend to fetch specific language versions at runtime via raw.githubusercontent.com.

Can I contribute manual translations instead of using the automated pipeline?

For lesson content, the architecture assumes automated translation from the English source to ensure consistency and scalability. However, the README internationalization system explicitly supports manual contributions through scripts/readme_translations.py, which stores hand-crafted translations keyed by exact English blocks. To add or improve a language, contributors can submit PRs against languages.json and readme_translations.py; the CI pipeline will then incorporate these into the generated site.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}