# How AI Engineering from Scratch Handles Internationalization and Translations: A Complete Technical Guide

> Explore how AI Engineering from Scratch automatically handles internationalization and translations using a pipeline that preserves syntax and supports NLLB-200 and commercial LLMs. Learn more today.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-08-30

---

**The AI Engineering from Scratch repository treats English as the single source of truth and generates all other language versions automatically through a pipeline that protects technical syntax and supports both free NLLB-200 and commercial LLM providers.**

The `rohitg00/ai-engineering-from-scratch` repository delivers AI engineering curriculum to a global audience through a sophisticated internationalization system. Unlike manual translation workflows, this repository implements a fully automated pipeline that preserves code integrity while scaling to dozens of languages. The architecture separates canonical English source files from generated translations, ensuring technical accuracy across all supported locales.

## Core Architecture of the i18n Pipeline

The internationalization system consists of three tightly-coupled components: a centralized language registry, an automated lesson translation engine, and a CI/CD matrix that handles distribution and caching.

### Language Registry ([`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json))

All supported locales are defined in [[`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json) at the repository root. This JSON file maps each language to its ISO code, native name, and the corresponding NLLB-200 model identifier required for the free default translator.

Each entry specifies whether the language should be built in CI (`ci: true`), enabling the GitHub Actions matrix to automatically include it in the translation workflow. Adding a new language requires only appending an entry to this file; no additional configuration changes are necessary.

### Lesson Translation Engine ([`translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/translate_lessons.py))

The [[`scripts/translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/translate_lessons.py)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/translate_lessons.py) script serves as the core translation worker. It walks every lesson markdown file located in `phases/**/docs/en.md`, processes the content to protect technical spans, and sends only the prose to the configured translation provider.

Before translation, the script replaces inline code, LaTeX math, fenced code blocks, markdown links, images, and bold technical terms with invisible placeholder tokens (`⁣PROTECT<n>⁣`). This guarantees that the translation never corrupts syntax or identifiers. After translation completes, the script restores the protected spans unchanged and writes the results to the dedicated `translations` branch under `i18n/<lang>/`.

### README Internationalization ([`build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build_readme_i18n.py))

While lesson content is fully automated, the repository landing page receives special treatment through [[`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py). This generator copies the English [[`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), substitutes only the translatable prose blocks with hand-crafted translations stored in [[`scripts/readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py), and rewrites relative links to resolve correctly from the `i18n/<lang>/` directory.

### CI/CD Matrix ([`.github/workflows/translate.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/translate.yml))

The [[`.github/workflows/translate.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/translate.yml)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/translate.yml) file defines a GitHub Actions matrix that runs one job per **(language, phase)** pair. Because a full language run would exceed the 6-hour job limit, the workflow slices work by phase (e.g., `01-math-foundations`).

Each job restores the language’s cache from the `translations` branch, translates only changed lessons using hash-cached entries stored in `i18n/<lang>/.cache/<phase>.json`, and pushes the slice back to the branch with race-safe work-tree handling. This sharding strategy allows independent retries and prevents job timeouts.

## How the Translation Pipeline Protects Technical Content

The system implements **zero-cost default translation** using the open-source **NLLB-200** model (`facebook/nllb-200-distilled-600M`) that runs inside the CI runner, requiring no API keys or per-token charges. Optional providers including Anthropic, OpenAI, and DeepL can be selected via the `--provider` flag when higher quality is required.

The protection mechanism ensures that technical content remains identical across all locales. Before sending text to any LLM, the script identifies and masks:

- Inline code spans and fenced code blocks
- LaTeX math expressions
- Markdown links and image references
- Bold technical terms and identifiers

These spans are replaced with indexed placeholders that are restored post-translation, guaranteeing that code syntax, mathematical notation, and file paths remain intact regardless of the target language.

## Running Translations Locally and in CI

Translate a single language for all phases using the default NLLB model:

```bash
python3 scripts/translate_lessons.py --lang zh

```

*This runs the NLLB-200 model for Simplified Chinese (`zh`), walks every lesson, writes results to `i18n/zh/...`, and updates the per-language cache.*

Translate only a specific phase for rapid CI testing:

```bash
python3 scripts/translate_lessons.py --lang es --phase 05-nlp-foundations-to-advanced

```

*This limits the job to the "05-nlp-foundations-to-advanced" phase, reducing runtime to a few minutes.*

Switch to a commercial provider for higher quality:

```bash
export LLM_API_KEY=sk-...
python3 scripts/translate_lessons.py --lang fr --provider anthropic

```

*The same placeholder protection guarantees that code blocks remain untouched regardless of the provider.*

Add a new language to the pipeline by editing [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json):

```json
{
  "code": "sw",
  "name": "Swahili",
  "native": "Kiswahili",
  "nllb": "swh_Latn",
  "ci": true
}

```

*Committing this change causes the CI matrix to automatically pick up the new language on the next push.*

Rebuild translated README files from hand-authored translations:

```bash
python3 scripts/build_readme_i18n.py

```

Verify that README translations are up-to-date in CI:

```bash
python3 scripts/build_readme_i18n.py --check

```

*This exits with a non-zero status if any language’s README is stale, causing the CI job to fail.*

## Frontend Integration and Language Switching

The repository separates translated lesson files from the main branch to keep the canonical source repository clean. All translated lesson files live exclusively on the `translations` branch, while the hand-curated README translations reside in `i18n/<lang>/` directories on `main`.

The build script [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) reads [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json) entries marked with `ci: true` and emits [`site/langs.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/langs.js). When a user selects a language via URL parameter (`?lang=zh`), the frontend fetches the corresponding markdown from the `translations` branch using `raw.githubusercontent.com`; if the translation is missing, it falls back to the original English file from the main branch.

## Summary

- **Canonical English source**: All lessons are written exclusively in `phases/**/docs/en.md` on the main branch.
- **Automated protection**: Technical spans are masked with `⁣PROTECT<n>⁣` placeholders before translation to prevent corruption.
- **Zero-cost default**: The NLLB-200 model provides free translation within CI runners, with optional API-based providers available.
- **Per-phase sharding**: The CI matrix processes (language, phase) pairs independently to avoid job timeouts.
- **Separate translations branch**: Generated lesson content lives on the `translations` branch, while hand-authored README translations commit to `main`.
- **Registry-driven scaling**: Adding a language requires only a JSON entry in [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json).

## Frequently Asked Questions

### What translation models does the repository use by default?

The pipeline defaults to the open-source **NLLB-200** model (`facebook/nllb-200-distilled-600M`) which runs locally inside the CI runner. This approach requires no API keys and incurs no per-token charges. Users can optionally specify `--provider anthropic`, `--provider openai`, or `--provider deepl` to use commercial LLMs with higher translation quality.

### How does the pipeline prevent code blocks from being translated?

The [`translate_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/translate_lessons.py) script identifies technical spans—including inline code, fenced code blocks, LaTeX math, markdown links, and bold identifiers—and replaces them with indexed placeholder tokens (`⁣PROTECT<n>⁣`) before sending text to the translation service. After translation completes, the script restores the original technical content exactly, ensuring that syntax, file paths, and mathematical expressions remain unchanged across all language versions.

### Why are translations stored on a separate branch instead of main?

The repository maintains a strict separation where the `main` branch contains only the canonical English source files ([`en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/en.md)) and hand-crafted README translations. All machine-generated lesson translations are pushed to the `translations` branch. This design keeps the main repository history clean, enables easy review of translation changes without cluttering source code PRs, and allows the frontend to fetch specific language versions at runtime via `raw.githubusercontent.com`.

### Can I contribute manual translations instead of using the automated pipeline?

For lesson content, the architecture assumes automated translation from the English source to ensure consistency and scalability. However, the README internationalization system explicitly supports manual contributions through [`scripts/readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py), which stores hand-crafted translations keyed by exact English blocks. To add or improve a language, contributors can submit PRs against [`languages.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/languages.json) and [`readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/readme_translations.py); the CI pipeline will then incorporate these into the generated site.