# How the readme-translations.py Script Manages the i18n Landing Page Pipeline

> Discover how readme-translations.py streamlines the i18n landing page pipeline by managing translation tables and identifying translatable text for localization.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-08-29

---

**The [`readme-translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/readme-translations.py) script stores hand-crafted translation tables that map exact English text blocks to localized versions, while [`build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build_readme_i18n.py) scans the canonical [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), identifies translatable spans using regex patterns, and generates language-specific landing pages in `i18n/<lang>/README.md`.**

Managing multilingual documentation for open-source repositories requires a pipeline that preserves Markdown formatting while enabling community-driven translations. The `rohitg00/ai-engineering-from-scratch` repository implements a loss-less internationalization system using two coordinated Python utilities that separate translation data from rendering logic. The [`readme-translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/readme-translations.py) file serves as the authoritative source for all localized content, enabling incremental language additions without breaking the build process.

## The Two-Script Architecture

The localization pipeline splits responsibilities between two dedicated files in the `scripts/` directory:

- **[`scripts/readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py)** – Maintains the translation dictionary and defines translatable keys as string constants (e.g., `HERO1`, `HERO2`). This file contains the `TRANSLATIONS` nested dictionary where outer keys are ISO language codes (`"es"`, `"fr"`, `"de"`) and inner maps associate English blocks with their target translations.
- **[`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py)** – Implements the extraction and rendering engine. It parses the canonical [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), identifies translatable regions using the `spans()` function, performs dictionary lookups against the translation tables, and writes localized versions to `i18n/<lang>/README.md`.

## Translation Data Model in readme_translations.py

### Block Keys as Translation Constants

Rather than using arbitrary identifiers, the script defines translation keys as **exact English strings** stored in module-level constants. For example:

```python
HERO1 = "**84% of students already use AI tools…**"
HERO2 = "503 lessons. 20 phases. ~320 hours..."

```

These constants act as lookup keys in the `TRANSLATIONS` dictionary. Because each key matches the exact English text (with whitespace normalization via `block_key()`), any missing translation automatically falls back to the original English string, ensuring the build never breaks.

### The TRANSLATIONS Dictionary Structure

The `TRANSLATIONS` constant is a nested dictionary mapping language codes to translation maps:

```python
TRANSLATIONS = {
    "es": {
        HERO1: "**El 84% de los estudiantes ya usan herramientas de IA...",
        HERO2: "503 lecciones. 20 fases. ~320 horas...",
    },
    "fr": {
        HERO1: "**84% des étudiants utilisent déjà des outils d'IA...",
        HERO2: "503 leçons. 20 phases. ~320 heures...",
    }
}

```

### Canonical Version Banner

Each translated README includes an optional `README_NOTE` constant—an HTML snippet inserted at the top of generated files indicating that the English version remains the authoritative source.

## How build_readme_i18n.py Identifies Translatable Content

The extraction engine uses the `spans()` function to segment the canonical README into discrete, replaceable units while preserving code blocks and structural elements.

### Span Detection Logic

The parser walks the README line-by-line in [`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py) and categorizes content:

- **Code fences** are identified via the `FENCE` regex and excluded from translation to preserve syntax integrity.
- **Headings** match the `HEADING` regex and generate single-line spans categorized as `"heading"`.
- **Prose blocks** are collected using the `is_prose()` helper to aggregate consecutive lines of narrative text into maximal runs marked as `"prose"` spans.

### Key Normalization

The `block_key()` helper function normalizes extracted text by cleaning whitespace and line breaks to match the constants defined in [`readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/readme_translations.py), enabling reliable dictionary lookups during the rendering phase.

## Rendering and Link Localization

### Bottom-Up Text Replacement

The `render()` function in [`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py) performs substitutions in reverse index order to maintain valid string positions during modification:

```python
for sp in sorted(spans(text), key=lambda s: s["start"], reverse=True):
    t = table.get(sp["key"])
    if not t: 
        continue
    replacement = [sp["prefix"] + ln for ln in t.split("\n")]
    lines[sp["start"]:sp["end"]] = replacement

```

If a translation is missing, the original English block remains untouched, ensuring graceful degradation and allowing incremental translation workflows.

### Relative Link Adjustment

After text substitution, the `localize_links()` function rewrites repository-root-relative Markdown and HTML links to point two levels up (`../../`). This adjustment ensures that images, tables, and the language-selection bar resolve correctly from the generated file's location at `i18n/<lang>/README.md`.

## Running the i18n Pipeline

The [`scripts/build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_readme_i18n.py) script supports three execution modes via command-line arguments:

```bash

# Generate all localized READMEs

python3 scripts/build_readme_i18n.py

# List all translatable block keys (useful for new translations)

python3 scripts/build_readme_i18n.py --dump

# Verify generated files match current sources (CI check)

python3 scripts/build_readme_i18n.py --check

```

In **generate mode**, the script iterates through each language entry in `TRANSLATIONS`, prepends the optional `README_NOTE`, and writes the final Markdown to `i18n/<lang>/README.md`.

In **dump mode**, the script prints every `block_key` discovered by `spans()` across the canonical README, providing contributors with the exact strings requiring translation.

In **check mode**, the script compares existing generated files against freshly computed outputs, reporting mismatches as stale translations that need regeneration before committing.

## Adding a New Language to the Pipeline

Extending support to a new language requires three steps:

1. **Update the translation table** in [`scripts/readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/readme_translations.py) by adding a new language code entry (e.g., `"no"` for Norwegian) with translations for all defined keys such as `HERO1`, `HERO2`, and navigation headings.

2. **Add the language code** to the README's language selection bar in the canonical English file.

3. **Run the generator** to create `i18n/<lang>/README.md` automatically:

```bash
python3 scripts/build_readme_i18n.py

```

Because the system uses hand-authored translation tables, the landing page retains exact formatting control, and partial translations safely default to English for any missing keys.

## Summary

- The **readme-translations.py** script stores hand-crafted translation tables using exact English text as lookup keys, enabling automatic fallback to English when translations are missing.
- **build_readme_i18n.py** scans the canonical README using the `spans()` function to identify headings and prose blocks while ignoring code fences marked by the `FENCE` regex.
- The `render()` function performs bottom-up text substitution to maintain index integrity, while `localize_links()` adjusts relative paths to `../../` for files in `i18n/<lang>/` subdirectories.
- The pipeline supports three CLI modes: generation, key dumping for contributors, and staleness checking for continuous integration validation.
- New languages are added incrementally by extending the `TRANSLATIONS` dictionary without modifying the rendering logic, ensuring type-safe, community-driven localization.

## Frequently Asked Questions

### What happens if a translation is missing for a specific text block?

The renderer gracefully falls back to the original English text. Because translation keys in [`readme_translations.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/readme_translations.py) are the exact English strings, a missing entry in the `TRANSLATIONS` dictionary causes the `render()` function to skip replacement and preserve the source text from the canonical README.

### How does the script handle code blocks and technical syntax?

The `spans()` function in [`build_readme_i18n.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build_readme_i18n.py) uses the `FENCE` regex pattern to identify and skip code blocks delimited by triple backticks. This ensures that shell commands, Python snippets, and configuration examples remain untranslated and syntactically intact during the localization process.

### Can I test translations locally before submitting a pull request?

Yes. Run `python3 scripts/build_readme_i18n.py --dump` to extract all translatable keys and verify your translation table contains entries for each required constant. Then execute the standard generation command to produce the localized README locally in `i18n/<your-lang>/README.md`, allowing you to preview formatting and link resolution without affecting the canonical English source.

### Why does the script rewrite relative links to `../../`?

Generated READMEs live in `i18n/<lang>/` subdirectories, two levels deeper than the repository root. The `localize_links()` function prepends `../../` to repository-root-relative URLs so that images, architecture diagrams, and license links resolve correctly when visitors view the localized landing pages.