AI Engineering i18n Translation Pipeline: Automated Machine-Translation on the translations Branch

The ai-engineering-from-scratch repository maintains a fully automated internationalization pipeline that machine-translates English lesson content into multiple languages using Python scripts and GitHub Actions, storing all generated files on the dedicated translations branch.

The rohitg00/ai-engineering-from-scratch project implements a deterministic i18n translation pipeline that eliminates manual localization overhead. Rather than polluting the main branch with generated content, the system isolates all machine-translated curriculum files on a separate translations branch, enabling the static site generator to serve localized pages while preserving a clean source history.

Architecture of the i18n Translation Pipeline

The pipeline follows a reproducible four-stage flow: source extraction, machine-translation, file generation, and automated deployment. Canonical English content lives in phases/*/docs/en.md on the main branch. When triggered, the system extracts markdown text—intentionally ignoring code blocks to prevent translation of syntax—processes it through a configurable translation service, and writes parallel directory structures under i18n/<lang>/ and phases/.../docs/<lang>.md on the translations branch.

The Translation Driver (scripts/translate_lessons.py)

The core orchestration logic resides in scripts/translate_lessons.py. This script walks the lesson tree using pathlib, loads source files with the frontmatter library to preserve YAML metadata, and delegates translation to a generic wrapper function.

Key implementation details include:

• LANGUAGES list: A hardcoded array in scripts/translate_lessons.py defines supported locales: ["es", "fr", "de", "zh", "ja", "ko", "hi", "ar", "ru", "tr"].
• translate_text() wrapper: A helper function abstracts the machine-translation API, allowing seamless swapping between OpenAI ChatCompletion, Hugging Face MarianMT, or Google Translate.
• Frontmatter preservation: The script copies the original YAML frontmatter to translated files, ensuring the curriculum UI renders them without template modifications.

# scripts/translate_lessons.py – main loop (simplified)

import os, pathlib, frontmatter
from translate import translate_text   # wrapper around the chosen MT service

LANGUAGES = ["es", "fr", "de", "zh", "ja", "ko", "hi", "ar", "ru", "tr"]

def translate_lesson(md_path: pathlib.Path):
    source = frontmatter.load(md_path)
    body = source.content
    for lang in LANGUAGES:
        translated = translate_text(body, target_lang=lang)
        target_dir = md_path.parent.parent / "i18n" / lang / md_path.parent.name
        target_dir.mkdir(parents=True, exist_ok=True)
        target_path = target_dir / "README.md"
        with target_path.open("w", encoding="utf‑8") as f:
            f.write("---\n")
            f.write(f"lang: {lang}\n")
            f.write("---\n\n")
            f.write(translated)

if __name__ == "__main__":
    for md in pathlib.Path("phases").rglob("docs/en.md"):
        translate_lesson(md)

Machine-Translation Service Integration

By default, the pipeline utilizes the OpenAI ChatCompletion endpoint with a system prompt instructing the model to translate to the specified target language. The service layer is fully configurable via environment variables, decoupling the translation logic from specific vendors.

Configuration requirements:

• Set OPENAI_API_KEY in repository secrets for the default backend.
• Override the translate_text() implementation in scripts/translate_lessons.py to integrate alternative providers such as Hugging Face MarianMT or Google Cloud Translation.

The script specifically excludes fenced code blocks from the translation payload, ensuring that Python syntax, shell commands, and configuration examples remain intact across all language versions.

CI/CD Automation with GitHub Actions

The .github/workflows/translate.yml workflow automates pipeline execution on every push to main. It ensures the translations branch remains synchronized with the latest English curriculum without manual intervention.

# .github/workflows/translate.yml – CI trigger

name: Translate Lessons
on:
  push:
    branches: [main]
jobs:
  translate:
    runs-on: ubuntu‑latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Set up Python
        uses: actions/setup-python@v4
        with: {python-version: "3.11"}
      - name: Install deps
        run: pip install -r .github/translate-requirements.txt
      - name: Run translation script
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python -m scripts.translate_lessons
      - name: Push translations
        uses: ad-m/github-push-action@v0.6.0
        with:
          branch: translations
          force: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

The workflow installs Python 3.11, resolves dependencies from .github/translate-requirements.txt (which includes the openai client), executes the driver, and force-pushes the generated i18n files to the translations branch.

Static Site Generation and Language Routing

When the website builds, site/build.js reads the language-specific README.md files from the translations branch. The generator constructs navigation menus by parsing the markdown links within i18n/<lang>/README.md files, which follow the same relative path patterns as the English source. This design allows the static site to serve localized lesson pages at predictable URLs without requiring duplicate template logic.

Extending the Pipeline to New Languages

Adding support for additional locales requires minimal changes:

1. Create a new directory under i18n/ (e.g., i18n/xx/).
2. Append the ISO language code to the LANGUAGES list in scripts/translate_lessons.py.
3. Commit and push to main; the GitHub Actions workflow will generate the full translation set on the next run.

This modular approach ensures the i18n translation pipeline scales horizontally as the curriculum expands into new regions.

Summary

• The translations branch isolates all machine-generated content from the main branch source files.
• scripts/translate_lessons.py extracts markdown from phases/*/docs/en.md while preserving code blocks and frontmatter.
• Translation occurs via a configurable translate_text() wrapper, defaulting to OpenAI ChatCompletion.
• .github/workflows/translate.yml triggers on every push to main, automating the full generation and deployment cycle.
• The static site generator consumes i18n/<lang>/README.md files to build language-specific navigation menus.

Frequently Asked Questions

What triggers the i18n translation pipeline?

A push event to the main branch triggers the .github/workflows/translate.yml GitHub Actions workflow. This ensures the translations branch always reflects the current state of the English curriculum without requiring manual execution of the Python scripts.

How does the pipeline handle code blocks during machine-translation?

The translate_lessons.py script parses markdown to extract only prose content, deliberately excluding fenced code blocks from the translation payload. This prevents corruption of Python syntax, YAML configurations, and shell commands, inserting them unchanged into the translated output files.

Can I use a different machine-translation service instead of OpenAI?

Yes. The translate_text() function in scripts/translate_lessons.py serves as a pluggable abstraction layer. You can replace the default OpenAI ChatCompletion implementation with calls to Hugging Face MarianMT, Google Cloud Translation, or any other MT API by modifying this wrapper function and updating the environment variables in the GitHub Actions workflow.

Why are translations stored on a separate branch instead of main?

The translations branch acts as a generated artifact store, similar to a gh-pages branch. This separation of concerns keeps the main branch history clean and reviewable, prevents merge conflicts in generated files, and allows the static site builder to pull localized content via a simple branch reference without cluttering the source repository with machine-generated markdown.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}