# How backfill_certification_references.py Ensures Complete Lesson References in AI Certifications > Discover how backfill_certification_references.py ensures complete lesson references in AI certifications. This script uses a deterministic, domain-aware process for robust linking. - Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch) - Tags: how-to-guide - Published: 2026-09-04 --- **backfill_certification_references.py** is a deterministic, idempotent Python script that guarantees every certification assessment question links to at least one internal lesson by orchestrating a three-phase domain-aware matching process. In the `rohitg00/ai-engineering-from-scratch` repository, maintaining referential integrity between certification assessments and their source lessons is critical for learner navigation. The [`backfill_certification_references.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/backfill_certification_references.py) script automates this linkage by systematically analyzing track definitions and assessment JSON files to ensure **complete lesson references** across all certification questions. ## Phase 1: Domain-Based Lesson Candidate Collection The process begins in [`scripts/backfill_certification_references.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/backfill_certification_references.py) with the `lesson_candidates()` function (lines 29‑44). This function constructs a mapping from each domain to a list of eligible lesson paths by scanning the `lessons` array defined in the track configuration. ### Prioritization Logic for Lesson Selection Not all lessons are treated equally. Within the candidate selection logic (lines 38‑42), the script applies a deterministic sorting strategy: - **Capstone lessons** are prioritized when multiple candidates exist for a domain - **Strategy lessons** are deliberately sorted to appear last in the candidate list This ordering ensures that fundamental, content-heavy lessons are preferred over meta-cognitive "strategy" lessons when establishing primary references. ## Phase 2: Injecting Missing Internal References Once candidates are mapped, the script processes every assessment file discovered via `assessment_paths()` (lines 46‑52). For each JSON file, it iterates over the `questions` array and checks for existing references using `internal_references()` (lines 55‑60). ### Skipping Already-Referenced Questions If a question already contains an internal reference, the script immediately skips it. This preserves manually-curated linkages while focusing automation only on gaps. ### Domain-Matched Reference Injection For questions lacking references, the script performs a lookup in the domain-candidate map. When candidates exist for the question's specific domain (retrieved via `question.get("domain")`), the first lesson path from the sorted list is appended to the question's `references` field (lines 72‑77). This guarantees that every question points to at least one relevant lesson belonging to its domain. ## Phase 3: Guaranteeing Complete Coverage for Route Lessons The third phase addresses a subtle edge case: ensuring every lesson that appears in the certification track is referenced somewhere, even if no questions explicitly requested that lesson's domain. ### Coverage Gap Analysis After the initial pass, the script builds a set called `covered` containing all lesson paths already successfully linked (lines 78‑83). It then iterates over `route_lessons`—the complete list of lessons defined in the track. For any lesson not present in the `covered` set, the script searches for a question whose domain matches one of the lesson's declared domains (lines 93‑103). When such a question is identified, the missing lesson path is appended to that question's references (lines 105‑107). This secondary sweep ensures **complete lesson coverage**, preventing "orphaned" lessons that exist in the track but remain unreferenced by any assessment item. ## Deterministic Write-Back and Reporting The script concludes with a safe, idempotent write phase. It diffs the original JSON against the mutated version (lines 110‑117). Changes are only written if the `--check` flag is not present, enabling dry-run verification. The final reporting block outputs the count of remediated references and affected files, providing clear auditability of modifications. ## Usage Examples Run the script in check mode to identify missing references without modifying files: ```bash python3 scripts/backfill_certification_references.py --check # Output: need X remediation reference(s) across Y assessment file(s) ``` Execute the actual backfill to inject missing references: ```bash python3 scripts/backfill_certification_references.py # Output: added X remediation reference(s) across Y assessment file(s) ``` ## Summary - **Domain-aware mapping**: The `lesson_candidates()` function creates a prioritized mapping from domains to lesson paths, ensuring relevant content is selected first. - **Gap-filling logic**: Questions without internal references receive the highest-priority lesson for their domain, guaranteeing every assessment item links to instructional content. - **Orphan prevention**: The secondary coverage sweep ensures every lesson defined in the track appears in at least one question's references. - **Idempotent execution**: JSON diffing and conditional write-back allow safe repeated runs without corrupting existing data. - **CI/CD integration**: The `--check` flag enables non-destructive validation in automated pipelines. ## Frequently Asked Questions ### What makes backfill_certification_references.py idempotent? The script never removes or modifies existing references; it only appends missing ones. By comparing the original and mutated JSON structures before writing (lines 110‑117), it ensures that running the script multiple times produces the same result as running it once, provided the input data remains unchanged. ### How does the script handle questions that already have references? Questions containing existing internal references are immediately skipped by the `internal_references()` check (lines 55‑60). This preservation logic prevents overwriting manually curated linkages while the script focuses exclusively on backfilling gaps. ### What happens if a lesson belongs to multiple domains? During the coverage gap analysis (lines 93‑103), the script examines all domains associated with an uncovered lesson. It searches for any question matching one of those domains and appends the lesson reference to the first suitable candidate found, ensuring multi-domain lessons are still properly linked. ### Can I run this script in CI/CD without modifying files? Yes. By passing the `--check` flag, the script performs a dry-run that identifies missing references and reports statistics without writing changes to disk. This allows integration into continuous integration pipelines to enforce reference completeness as a quality gate.