CLI Skills for Placement Testing and Phase Quizzes in AI Engineering From Scratch

The rohitg00/ai-engineering-from-scratch repository provides four command-line utilities—install_skills.py, lesson_run.py, audit_lessons.py, and check_readme_counts.py—that let you install curriculum artefacts, execute placement tests, validate quiz integrity, and synchronize the lesson catalogue.

The AI Engineering From Scratch curriculum ships with a dedicated CLI toolchain designed specifically for placement testing and phase quizzes. These command-line utilities, located in the scripts/ directory, provide the only authorized method for accessing and executing the skill artefacts stored under phases/**/outputs/. By leveraging these tools, educators and learners can programmatically install test prompts, run individual lessons with their associated quizzes, and audit the curriculum for structural integrity before any assessment begins.

Installing and Managing Skill Artefacts with install_skills.py

The scripts/install_skills.py script serves as the primary CLI tool for copying curriculum artefacts—skills, prompts, and agents—from the repository into a target directory. This is the essential first step for preparing placement test environments, as it extracts the specific markdown files used as quiz inputs.

The script accepts several critical flags for filtering artefacts:

• --type {skill,prompt,agent,all} – Selects the artefact kind; placement tests specifically require skill artefacts
• --phase N – Limits installation to a single phase (e.g., 3 for Phase 03 quizzes)
• --tag TAG – Filters by metadata tags; quizzes are typically tagged quiz
• --layout {flat,by-phase,skills} – Controls the directory structure of the output
• --dry-run – Previews actions without writing files
• --force – Overwrites existing files without prompting

When executed, the script generates a manifest.json file that records every installed artefact and its source path. This manifest is crucial for CI pipelines that execute automated placement tests, as it provides a verifiable record of which quiz questions were deployed.

# Install only Phase 03 quiz artefacts into a sandbox directory

python3 scripts/install_skills.py ./placement_test_dir \
    --type skill \
    --phase 3 \
    --tag quiz \
    --layout by-phase

Running Lessons and Quizzes with lesson_run.py

The scripts/lesson_run.py utility executes individual lessons from the command line, including their demo code and associated quizzes. This script replicates the exact interface used by placement-test runners to present questions and capture learner responses.

To execute a lesson, pass the phase slug and lesson slug in the format <phase-slug>/<lesson-slug>:

# Run Phase 03, Lesson 05 (includes both demo code and quiz)

python3 scripts/lesson_run.py 03-advanced-ml/05-regularization

The script supports flags to control execution flow:

• --no-tests – Skips the unit-test suite and demo code, displaying only the quiz prompts
• --quiet – Suppresses non-essential output for cleaner CI integration

When running a lesson containing a quiz, the script prints quiz prompts to stdout. This output stream is what placement-test harnesses capture to present questions and record answers, making lesson_run.py the core engine for interactive assessment.

Validating Curriculum Integrity with audit_lessons.py

Before launching any placement testing session, run scripts/audit_lessons.py to verify that every lesson complies with the curriculum contract. This script checks for the presence and syntactic validity of quiz.json files across all phases.

Unlike the other utilities, audit_lessons.py requires no command-line flags:

# Verify all quiz definitions are well-formed

python3 scripts/audit_lessons.py

The audit catches malformed quiz artefacts before they reach the CLI test runner, preventing runtime errors during placement assessments. According to the repository's AGENTS.md policy, this validation step is mandatory for maintaining the integrity of the CLI-first curriculum access pattern.

Synchronizing the Quiz Catalogue with check_readme_counts.py

The scripts/check_readme_counts.py utility ensures the repository README accurately reflects the current number of lessons and available quizzes. This script supports the CI pipeline that publishes the quiz catalogue used by placement test systems.

• --fix – Automatically corrects count mismatches in the README

While this tool does not directly execute tests, it maintains the public-facing documentation that lists available placement tests and phase quizzes, ensuring learners always see the correct curriculum scope.

Step-by-Step Workflow for Placement Testing

The four CLI tools combine into a complete workflow for managing curriculum assessments:

1. Discover the skill-files under phases/**/outputs/skill-*.md that implement placement-test questions
2. Install the required artefacts using install_skills.py with appropriate --type and --phase filters
3. Run the specific lesson or quiz with lesson_run.py to display prompts and capture answers
4. Audit the lesson definitions with audit_lessons.py to catch malformed quiz.json before test execution

This workflow ensures that all CLI skills for placement testing and phase quizzes are validated, installed, and executed through the authorized command-line interface defined in AGENTS.md.

Code Examples

Here are the essential commands for working with the placement testing CLI:

# 1. Install skill artefacts for Phase 03 quizzes only

python3 scripts/install_skills.py ./placement_test_dir \
    --type skill \
    --phase 3 \
    --tag quiz \
    --layout by-phase

# 2. Run a specific lesson and display its quiz

python3 scripts/lesson_run.py 03-advanced-ml/05-regularization

# 3. Run lesson in quiz-only mode (skip demo code)

python3 scripts/lesson_run.py 03-advanced-ml/05-regularization --no-tests

# 4. Validate all quiz files before testing

python3 scripts/audit_lessons.py

# 5. Fix README lesson counts for the quiz catalogue

python3 scripts/check_readme_counts.py --fix

Summary

• scripts/install_skills.py copies skill artefacts from phases/**/outputs/ to a target directory using filters like --type skill, --phase, and --tag quiz, generating a manifest.json for CI tracking.
• scripts/lesson_run.py executes lessons and quizzes via the <phase-slug>/<lesson-slug> syntax, supporting --no-tests for quiz-only output.
• scripts/audit_lessons.py validates the integrity of quiz.json files across the curriculum without requiring additional flags.
• scripts/check_readme_counts.py synchronizes the public quiz catalogue with the --fix option for automated documentation updates.
• The complete CLI workflow follows four steps: discover artefacts, install with filters, run lessons with lesson_run.py, and audit with audit_lessons.py.
• All tools operate according to the AGENTS.md policy, which mandates CLI utilities as the only authorized access method for curriculum artefacts.

Frequently Asked Questions

How do I extract only the quiz questions for a specific phase?

Run python3 scripts/install_skills.py <target_dir> --type skill --phase <N> --tag quiz --layout by-phase. This installs only the markdown skill artefacts tagged as quizzes from the specified phase into your target directory, organized by phase.

Can I run a quiz without executing the lesson's demo code?

Yes. Pass the --no-tests flag to scripts/lesson_run.py when executing the lesson. This skips the unit-test suite and demo code, printing only the quiz prompts to stdout for immediate interaction or capture by a test harness.

What ensures the quiz files are valid before I run them?

The scripts/audit_lessons.py utility validates every quiz.json file in the curriculum for syntactic correctness and structural compliance. Run this script before any placement testing session to eliminate malformed artefacts that could cause runtime failures.

Where are the actual quiz questions stored in the repository?

The quiz content lives in markdown files under phases/**/outputs/skill-*.md. These files contain the test prompts and metadata used by the placement testing framework. The install_skills.py script is the only authorized method for extracting these files from their phase-organized locations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}