Developing a Mini Framework for AI Applications: A Step-by-Step Engineering Guide

The AI Engineering from Scratch repository teaches you to build a production-ready mini framework for AI applications by implementing 503 hands-on lessons across 20 phases, from linear algebra fundamentals to autonomous agents using the Model Context Protocol.

This comprehensive curriculum in rohitg00/ai-engineering-from-scratch functions as both a living textbook and a reusable codebase. Each lesson contributes directly to a modular AI stack that you can extract into real-world projects, covering everything from raw mathematical primitives to high-level agent orchestration.

Curriculum Architecture: 20 Phases to a Modular AI Stack

The repository structures 503 lessons across 20 sequential phases that progressively assemble your mini framework. Each phase adds a new abstraction layer—starting with foundational math and culminating in autonomous agent engineering.

The Lesson Structure

Every lesson follows a strict directory convention under phases/<NN>-<phase-name>/<NN>-<lesson-name>/:

• code/ – Runnable implementations in Python, TypeScript, Rust, or Julia
• docs/en.md – Narrative documentation with front-matter describing learning objectives
• outputs/ – The artifact produced by the "USE IT" step (prompts, skills, agents, or MCP servers)

This layout is documented in the repository's README.md and ensures every lesson is self-contained and importable into production codebases.

Build-It versus Use-It Methodology

Each lesson employs a dual-track approach that forces deep understanding of both mechanics and APIs:

1. Build-It – Implement algorithms from raw math without external libraries
2. Use-It – Run the same logic using production libraries like PyTorch, JAX, or the custom MCP framework

As noted in the README.md, this split ensures you understand low-level tensor operations before abstracting them behind high-level interfaces.

Assembling the Mini Framework: From Math to Agents

The curriculum gradually constructs your reusable mini framework through six evolutionary stages:

• Math primitives – Vectors, matrices, and tensor operations
• ML building blocks – Linear regression, gradient descent, and optimizers
• Neural-network core – Perceptron implementation through back-propagation
• High-level components – Tokenizers, attention mechanisms, transformers, and diffusion models
• Production glue – Model Context Protocol (MCP) servers, tool schemas, async handling, and security layers
• Agent workbench – Memory systems, planning modules, orchestration logic, and observability hooks

Phase 1-3: Foundations and Core Implementation

Early phases establish the framework's computational backbone. In phases/03-deep-learning-core/10-mini-framework/code/main.py, you construct the core autograd engine and neural network layers from scratch using only NumPy arrays and Python classes.

This lesson implements the fundamental Tensor class with automatic differentiation, demonstrating how modern frameworks like PyTorch actually work under the hood before you ever import torch.

Phase 14: The Agent Workbench and MCP Integration

Later phases transition from model training to agent deployment. The file phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py contains a minimal but complete agent workbench that ties together:

• Memory management – Context window optimization and conversation history
• Tool use – Function calling schemas and execution handlers
• MCP integration – Model Context Protocol servers for standardized context exchange

This workbench serves as the culmination of your mini framework, providing a plug-and-play package for shipping autonomous agents.

Automation and Tooling for Framework Maintenance

The scripts/ directory contains automation that keeps your mini framework healthy and synchronized:

• scripts/install_skills.py – Installs all generated prompts, skills, and MCP servers into your local LLM environment
• scripts/audit_lessons.py – Validates lesson contracts, enforces test coverage requirements, and verifies file naming conventions
• scripts/build_catalog.py and scripts/check_readme_counts.py – Maintain synchronization between lesson metadata and the top-level documentation

These tools ensure that as you extend the framework with custom lessons, the codebase remains linted, tested, and properly indexed.

Running the Framework: Practical Examples

Clone the repository and execute lessons to see the framework in action:

# Clone and navigate to the repository

git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch

# Run the linear algebra foundation lesson (Phase 1)

python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py

Install all generated artifacts for immediate use with local LLMs:

# Install prompts, skills, and agents into your environment

python scripts/install_skills.py

Execute the core mini framework implementation:

# Run the deep learning framework lesson (Phase 3, Lesson 10)

python phases/03-deep-learning-core/10-mini-framework/code/main.py

Deploy the agent workbench against a real repository:

# Execute the minimal agent workbench (Phase 14, Lesson 32)

python phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py /path/to/your/repo

Summary

• The AI Engineering from Scratch repository provides a 503-lesson curriculum that builds a complete mini framework for AI applications from first principles
• Each lesson follows a strict structure with code/, docs/en.md, and outputs/ directories, ensuring reusable, production-ready artifacts
• The Build-It/Use-It methodology forces implementation of algorithms from raw math before using PyTorch or JAX equivalents
• Key implementation files include phases/03-deep-learning-core/10-mini-framework/code/main.py for the core engine and phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py for agent orchestration
• Automation scripts like scripts/audit_lessons.py and scripts/install_skills.py maintain framework integrity and deployment readiness

Frequently Asked Questions

How is the mini framework structured across the 20 phases?

The framework evolves through six conceptual layers: mathematical primitives (Phases 1-2), machine learning fundamentals (Phase 2), deep learning cores (Phase 3), high-level AI components like transformers and diffusion models (Phases 4-9), production infrastructure including MCP servers (Phases 10-13), and finally autonomous agent engineering (Phases 14-20). Each phase adds reusable modules that accumulate into your final framework.

What is the Model Context Protocol (MCP) and how does it fit into the framework?

MCP (Model Context Protocol) is a standardized protocol for how agents exchange context, invoke tools, and stream results. Within the mini framework, MCP servers built in later phases provide a unified interface for tool calling and memory management, allowing your agents to interact with external systems through a consistent, type-safe API rather than ad-hoc integrations.

Can I use code from specific lessons in my production applications?

Yes. The repository is explicitly designed as a scaffolding generator where every lesson produces production-ready code. The outputs/ directory in each lesson contains the final artifact—whether a prompt template, skill definition, or MCP server—that you can copy directly into your projects. The scripts/install_skills.py utility automates this extraction for dependency management.

What is the purpose of the audit_lessons.py script?

The scripts/audit_lessons.py file validates that every lesson adheres to the repository's strict contracts: it checks for required documentation files, verifies that code examples run without errors, ensures proper test coverage exists, and enforces the file naming conventions defined in AGENTS.md. Running this script keeps your mini framework codebase consistent and error-free as you add custom lessons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}