# Developing a Mini Framework for AI Applications: A Step-by-Step Engineering Guide

> Build a production AI framework from scratch. This guide offers 503 hands-on lessons covering linear algebra to autonomous agents. Learn AI engineering principles step-by-step.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-07-19

---

**The AI Engineering from Scratch repository teaches you to build a production-ready mini framework for AI applications by implementing 503 hands-on lessons across 20 phases, from linear algebra fundamentals to autonomous agents using the Model Context Protocol.**

This comprehensive curriculum in `rohitg00/ai-engineering-from-scratch` functions as both a living textbook and a reusable codebase. Each lesson contributes directly to a modular AI stack that you can extract into real-world projects, covering everything from raw mathematical primitives to high-level agent orchestration.

## Curriculum Architecture: 20 Phases to a Modular AI Stack

The repository structures **503 lessons** across **20 sequential phases** that progressively assemble your mini framework. Each phase adds a new abstraction layer—starting with foundational math and culminating in autonomous agent engineering.

### The Lesson Structure

Every lesson follows a strict directory convention under `phases/<NN>-<phase-name>/<NN>-<lesson-name>/`:

- **`code/`** – Runnable implementations in Python, TypeScript, Rust, or Julia
- **[`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md)** – Narrative documentation with front-matter describing learning objectives
- **`outputs/`** – The artifact produced by the "USE IT" step (prompts, skills, agents, or MCP servers)

This layout is documented in the repository's [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and ensures every lesson is self-contained and importable into production codebases.

### Build-It versus Use-It Methodology

Each lesson employs a dual-track approach that forces deep understanding of both mechanics and APIs:

1. **Build-It** – Implement algorithms from raw math without external libraries
2. **Use-It** – Run the same logic using production libraries like PyTorch, JAX, or the custom MCP framework

As noted in the [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), this split ensures you understand low-level tensor operations before abstracting them behind high-level interfaces.

## Assembling the Mini Framework: From Math to Agents

The curriculum gradually constructs your reusable mini framework through six evolutionary stages:

- **Math primitives** – Vectors, matrices, and tensor operations
- **ML building blocks** – Linear regression, gradient descent, and optimizers
- **Neural-network core** – Perceptron implementation through back-propagation
- **High-level components** – Tokenizers, attention mechanisms, transformers, and diffusion models
- **Production glue** – Model Context Protocol (MCP) servers, tool schemas, async handling, and security layers
- **Agent workbench** – Memory systems, planning modules, orchestration logic, and observability hooks

### Phase 1-3: Foundations and Core Implementation

Early phases establish the framework's computational backbone. In [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py), you construct the core autograd engine and neural network layers from scratch using only NumPy arrays and Python classes.

This lesson implements the fundamental `Tensor` class with automatic differentiation, demonstrating how modern frameworks like PyTorch actually work under the hood before you ever import `torch`.

### Phase 14: The Agent Workbench and MCP Integration

Later phases transition from model training to agent deployment. The file [`phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py) contains a minimal but complete agent workbench that ties together:

- **Memory management** – Context window optimization and conversation history
- **Tool use** – Function calling schemas and execution handlers
- **MCP integration** – Model Context Protocol servers for standardized context exchange

This workbench serves as the culmination of your mini framework, providing a plug-and-play package for shipping autonomous agents.

## Automation and Tooling for Framework Maintenance

The `scripts/` directory contains automation that keeps your mini framework healthy and synchronized:

- **[`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py)** – Installs all generated prompts, skills, and MCP servers into your local LLM environment
- **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** – Validates lesson contracts, enforces test coverage requirements, and verifies file naming conventions
- **[`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py)** and **[`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py)** – Maintain synchronization between lesson metadata and the top-level documentation

These tools ensure that as you extend the framework with custom lessons, the codebase remains linted, tested, and properly indexed.

## Running the Framework: Practical Examples

Clone the repository and execute lessons to see the framework in action:

```bash

# Clone and navigate to the repository

git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch

# Run the linear algebra foundation lesson (Phase 1)

python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py

```

Install all generated artifacts for immediate use with local LLMs:

```bash

# Install prompts, skills, and agents into your environment

python scripts/install_skills.py

```

Execute the core mini framework implementation:

```bash

# Run the deep learning framework lesson (Phase 3, Lesson 10)

python phases/03-deep-learning-core/10-mini-framework/code/main.py

```

Deploy the agent workbench against a real repository:

```bash

# Execute the minimal agent workbench (Phase 14, Lesson 32)

python phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py /path/to/your/repo

```

## Summary

- **The AI Engineering from Scratch repository** provides a 503-lesson curriculum that builds a complete mini framework for AI applications from first principles
- **Each lesson follows a strict structure** with `code/`, [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), and `outputs/` directories, ensuring reusable, production-ready artifacts
- **The Build-It/Use-It methodology** forces implementation of algorithms from raw math before using PyTorch or JAX equivalents
- **Key implementation files** include [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py) for the core engine and [`phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/14-agent-engineering/32-minimal-agent-workbench/code/main.py) for agent orchestration
- **Automation scripts** like [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) and [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) maintain framework integrity and deployment readiness

## Frequently Asked Questions

### How is the mini framework structured across the 20 phases?

The framework evolves through six conceptual layers: mathematical primitives (Phases 1-2), machine learning fundamentals (Phase 2), deep learning cores (Phase 3), high-level AI components like transformers and diffusion models (Phases 4-9), production infrastructure including MCP servers (Phases 10-13), and finally autonomous agent engineering (Phases 14-20). Each phase adds reusable modules that accumulate into your final framework.

### What is the Model Context Protocol (MCP) and how does it fit into the framework?

**MCP (Model Context Protocol)** is a standardized protocol for how agents exchange context, invoke tools, and stream results. Within the mini framework, MCP servers built in later phases provide a unified interface for tool calling and memory management, allowing your agents to interact with external systems through a consistent, type-safe API rather than ad-hoc integrations.

### Can I use code from specific lessons in my production applications?

Yes. The repository is explicitly designed as a **scaffolding generator** where every lesson produces production-ready code. The `outputs/` directory in each lesson contains the final artifact—whether a prompt template, skill definition, or MCP server—that you can copy directly into your projects. The [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) utility automates this extraction for dependency management.

### What is the purpose of the audit_lessons.py script?

The [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) file validates that every lesson adheres to the repository's strict contracts: it checks for required documentation files, verifies that code examples run without errors, ensures proper test coverage exists, and enforces the file naming conventions defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md). Running this script keeps your mini framework codebase consistent and error-free as you add custom lessons.