# How Agent Memory Integration Works with the Curriculum's Skill Installation System

> Learn how Agent Memory integration deploys vector KV and graph stores with fusion logic and reflection pipelines for reusable memory capabilities in AI agents.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-08-29

---

**The curriculum's declarative skill installer materializes reusable memory capabilities by deploying parallel vector, KV, and graph stores alongside fusion logic and reflection pipelines that agents load at runtime.**

The rohitg00/ai-engineering-from-scratch repository implements a modular curriculum where AI capabilities are packaged as installable skills. The **Agent Memory integration** leverages this system to provide agents with persistent, multi-modal storage through a standardized installation and runtime interface.

## The Declarative Skill Installation Framework

At the core of the integration is [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py), a declarative installer that processes **SKILL.md** manifests and meta-files under each skill directory. When executed, the script reads the dependency graph and artifact definitions, then copies the required runtime components into `outputs/skills/`.

For the Agent Memory capability, the installer specifically targets [`skills/learn-agent-skills/SKILL.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skills/learn-agent-skills/SKILL.md). This manifest declares the **three parallel back-ends** (vector, KV, and graph stores), their persisted data paths, and the runtime entry points. The declarative approach ensures that adding new memory implementations requires only dropping a new [`SKILL.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/SKILL.md) under `skills/` and re-running the installer, preserving the curriculum’s plug-and-play philosophy.

## Agent Memory Architecture

The installed skill is not a monolithic database but a composite system designed for different retrieval patterns.

### Triple-Store Backend

The skill provisions three distinct storage layers:

- **Vector Store** – Persists semantic embeddings for similarity-based retrieval.
- **KV Store** – Maintains factual key-value pairs for exact lookups.
- **Graph Store** – Holds relational data for traversing connections and entity relationships.

These stores are instantiated from the artifacts copied to `outputs/skills/memory/` during installation.

### Fusion Logic and Scoring

When an agent queries its memory, the runtime executes a fusion algorithm defined in [`site/figures-agents4.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-agents4.js) (referenced as the *ae-memory-fusion* figure). The system queries all three stores in parallel and scores each result using the formula:

```

score = relevance × importance × recency

```

The highest-scoring results are synthesized into a unified answer, ensuring the agent retrieves the most contextually appropriate information regardless of which store originally housed it.

### Reflection Pipeline

To prevent unbounded growth of raw observations, the skill implements a **reflection pipeline** codified in [`site/figures-autoswarm5.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-autoswarm5.js). This periodic process (triggered every *N* steps or manually) synthesizes recent events into a higher-level *reflection* that summarizes key learnings. The compressor then writes these summaries back to all three stores, effectively distilling raw logs into actionable institutional knowledge.

## Runtime Integration and API

During agent initialization, the runtime creates a `Memory` object provided by `agents.memory`. This object automatically loads the three persisted stores from `outputs/skills/memory/` and registers the fusion layer.

The primary interface for retrieval is the `lookup(query)` method, which internally:

1. Queries the vector store for semantic matches.
2. Retrieves exact facts from the KV store.
3. Traverses the graph store for relational reasoning.
4. Applies the fusion scoring algorithm and returns the aggregated result.

After each interaction, the agent updates the stores with new observations. When the reflection threshold is reached, the pipeline compresses recent history into a reflection and re-injects it into the storage layers.

## Installing and Configuring Agent Memory

To deploy the memory capability in your environment:

```bash

# Install all declared skills, including the Agent Memory artifact

python scripts/install_skills.py

```

Once installed, agents can load the skill programmatically:

```python
from agents.core import Agent
from agents.memory import Memory

# Initialize memory (loads vector, KV, and graph stores from outputs/skills/memory)

mem = Memory()

# Instantiate agent with memory augmentation

agent = Agent(memory=mem)

# Process a query that requires historical context

response = agent.process("What did we decide about the data-privacy policy last week?")
print(response)  # Returns fused answer from all three stores

```

For debugging or forced consolidation, trigger the reflection routine manually:

```python

# Synthesize recent events into a high-level reflection

agent.memory.reflect()

```

## Summary

- The [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) installer processes declarative **SKILL.md** manifests to deploy memory artifacts into `outputs/skills/memory/`.
- The Agent Memory skill provides three parallel back-ends—**vector**, **KV**, and **graph**—for different retrieval patterns.
- Runtime fusion logic in [`site/figures-agents4.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-agents4.js) scores memories by relevance, importance, and recency to synthesize unified responses.
- The reflection pipeline in [`site/figures-autoswarm5.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-autoswarm5.js) compresses recent observations into high-level summaries and persists them back to the stores.
- Agents instantiate the `Memory` class to automatically load installed stores and expose the `lookup()` API for augmented reasoning.

## Frequently Asked Questions

### What file triggers the skill installation process?

The [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) script serves as the entry point. When executed, it scans the `skills/` directory for **SKILL.md** files, parses their artifact declarations, and copies the necessary components to `outputs/skills/`.

### How does the fusion algorithm prioritize memory sources?

The algorithm queries all three stores (vector, KV, and graph) simultaneously and calculates a composite score for each candidate using **relevance × importance × recency**. The candidate with the highest score across all stores is returned to the agent, ensuring optimal contextual retrieval.

### What triggers the memory reflection process?

Reflection occurs automatically every *N* execution steps or can be triggered manually by calling `agent.memory.reflect()`. This routine, implemented in [`site/figures-autoswarm5.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-autoswarm5.js), synthesizes recent raw observations into a compressed summary and writes it back to the stores to maintain efficient working memory.

### Where are the persisted memory databases located after installation?

Following a successful run of [`install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/install_skills.py), the runtime artifacts—including the persisted vector, KV, and graph databases—are located under `outputs/skills/memory/`. The `Memory` class loads these files at agent startup to restore the previous session state.