# How to Create a Feedback Loop for Guideline Improvement in LLM-Assisted Development

> Learn to build an effective feedback loop for guideline improvement in LLM-assisted development. Capture failures, map principles, and validate fixes for better AI guidance.

- Repository: [multica-ai/andrej-karpathy-skills](https://github.com/multica-ai/andrej-karpathy-skills)
- Tags: how-to-guide
- Published: 2026-04-19

---

**A feedback loop for guideline improvement captures real-world LLM failures, maps them to specific principles in [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md), and validates fixes through updated examples in [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) before merging changes to [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) and `.cursor/rules/karpathy-guidelines.mdc`.**

The `multica-ai/andrej-karpathy-skills` repository maintains a single source of truth for coding guidelines that steer Claude-Code and other LLM-powered tools. To keep these guidelines effective as model behavior evolves, you need a repeatable feedback loop for guideline improvement that converts observation into concrete, validated updates.

## The Four Stages of the Feedback Loop

The feedback loop for guideline improvement consists of four tightly-coupled stages that map directly to the four principles defined in [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md) (Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution).

### 1. Capture

Record a "what went wrong / what could be better" note after each LLM-assisted session. The canonical place for persistent feedback is the **[`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md)** "Install" and "How to know it’s working" sections, which already describe success signals 【README.md†L40-L48】. Alternatively, open a repository issue using a structured template.

### 2. Analyze

Review the captured note, identify which principle was violated, and draft a concrete wording change. Reference the principle definitions in **[`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md)** when writing the amendment 【SKILL.md†L13-L35】. This stage transforms vague observations into specific guideline violations.

### 3. Iterate

Create a PR that updates the authoritative guideline files ([`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md), [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md), [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md)) and, if needed, the Cursor rule (`.cursor/rules/karpathy-guidelines.mdc`). Add a minimal test or example that demonstrates the new wording prevents the previous mistake. The example section in **[`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md)** shows the before/after pattern used throughout the repo 【EXAMPLES.md†L7-L30】.

### 4. Verify

Run the repository’s own examples to confirm the updated wording guides the LLM to the desired behavior. The "How to know it’s working" checklist in **[`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md)** lists the observable signals that indicate the feedback loop for guideline improvement succeeded 【README.md†L40-L48】. Because every change is gate-kept by a PR, the loop is automatically versioned, reviewed, and documented.

## Practical Implementation: From Observation to Merge

Follow this workflow to execute the feedback loop for guideline improvement in the `multica-ai/andrej-karpathy-skills` repository:

1. **Run Claude-Code** and identify a guideline-related problem (e.g., the LLM adds an unnecessary abstraction).
2. **Open an issue** titled "Feedback: [Principle] – [Brief description]". Quote the offending output and reference the violated principle from [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md).
3. **Create a branch** `feedback/[principle]-[description]` and edit the affected sections.
4. **Add a new example** to [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) showing the bad LLM output and the corrected interaction.
5. **Run the example** locally to confirm the LLM now behaves correctly.
6. **Submit the PR** for review.
7. **Merge** once validated – the updated guidelines propagate to all downstream users via the installation script.

## Code Examples: Feedback Loop in Action

### Capturing a Problem (Issue Template)

When initiating the feedback loop for guideline improvement, use this structured template:

```markdown
**Title:** Feedback – Simplicity First: Unnecessary abstraction added

**Problem:**  
Claude added a full Strategy pattern for a simple discount calculation.

**Observed Violation:**  
*Simplicity First* – created 30+ lines of infrastructure that weren't requested.

**Suggested Guideline Update:**  
Add a bullet to `CLAUDE.md` warning against premature abstraction.

```

### Drafting the Guideline Change

Edit [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) to clarify the principle. The original wording lives at lines 30-38 【CLAUDE.md†L30-L38】:

```diff
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@
- **Simplicity First** – Minimum code that solves the problem. Nothing speculative.
+ **Simplicity First** – Write the smallest, most direct solution that satisfies the request.  
+   *Avoid adding design patterns, abstractions, or configurability before a real need is expressed.*

```

### Adding Validation Examples

Append to [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) to demonstrate the fix. Reference the before/after pattern at lines 7-30 【EXAMPLES.md†L7-L30】:

```markdown

## 5. Feedback Loop – Adding Guidelines

### Before

```python
class DiscountStrategy(ABC):
    ...

```

### After applying the updated guideline

```python
def calculate_discount(amount: float, percent: float) -> float:
    """Calculate discount amount. percent should be 0‑100."""
    return amount * (percent / 100)

```

**Result:** The LLM now asks "Do you need multiple discount types?" instead of emitting the full pattern.

```

### Verifying the Fix

Use the checklist in [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) (lines 40-48) 【README.md†L40-L48】 to confirm the feedback loop for guideline improvement succeeded:

```bash

# Run the example script

python examples/run_discount_example.py

# Expected: prompt for clarification, not full strategy implementation

```

## Key Files That Power the Feedback Loop

These seven files constitute the single source of truth and enable the feedback loop for guideline improvement:

| File | Role | Critical Lines |
|------|------|----------------|
| [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) | Overview, installation, and success criteria checklist | 【L40-L48】 |
| [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) | Canonical guideline document imported by downstream projects | 【L30-L38】 |
| [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md) | Structured principle definitions (Think Before Coding, Simplicity First, etc.) | 【L13-L35】 |
| [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) | Before/after validation cases for each guideline | 【L7-L30】 |
| `.cursor/rules/karpathy-guidelines.mdc` | Cursor IDE integration rule | N/A |
| Issue templates | Structured feedback capture | N/A |
| Installation script | `curl … > CLAUDE.md` distribution mechanism | N/A |

Because the repository uses a minimal `curl` installation script, every merged improvement to the feedback loop for guideline improvement immediately propagates to all Claude-Code users.

## Summary

- A **feedback loop for guideline improvement** requires four stages: **Capture**, **Analyze**, **Iterate**, and **Verify**.
- Capture observations in [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) sections or GitHub issues, analyzing them against the four principles defined in [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md) 【SKILL.md†L13-L35】.
- Iterate by updating [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md), [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md), and [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md), adding before/after examples at lines 7-30 【EXAMPLES.md†L7-L30】 to validate the change.
- Verify success using the checklist in [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) lines 40-48 【README.md†L40-L48】.
- All changes propagate downstream through the repository's `curl` installation mechanism, making the feedback loop for guideline improvement a continuous, versioned process.

## Frequently Asked Questions

### What triggers the feedback loop for guideline improvement?

A trigger occurs whenever an LLM-assisted coding session produces output that violates one of the four principles (Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution). Common triggers include unnecessary abstractions, over-refactoring, or guessing requirements rather than asking. Document these observations in a GitHub issue or the [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) feedback sections to initiate the loop.

### Which file should I edit first when implementing guideline changes?

Start with [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md) to reference the specific principle definition being violated 【SKILL.md†L13-L35】, then edit [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) to update the canonical wording that downstream projects import 【CLAUDE.md†L30-L38】. Finally, add a validation example to [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) showing the before/after behavior 【EXAMPLES.md†L7-L30】 to ensure the change has the intended effect on LLM output.

### How do I verify that a guideline update actually improves LLM behavior?

Use the "How to know it’s working" checklist in [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) 【README.md†L40-L48】. Run the specific example you added to [`EXAMPLES.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/EXAMPLES.md) through Claude-Code or the test script. The LLM should now exhibit the desired behavior (e.g., asking for clarification instead of over-refactoring). If the output matches the "After" state in your example, the verification stage is complete.

### Do I need to update the Cursor rule file separately?

Yes. When you update [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) or [`SKILL.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/SKILL.md), mirror the changes to `.cursor/rules/karpathy-guidelines.mdc` to maintain consistency across IDEs. The repository treats [`CLAUDE.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/CLAUDE.md) as the canonical source, but the Cursor rule file ensures that users of the Cursor IDE benefit from the same feedback loop for guideline improvements. Both files should remain synchronized after each iteration.