How to Create a Feedback Loop for Guideline Improvement in LLM-Assisted Development

A feedback loop for guideline improvement captures real-world LLM failures, maps them to specific principles in SKILL.md, and validates fixes through updated examples in EXAMPLES.md before merging changes to CLAUDE.md and .cursor/rules/karpathy-guidelines.mdc.

The multica-ai/andrej-karpathy-skills repository maintains a single source of truth for coding guidelines that steer Claude-Code and other LLM-powered tools. To keep these guidelines effective as model behavior evolves, you need a repeatable feedback loop for guideline improvement that converts observation into concrete, validated updates.

The Four Stages of the Feedback Loop

The feedback loop for guideline improvement consists of four tightly-coupled stages that map directly to the four principles defined in SKILL.md (Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution).

1. Capture

Record a "what went wrong / what could be better" note after each LLM-assisted session. The canonical place for persistent feedback is the README.md "Install" and "How to know it’s working" sections, which already describe success signals 【README.md†L40-L48】. Alternatively, open a repository issue using a structured template.

2. Analyze

Review the captured note, identify which principle was violated, and draft a concrete wording change. Reference the principle definitions in SKILL.md when writing the amendment 【SKILL.md†L13-L35】. This stage transforms vague observations into specific guideline violations.

3. Iterate

Create a PR that updates the authoritative guideline files (CLAUDE.md, SKILL.md, EXAMPLES.md) and, if needed, the Cursor rule (.cursor/rules/karpathy-guidelines.mdc). Add a minimal test or example that demonstrates the new wording prevents the previous mistake. The example section in EXAMPLES.md shows the before/after pattern used throughout the repo 【EXAMPLES.md†L7-L30】.

4. Verify

Run the repository’s own examples to confirm the updated wording guides the LLM to the desired behavior. The "How to know it’s working" checklist in README.md lists the observable signals that indicate the feedback loop for guideline improvement succeeded 【README.md†L40-L48】. Because every change is gate-kept by a PR, the loop is automatically versioned, reviewed, and documented.

Practical Implementation: From Observation to Merge

Follow this workflow to execute the feedback loop for guideline improvement in the multica-ai/andrej-karpathy-skills repository:

1. Run Claude-Code and identify a guideline-related problem (e.g., the LLM adds an unnecessary abstraction).
2. Open an issue titled "Feedback: [Principle] – [Brief description]". Quote the offending output and reference the violated principle from SKILL.md.
3. Create a branch feedback/[principle]-[description] and edit the affected sections.
4. Add a new example to EXAMPLES.md showing the bad LLM output and the corrected interaction.
5. Run the example locally to confirm the LLM now behaves correctly.
6. Submit the PR for review.
7. Merge once validated – the updated guidelines propagate to all downstream users via the installation script.

Code Examples: Feedback Loop in Action

Capturing a Problem (Issue Template)

When initiating the feedback loop for guideline improvement, use this structured template:

**Title:** Feedback – Simplicity First: Unnecessary abstraction added

**Problem:**  
Claude added a full Strategy pattern for a simple discount calculation.

**Observed Violation:**  
*Simplicity First* – created 30+ lines of infrastructure that weren't requested.

**Suggested Guideline Update:**  
Add a bullet to `CLAUDE.md` warning against premature abstraction.

Drafting the Guideline Change

Edit CLAUDE.md to clarify the principle. The original wording lives at lines 30-38 【CLAUDE.md†L30-L38】:

--- a/CLAUDE.md
+++ b/CLAUDE.md
@@
- **Simplicity First** – Minimum code that solves the problem. Nothing speculative.
+ **Simplicity First** – Write the smallest, most direct solution that satisfies the request.  
+   *Avoid adding design patterns, abstractions, or configurability before a real need is expressed.*

Adding Validation Examples

Append to EXAMPLES.md to demonstrate the fix. Reference the before/after pattern at lines 7-30 【EXAMPLES.md†L7-L30】:

## 5. Feedback Loop – Adding Guidelines

### Before

```python
class DiscountStrategy(ABC):
    ...

After applying the updated guideline

def calculate_discount(amount: float, percent: float) -> float:
    """Calculate discount amount. percent should be 0‑100."""
    return amount * (percent / 100)

Result: The LLM now asks "Do you need multiple discount types?" instead of emitting the full pattern.

### Verifying the Fix

Use the checklist in [`README.md`](https://github.com/multica-ai/andrej-karpathy-skills/blob/main/README.md) (lines 40-48) 【README.md†L40-L48】 to confirm the feedback loop for guideline improvement succeeded:

```bash

# Run the example script

python examples/run_discount_example.py

# Expected: prompt for clarification, not full strategy implementation

Key Files That Power the Feedback Loop

These seven files constitute the single source of truth and enable the feedback loop for guideline improvement:

File	Role	Critical Lines
README.md	Overview, installation, and success criteria checklist	【L40-L48】
CLAUDE.md	Canonical guideline document imported by downstream projects	【L30-L38】
SKILL.md	Structured principle definitions (Think Before Coding, Simplicity First, etc.)	【L13-L35】
EXAMPLES.md	Before/after validation cases for each guideline	【L7-L30】
.cursor/rules/karpathy-guidelines.mdc	Cursor IDE integration rule	N/A
Issue templates	Structured feedback capture	N/A
Installation script	curl … > CLAUDE.md distribution mechanism	N/A

Because the repository uses a minimal curl installation script, every merged improvement to the feedback loop for guideline improvement immediately propagates to all Claude-Code users.

Summary

• A feedback loop for guideline improvement requires four stages: Capture, Analyze, Iterate, and Verify.
• Capture observations in README.md sections or GitHub issues, analyzing them against the four principles defined in SKILL.md 【SKILL.md†L13-L35】.
• Iterate by updating CLAUDE.md, SKILL.md, and EXAMPLES.md, adding before/after examples at lines 7-30 【EXAMPLES.md†L7-L30】 to validate the change.
• Verify success using the checklist in README.md lines 40-48 【README.md†L40-L48】.
• All changes propagate downstream through the repository's curl installation mechanism, making the feedback loop for guideline improvement a continuous, versioned process.

Frequently Asked Questions

What triggers the feedback loop for guideline improvement?

A trigger occurs whenever an LLM-assisted coding session produces output that violates one of the four principles (Think Before Coding, Simplicity First, Surgical Changes, Goal-Driven Execution). Common triggers include unnecessary abstractions, over-refactoring, or guessing requirements rather than asking. Document these observations in a GitHub issue or the README.md feedback sections to initiate the loop.

Which file should I edit first when implementing guideline changes?

Start with SKILL.md to reference the specific principle definition being violated 【SKILL.md†L13-L35】, then edit CLAUDE.md to update the canonical wording that downstream projects import 【CLAUDE.md†L30-L38】. Finally, add a validation example to EXAMPLES.md showing the before/after behavior 【EXAMPLES.md†L7-L30】 to ensure the change has the intended effect on LLM output.

How do I verify that a guideline update actually improves LLM behavior?

Use the "How to know it’s working" checklist in README.md 【README.md†L40-L48】. Run the specific example you added to EXAMPLES.md through Claude-Code or the test script. The LLM should now exhibit the desired behavior (e.g., asking for clarification instead of over-refactoring). If the output matches the "After" state in your example, the verification stage is complete.

Do I need to update the Cursor rule file separately?

Yes. When you update CLAUDE.md or SKILL.md, mirror the changes to .cursor/rules/karpathy-guidelines.mdc to maintain consistency across IDEs. The repository treats CLAUDE.md as the canonical source, but the Cursor rule file ensures that users of the Cursor IDE benefit from the same feedback loop for guideline improvements. Both files should remain synchronized after each iteration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}