How GPT-Engineer Learning Mode Collects and Incorporates User Feedback

GPT-Engineer's learning mode captures user feedback through a privacy-first pipeline that secures explicit consent, prompts for structured interactive reviews, and packages responses into serializable Learning records for analytics integration.

The learning mode in AntonOsika/gpt-engineer provides a systematic approach to collecting user feedback on generated code. Implemented primarily in gpt_engineer/applications/cli/learning.py, this feature enables the project to gather structured insights while respecting user privacy through explicit consent mechanisms and session management.

The Learning Mode Pipeline

The learning system operates through a five-step pipeline that transforms raw user interactions into structured analytics data. Each step is designed to balance data richness with user privacy and control.

Step 1: Securing User Consent

Before collecting any feedback, the system verifies user consent through check_collection_consent() in gpt_engineer/applications/cli/learning.py. This function looks for a hidden file named .gpte_consent in the working directory.

If the file exists and contains true, consent is granted immediately. If the file is missing or contains any other value, ask_collection_consent() prompts the user with a y/n question and persists the answer to .gpte_consent for future runs. This ensures users are not repeatedly prompted while maintaining a clear audit trail of consent.

Step 2: Capturing Interactive Feedback

With consent secured, human_review_input() initiates an interactive review process. The function asks users three specific questions about the generated code:

• Did the code run successfully?
• Was the generated code perfect?
• Was the generated code useful?

Each question accepts y/n/u (yes/no/unsure) responses, validated through ask_for_valid_input() which repeats until a valid answer is provided. Users may also provide optional free-form comments. These responses are normalized into booleans and stored in a Review dataclass containing fields for ran, perfect, useful, and comments.

Step 3: Session Identification

To correlate feedback across multiple interactions, get_session() generates stable session identifiers. This function checks for a file named gpt_engineer_user_id.txt in the operating system's temporary directory.

If the file exists, its contents are returned as the session ID. If absent, the function generates a random 32-bit integer, writes it to the file, and returns it. In cases where file access fails, the system falls back to an ephemeral identifier prefixed with ephemeral_. This approach balances persistence with privacy, avoiding personally identifiable information while enabling longitudinal analysis.

Step 4: Building the Learning Record

The extract_learning() function aggregates all collected data into a structured Learning dataclass. This function receives the original Prompt object, model name, temperature, configuration tuple, a DiskMemory instance containing session logs, and the Review from step 2.

The function serializes the prompt, configuration, and memory logs to JSON format, incorporates the session ID from step 3, and returns a Learning object containing:

• Prompt JSON representation
• Model name and temperature settings
• Configuration JSON
• Full memory logs
• Review data (or None)
• UTC timestamp
• Schema version identifier

Step 5: Analytics Integration

The finalized Learning record is passed to collect_learnings() in gpt_engineer/applications/cli/collect.py. This module optionally transmits the data to RudderStack via send_learning() and implements graceful truncation if the payload exceeds size limits. While this step operates outside the core learning module, it represents the final incorporation of user feedback into analytics pipelines for model and UX improvements.

Implementation Examples

Collecting Feedback During a GPT-Engineer Run

from gpt_engineer.applications.cli.learning import (
    human_review_input,
    extract_learning,
)
from gpt_engineer.core.prompt import Prompt
from gpt_engineer.core.default.disk_memory import DiskMemory

# Initialize components

prompt = Prompt.from_file("projects/example/prompt")
model = "gpt-4"
temperature = 0.7
config = ("project_name=example", "max_steps=10")
memory = DiskMemory()

# Request user review (consent handled internally)

review = human_review_input()

if review:
    # Build the learning record

    learning = extract_learning(
        prompt, model, temperature, config, memory, review
    )
    print(f"Collected learning data: {learning.to_dict()}")
else:
    print("User opted out of feedback collection.")

Resetting Consent for Testing

from pathlib import Path

consent_file = Path(".gpte_consent")
if consent_file.exists():
    consent_file.unlink()
print("Consent cleared. Next run will prompt for consent again.")

Inspecting Session Data

from gpt_engineer.applications.cli.learning import get_session

session_id = get_session()
print(f"Current session ID: {session_id}")

Summary

• Consent Management: The learning mode stores consent in .gpte_consent and only proceeds with explicit user approval via check_collection_consent().
• Structured Feedback: human_review_input() captures boolean responses to three quality questions plus optional comments, stored in a Review dataclass.
• Session Tracking: get_session() generates stable identifiers using gpt_engineer_user_id.txt in the temp directory, enabling longitudinal analysis without PII.
• Data Aggregation: extract_learning() serializes prompts, configuration, memory logs, and reviews into a Learning dataclass with UTC timestamps.
• Analytics Integration: The collect.py module transmits Learning records to RudderStack, completing the feedback incorporation pipeline.

Frequently Asked Questions

Where is user consent stored in GPT-Engineer?

User consent is persisted in a hidden file named .gpte_consent in the working directory. The check_collection_consent() function reads this file to determine if consent has been granted, while ask_collection_consent() writes the user's y/n response to this file for future sessions.

What specific questions does the learning mode ask users?

The learning mode asks three structured questions via human_review_input(): whether the generated code ran successfully, whether it was perfect, and whether it was useful. Users respond with y (yes), n (no), or u (unsure), and may optionally provide free-form comments explaining their ratings.

How does GPT-Engineer handle session tracking without collecting PII?

The get_session() function generates a stable session identifier by reading or creating a file named gpt_engineer_user_id.txt in the operating system's temporary directory. This file contains a random 32-bit integer that persists across runs but contains no personally identifiable information, enabling longitudinal analysis while maintaining privacy.

Can users opt out of data collection after previously consenting?

Yes, users can revoke consent by deleting the .gpte_consent file from their working directory. The next time GPT-Engineer runs, check_collection_consent() will fail to find the file and trigger ask_collection_consent(), allowing the user to decline participation. Additionally, simply answering n to the review prompt in human_review_input() skips data collection for that specific session without affecting global consent settings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}