How the gpt-engineer FileSelector Determines Which Files to Include in Context

The gpt-engineer FileSelector uses a two-stage pipeline—first scanning the project directory while filtering out hidden files, dependency folders, and gitignored paths, then applying user selections from an interactive TOML file—to build the final list of files fed to the LLM as context.

The FileSelector in the AntonOsika/gpt-engineer repository is the gatekeeper that decides exactly which source files appear in the LLM's prompt context. Located in gpt_engineer/applications/cli/file_selector.py, this component combines automated filtering with interactive user approval to prevent context overflow and ensure only relevant code reaches the model.

Stage 1: Automated Directory Scanning and Filtering

The first stage is handled by FileSelector.get_current_files, which builds a complete inventory of candidate files by walking the project tree and aggressively discarding irrelevant paths.

Resolving the Project Root and Globbing Files

The process begins by normalizing the input path and recursively discovering all file entries. According to the source code, the selector performs the following steps:

1. Resolve to absolute path: project_path = Path(project_path).resolve() ensures the search starts from a normalized, absolute location.【L94-L96】
2. Recursive glob: file_list = project_path.glob("**/*") captures every entry in the directory tree.【L98-L99】
3. Keep files only: The iterator filters for regular files using if path.is_file(): before processing.【L100-L101】
4. Convert to relative paths: Each file is converted to a project-relative path via relpath = path.relative_to(project_path) for consistent internal representation.【L102-L103】

Filtering Out Hidden Files and System Directories

Before a file enters the candidate pool, it must pass several exclusion checks implemented as guard clauses:

• Hidden files: Any path containing a component starting with . is skipped via if any(part.startswith(".") for part in parts): continue.【L104-L105】
• Dependency folders: The selector maintains an IGNORE_FOLDERS list containing site-packages, node_modules, venv, and __pycache__. If any path part matches these, the file is discarded with if any(part in self.IGNORE_FOLDERS for part in parts): continue.【L106-L107】
• Prompt sentinel: Files named exactly prompt are excluded as they serve a special role in the gpt-engineer workflow.【L108-L109】

Respecting Gitignore Rules

For projects under Git version control, the selector applies additional filtering via filter_by_gitignore from gpt_engineer/core/git.py. The logic checks if is_git_repo(project_path) and "projects" not in project_path.parts before invoking the filter, ensuring that .gitignore rules are respected unless the project is located under a projects folder (which triggers a bypass).【L113-L115】

Finally, the method returns a deterministic, sorted list: return sorted(all_files, key=lambda x: Path(x).as_posix()).【L116-L117】

Stage 2: Interactive User Selection via TOML

Once the automated scan produces a clean file inventory, the second stage—FileSelector.get_files_from_toml—applies human curation through a TOML configuration file.

Loading and Parsing the Selection File

The selector reads the user-edited TOML file (typically generated by editor_file_selector) using edited_tree = toml.load(toml_file).【L94-L95】 This file contains a files mapping where keys are relative paths and values indicate selection status.

Extracting and Validating User Choices

The method iterates through the TOML structure to collect explicitly selected files:

for file, _ in edited_tree["files"].items():
    selected_files.append(file)  # 【L108-L110】

If the resulting list is empty—meaning the user deselected everything—the selector raises an exception to prevent context-less execution: if not selected_files: raise Exception("No files were selected…").【L112-L115】

How ask_for_files Orchestrates the Workflow

The public entry point ask_for_files coordinates the two stages based on the runtime environment.

Test Mode vs. Interactive Mode

• Test mode: When the GPTE_TEST_MODE environment variable is set, the selector bypasses the interactive UI and directly calls get_files_from_toml on a pre-existing TOML file.
• Interactive mode: The selector either reuses an existing .toml or generates a fresh one via editor_file_selector, then reads user choices with selected_files = self.get_files_from_toml(self.project_path, self.toml_path).【L96-L98】

Each selected path is then opened, read, and inserted into a FilesDict instance—the final data structure consumed by the LLM engine as its context.

Practical Code Examples

Instantiating the Selector and Retrieving Context Files

from gpt_engineer.applications.cli.file_selector import FileSelector

# Assume we are inside a project root

selector = FileSelector(project_path=".")
files_dict, linting_enabled = selector.ask_for_files()

# `files_dict` now maps relative paths → file contents

Running in Test Mode

export GPTE_TEST_MODE=1   # environment variable

python -m pytest tests/core/test_file_selector_enhancements.py

In this mode, the selector reads the pre-written file_selection.toml without launching an editor.

Inspecting the Generated TOML

# After an interactive run, open the file to see which entries were commented out

with open("metadata/file_selection.toml") as f:
    print(f.read())

Lines prefixed with # are deselected; uncomment them to include the file in subsequent runs.

Summary

• The FileSelector operates in two distinct phases: automated directory scanning (get_current_files) and interactive user filtering (get_files_from_toml).
• Automatic filters exclude hidden files, common dependency directories (node_modules, venv, __pycache__, site-packages), and paths matching .gitignore rules.
• The prompt sentinel file is always excluded from context.
• User selections are persisted in a TOML file where commented lines represent deselected files.
• The GPTE_TEST_MODE environment variable enables headless operation for automated testing.

Frequently Asked Questions

What files does the FileSelector exclude automatically?

The FileSelector automatically removes hidden files (any path component starting with .), files located within IGNORE_FOLDERS (site-packages, node_modules, venv, __pycache__), and any file named exactly prompt. It also respects .gitignore rules for Git repositories unless the project resides under a projects folder.

How does the FileSelector handle .gitignore rules?

When is_git_repo(project_path) returns true and the path does not contain projects, the selector calls filter_by_gitignore from gpt_engineer/core/git.py to strip out any files that match patterns in the repository's .gitignore file. This occurs at the end of the get_current_files scan before the list is sorted and returned.

Can I use the FileSelector programmatically without the interactive UI?

Yes. By setting the environment variable GPTE_TEST_MODE=1, you enable test mode where ask_for_files skips the interactive editor and reads selections directly from the TOML file at metadata/file_selection.toml. This allows headless, scriptable usage of the selector in CI/CD pipelines or automated tests.

What happens if no files are selected in the TOML?

If get_files_from_toml parses the TOML and finds an empty selection list—either because all files were commented out or the files mapping is missing—it raises an exception with the message "No files were selected…". This prevents the LLM from receiving an empty context, which would cause the generation to fail.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}