# How the gpt-engineer FileSelector Determines Which Files to Include in Context

> Discover how gpt-engineer FileSelector selects files for LLM context. Learn about its two-stage pipeline: scanning directories and applying user TOML selections.

- Repository: [Anton Osika/gpt-engineer](https://github.com/AntonOsika/gpt-engineer)
- Tags: internals
- Published: 2026-03-06

---

**The gpt-engineer FileSelector uses a two-stage pipeline—first scanning the project directory while filtering out hidden files, dependency folders, and gitignored paths, then applying user selections from an interactive TOML file—to build the final list of files fed to the LLM as context.**

The FileSelector in the AntonOsika/gpt-engineer repository is the gatekeeper that decides exactly which source files appear in the LLM's prompt context. Located in [`gpt_engineer/applications/cli/file_selector.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/applications/cli/file_selector.py), this component combines automated filtering with interactive user approval to prevent context overflow and ensure only relevant code reaches the model.

## Stage 1: Automated Directory Scanning and Filtering

The first stage is handled by `FileSelector.get_current_files`, which builds a complete inventory of candidate files by walking the project tree and aggressively discarding irrelevant paths.

### Resolving the Project Root and Globbing Files

The process begins by normalizing the input path and recursively discovering all file entries. According to the source code, the selector performs the following steps:

1. **Resolve to absolute path**: `project_path = Path(project_path).resolve()` ensures the search starts from a normalized, absolute location.【L94-L96】
2. **Recursive glob**: `file_list = project_path.glob("**/*")` captures every entry in the directory tree.【L98-L99】
3. **Keep files only**: The iterator filters for regular files using `if path.is_file():` before processing.【L100-L101】
4. **Convert to relative paths**: Each file is converted to a project-relative path via `relpath = path.relative_to(project_path)` for consistent internal representation.【L102-L103】

### Filtering Out Hidden Files and System Directories

Before a file enters the candidate pool, it must pass several exclusion checks implemented as guard clauses:

- **Hidden files**: Any path containing a component starting with `.` is skipped via `if any(part.startswith(".") for part in parts): continue`.【L104-L105】
- **Dependency folders**: The selector maintains an `IGNORE_FOLDERS` list containing `site-packages`, `node_modules`, `venv`, and `__pycache__`. If any path part matches these, the file is discarded with `if any(part in self.IGNORE_FOLDERS for part in parts): continue`.【L106-L107】
- **Prompt sentinel**: Files named exactly `prompt` are excluded as they serve a special role in the gpt-engineer workflow.【L108-L109】

### Respecting Gitignore Rules

For projects under Git version control, the selector applies additional filtering via `filter_by_gitignore` from [`gpt_engineer/core/git.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/core/git.py). The logic checks `if is_git_repo(project_path) and "projects" not in project_path.parts` before invoking the filter, ensuring that `.gitignore` rules are respected unless the project is located under a `projects` folder (which triggers a bypass).【L113-L115】

Finally, the method returns a deterministic, sorted list: `return sorted(all_files, key=lambda x: Path(x).as_posix())`.【L116-L117】

## Stage 2: Interactive User Selection via TOML

Once the automated scan produces a clean file inventory, the second stage—`FileSelector.get_files_from_toml`—applies human curation through a TOML configuration file.

### Loading and Parsing the Selection File

The selector reads the user-edited TOML file (typically generated by `editor_file_selector`) using `edited_tree = toml.load(toml_file)`.【L94-L95】 This file contains a `files` mapping where keys are relative paths and values indicate selection status.

### Extracting and Validating User Choices

The method iterates through the TOML structure to collect explicitly selected files:

```python
for file, _ in edited_tree["files"].items():
    selected_files.append(file)  # 【L108-L110】

```

If the resulting list is empty—meaning the user deselected everything—the selector raises an exception to prevent context-less execution: `if not selected_files: raise Exception("No files were selected…")`.【L112-L115】

## How ask_for_files Orchestrates the Workflow

The public entry point `ask_for_files` coordinates the two stages based on the runtime environment.

### Test Mode vs. Interactive Mode

- **Test mode**: When the `GPTE_TEST_MODE` environment variable is set, the selector bypasses the interactive UI and directly calls `get_files_from_toml` on a pre-existing TOML file.
- **Interactive mode**: The selector either reuses an existing `.toml` or generates a fresh one via `editor_file_selector`, then reads user choices with `selected_files = self.get_files_from_toml(self.project_path, self.toml_path)`.【L96-L98】

Each selected path is then opened, read, and inserted into a `FilesDict` instance—the final data structure consumed by the LLM engine as its **context**.

## Practical Code Examples

### Instantiating the Selector and Retrieving Context Files

```python
from gpt_engineer.applications.cli.file_selector import FileSelector

# Assume we are inside a project root

selector = FileSelector(project_path=".")
files_dict, linting_enabled = selector.ask_for_files()

# `files_dict` now maps relative paths → file contents

```

### Running in Test Mode

```bash
export GPTE_TEST_MODE=1   # environment variable

python -m pytest tests/core/test_file_selector_enhancements.py

```

In this mode, the selector reads the pre-written [`file_selection.toml`](https://github.com/AntonOsika/gpt-engineer/blob/main/file_selection.toml) without launching an editor.

### Inspecting the Generated TOML

```python

# After an interactive run, open the file to see which entries were commented out

with open("metadata/file_selection.toml") as f:
    print(f.read())

```

Lines prefixed with `# ` are *deselected*; uncomment them to include the file in subsequent runs.

## Summary

- The FileSelector operates in two distinct phases: automated directory scanning (`get_current_files`) and interactive user filtering (`get_files_from_toml`).
- Automatic filters exclude hidden files, common dependency directories (`node_modules`, `venv`, `__pycache__`, `site-packages`), and paths matching `.gitignore` rules.
- The `prompt` sentinel file is always excluded from context.
- User selections are persisted in a TOML file where commented lines represent deselected files.
- The `GPTE_TEST_MODE` environment variable enables headless operation for automated testing.

## Frequently Asked Questions

### What files does the FileSelector exclude automatically?

The FileSelector automatically removes hidden files (any path component starting with `.`), files located within `IGNORE_FOLDERS` (`site-packages`, `node_modules`, `venv`, `__pycache__`), and any file named exactly `prompt`. It also respects `.gitignore` rules for Git repositories unless the project resides under a `projects` folder.

### How does the FileSelector handle .gitignore rules?

When `is_git_repo(project_path)` returns true and the path does not contain `projects`, the selector calls `filter_by_gitignore` from [`gpt_engineer/core/git.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/core/git.py) to strip out any files that match patterns in the repository's `.gitignore` file. This occurs at the end of the `get_current_files` scan before the list is sorted and returned.

### Can I use the FileSelector programmatically without the interactive UI?

Yes. By setting the environment variable `GPTE_TEST_MODE=1`, you enable test mode where `ask_for_files` skips the interactive editor and reads selections directly from the TOML file at [`metadata/file_selection.toml`](https://github.com/AntonOsika/gpt-engineer/blob/main/metadata/file_selection.toml). This allows headless, scriptable usage of the selector in CI/CD pipelines or automated tests.

### What happens if no files are selected in the TOML?

If `get_files_from_toml` parses the TOML and finds an empty selection list—either because all files were commented out or the `files` mapping is missing—it raises an exception with the message "No files were selected…". This prevents the LLM from receiving an empty context, which would cause the generation to fail.