# How the Evaluation System Prevents Duplicate Project Selections in interviewstreet/hiring-agent

> Discover how the evaluation system in interviewstreet/hiring-agent prevents duplicate project selections using a three-layer defense: LLM instructions, runtime filtering, and fallback mechanisms.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: internals
- Published: 2026-07-19

---

**The evaluation system prevents duplicate project selections through a three-layer defense: explicit LLM instructions, runtime set-based filtering in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py), and automatic fallback to the original repository list.**

The interviewstreet/hiring-agent repository automates technical recruiting by analyzing GitHub repositories to identify candidate projects. When the evaluation pipeline selects the top projects for assessment, ensuring uniqueness is critical to avoid redundant reviews. The system employs a robust strategy to prevent duplicate project selections, combining prompt engineering with deterministic code validation.

## Prompt-Level Enforcement

The first layer of defense occurs before the LLM generates any output. In [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) (lines 78-84), the system message explicitly constrains the model's behavior:

```python
chat_params = {
    "model": DEFAULT_MODEL,
    "messages": [
        {
            "role": "system",
            "content": (
                "You are an expert technical recruiter analyzing GitHub repositories "
                "to identify the most impressive projects. CRITICAL: You must select "
                "exactly 7 UNIQUE projects - no duplicates allowed. Each project "
                "must be different from the others."
            ),
        },
        {"role": "user", "content": prompt},
    ],
    "options": model_params,
}

```

This instruction steers the language model toward generating non-repeating lists. While prompt engineering reduces the likelihood of duplicates, the system does not rely solely on the LLM's compliance.

## Runtime De-duplication with Set-Based Filtering

After receiving the LLM response, the code performs deterministic validation. Lines 99-120 in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) implement a **set-based filtering mechanism** that guarantees uniqueness regardless of the LLM's output:

```python
selected_projects = json.loads(response_text)

unique_projects = []
seen_names = set()

for project in selected_projects:
    project_name = project.get("name", "")
    if project_name and project_name not in seen_names:
        unique_projects.append(project)
        seen_names.add(project_name)

```

The `seen_names` set provides **O(1) lookup complexity** for each project name check. Only projects whose names have not been recorded are appended to `unique_projects`, ensuring the final list contains strictly distinct entries.

## Fallback Mechanisms for List Completion

If the filtered list contains fewer than the required seven projects, the system triggers cascading fallbacks to reach the target count while maintaining uniqueness.

**Primary Fallback:** Lines 115-122 supplement the list from the original fetched data:

```python
if len(unique_projects) < 7:
    for project in projects_data:
        if len(unique_projects) >= 7:
            break
        project_name = project.get("name", "")
        if project_name and project_name not in seen_names:
            unique_projects.append(project)
            seen_names.add(project_name)

```

**Final Safety Net:** Should the previous steps still yield insufficient projects, lines 135-138 default to the first seven entries from the original repository list:

```python
if len(unique_projects) < 7:
    # guarantees a list of 7 distinct items

    unique_projects = projects_data[:7]

```

Because `projects_data` derives from distinct GitHub API responses, this final fallback inherently preserves uniqueness.

## Summary

- **Prompt engineering** instructs the LLM to select exactly seven unique projects via the system message in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py).
- **Set-based validation** (`seen_names`) filters duplicates at runtime with O(1) efficiency after parsing the LLM response.
- **Cascading fallbacks** first supplement from the original project list, then default to the first seven repositories if needed.
- **Source files**: The logic resides primarily in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) (selection and filtering) with prompt templates managed in [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py).

## Frequently Asked Questions

### What happens if the LLM returns fewer than seven unique projects?

The system falls back to the original repository list stored in `projects_data` (lines 115-122), checks each candidate against the `seen_names` set, and appends distinct projects until reaching seven entries. If this still fails, it defaults to the first seven items from the original data (lines 135-138).

### How does the system track which projects have already been selected?

The code maintains a Python set named `seen_names` during the filtering process. As each project passes validation, its name is added to this set, enabling constant-time lookup to detect duplicates in subsequent iterations.

### Is the uniqueness check case-sensitive?

Yes, the implementation uses exact string matching via `project.get("name", "")`. The comparison `project_name not in seen_names` respects the exact casing provided by the LLM or GitHub API, treating "MyProject" and "myproject" as distinct entries.

### Where is the prompt template defined that enforces uniqueness?

While the critical uniqueness instruction is hardcoded directly in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) (lines 78-84), the repository also maintains a template management system in [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) that handles Jinja-based prompt construction for other evaluation scenarios.