How the Evaluation System Prevents Duplicate Project Selections in interviewstreet/hiring-agent

The evaluation system prevents duplicate project selections through a three-layer defense: explicit LLM instructions, runtime set-based filtering in github.py, and automatic fallback to the original repository list.

The interviewstreet/hiring-agent repository automates technical recruiting by analyzing GitHub repositories to identify candidate projects. When the evaluation pipeline selects the top projects for assessment, ensuring uniqueness is critical to avoid redundant reviews. The system employs a robust strategy to prevent duplicate project selections, combining prompt engineering with deterministic code validation.

Prompt-Level Enforcement

The first layer of defense occurs before the LLM generates any output. In github.py (lines 78-84), the system message explicitly constrains the model's behavior:

chat_params = {
    "model": DEFAULT_MODEL,
    "messages": [
        {
            "role": "system",
            "content": (
                "You are an expert technical recruiter analyzing GitHub repositories "
                "to identify the most impressive projects. CRITICAL: You must select "
                "exactly 7 UNIQUE projects - no duplicates allowed. Each project "
                "must be different from the others."
            ),
        },
        {"role": "user", "content": prompt},
    ],
    "options": model_params,
}

This instruction steers the language model toward generating non-repeating lists. While prompt engineering reduces the likelihood of duplicates, the system does not rely solely on the LLM's compliance.

Runtime De-duplication with Set-Based Filtering

After receiving the LLM response, the code performs deterministic validation. Lines 99-120 in github.py implement a set-based filtering mechanism that guarantees uniqueness regardless of the LLM's output:

selected_projects = json.loads(response_text)

unique_projects = []
seen_names = set()

for project in selected_projects:
    project_name = project.get("name", "")
    if project_name and project_name not in seen_names:
        unique_projects.append(project)
        seen_names.add(project_name)

The seen_names set provides O(1) lookup complexity for each project name check. Only projects whose names have not been recorded are appended to unique_projects, ensuring the final list contains strictly distinct entries.

Fallback Mechanisms for List Completion

If the filtered list contains fewer than the required seven projects, the system triggers cascading fallbacks to reach the target count while maintaining uniqueness.

Primary Fallback: Lines 115-122 supplement the list from the original fetched data:

if len(unique_projects) < 7:
    for project in projects_data:
        if len(unique_projects) >= 7:
            break
        project_name = project.get("name", "")
        if project_name and project_name not in seen_names:
            unique_projects.append(project)
            seen_names.add(project_name)

Final Safety Net: Should the previous steps still yield insufficient projects, lines 135-138 default to the first seven entries from the original repository list:

if len(unique_projects) < 7:
    # guarantees a list of 7 distinct items

    unique_projects = projects_data[:7]

Because projects_data derives from distinct GitHub API responses, this final fallback inherently preserves uniqueness.

Summary

• Prompt engineering instructs the LLM to select exactly seven unique projects via the system message in github.py.
• Set-based validation (seen_names) filters duplicates at runtime with O(1) efficiency after parsing the LLM response.
• Cascading fallbacks first supplement from the original project list, then default to the first seven repositories if needed.
• Source files: The logic resides primarily in github.py (selection and filtering) with prompt templates managed in prompts/template_manager.py.

Frequently Asked Questions

What happens if the LLM returns fewer than seven unique projects?

The system falls back to the original repository list stored in projects_data (lines 115-122), checks each candidate against the seen_names set, and appends distinct projects until reaching seven entries. If this still fails, it defaults to the first seven items from the original data (lines 135-138).

How does the system track which projects have already been selected?

The code maintains a Python set named seen_names during the filtering process. As each project passes validation, its name is added to this set, enabling constant-time lookup to detect duplicates in subsequent iterations.

Is the uniqueness check case-sensitive?

Yes, the implementation uses exact string matching via project.get("name", ""). The comparison project_name not in seen_names respects the exact casing provided by the LLM or GitHub API, treating "MyProject" and "myproject" as distinct entries.

Where is the prompt template defined that enforces uniqueness?

While the critical uniqueness instruction is hardcoded directly in github.py (lines 78-84), the repository also maintains a template management system in prompts/template_manager.py that handles Jinja-based prompt construction for other evaluation scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}