# How to Customize or Add New Evaluation Scoring Categories in the Hiring Agent

> Customize or add new evaluation scoring categories in the Hiring Agent. Learn to extend Pydantic models, update LLM prompts, and propagate fields for console and CSV display.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: how-to-guide
- Published: 2026-07-05

---

**To add a new evaluation scoring category to the hiring agent, extend the `Scores` Pydantic model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), update the LLM prompt template to request the new category, and propagate the field through [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) for console display and [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) for CSV export.**

The interviewstreet/hiring-agent is an open-source résumé evaluation tool that scores candidates across four fixed dimensions: open source contributions, self projects, production experience, and technical skills. When your organization needs to assess additional competencies—such as leadership, communication, or domain expertise—you must modify the data models, prompts, and output formatters to recognize and display the new scoring category.

## Understanding the Fixed Category Architecture

The hiring agent evaluates résumés using a strict schema defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py). The `Scores` class (lines 225-230) currently defines four mandatory fields: `open_source`, `self_projects`, `production`, and `technical_skills`. Each field is a **CategoryScore** object containing `score`, `max`, and `evidence` attributes. The aggregate **EvaluationData** model (lines 44-51) collects these category scores alongside bonus points, deductions, strengths, and improvement areas to produce the final evaluation.

## Step-by-Step Guide to Adding a New Category

To introduce a custom category such as **leadership**, you must synchronize changes across the data layer, LLM interface, and output formatters.

### 1. Extend the Scores Model in models.py

First, add the new category field to the `Scores` class in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py). This ensures Pydantic can validate and parse the LLM's JSON response without field errors.

```python
class Scores(BaseModel):
    open_source: CategoryScore
    self_projects: CategoryScore
    production: CategoryScore
    technical_skills: CategoryScore
    # New custom category

    leadership: CategoryScore

```

### 2. Update the LLM Prompt Template

The LLM must be instructed to generate scores for your new category. Edit the resume evaluation template (managed by [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) and rendered in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py)) to include a section defining the new criteria and expected JSON structure:

```text

### Leadership

Assess the candidate's experience leading teams, initiatives, or open-source communities.
Return a JSON block with:
{
  "score": <0-20>,
  "max": 20,
  "evidence": "<justification>"
}

```

Because [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) parses the LLM response directly into `EvaluationData` using the updated `Scores` model, no code changes are required in the evaluator itself once the template is updated.

### 3. Configure Maximum Score Mapping in score.py

The **category_maxes** dictionary in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) caps category scores to prevent outliers. Add your new category with its maximum possible value (see lines 66-68 for the existing logic):

```python
category_maxes = {
    "open_source": 35,
    "self_projects": 30,
    "production": 25,
    "technical_skills": 10,
    "leadership": 20,  # New category cap

}

```

If your evaluation logic relies on a static total (e.g., 120 points), recalculate the `max_possible_score` variable to include the new category's maximum.

### 4. Render the Category in Console Output

Update the display logic in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (following the pattern at lines 81-90) to print the new category:

```python
if hasattr(evaluation.scores, "leadership") and evaluation.scores.leadership:
    lead_score = evaluation.scores.leadership
    capped_score = min(lead_score.score, category_maxes["leadership"])
    print(f"🦸 Leadership:          {capped_score}/{lead_score.max}")
    print(f"   Evidence: {lead_score.evidence}\n")

```

### 5. Export to CSV via transform.py

Finally, extend the CSV conversion logic in [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) (compare with lines 676-686) to include the new score columns:

```python
if evaluation and hasattr(evaluation, "scores"):
    scores = evaluation.scores
    # Existing fields...

    csv_row["leadership_score"] = scores.leadership.score
    csv_row["leadership_max"] = scores.leadership.max

```

## Summary

- **Data Model**: Add the new `CategoryScore` field to the `Scores` class in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) to enable JSON validation.
- **LLM Prompt**: Extend the evaluation template to request the new category and specify its scoring rubric.
- **Score Capping**: Update the `category_maxes` dictionary in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) to set the category's weight and prevent over-scoring.
- **Display**: Mirror existing console output patterns in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) to render the category with formatted headers and evidence.
- **Export**: Append the new score fields to the CSV row dictionary in [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) for downstream analytics.

## Frequently Asked Questions

### Do I need to modify evaluator.py to parse new categories?

No. The [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) script uses Pydantic's `EvaluationData` model to parse the LLM's JSON response automatically. As long as you added the field to `Scores` in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) and updated the prompt template, the evaluator will ingest the new category without additional code changes.

### How do I adjust the total maximum score when adding a category?

Locate the `max_possible_score` calculation in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (typically around lines 66-68). Add your new category's maximum to this total, or refactor the logic to sum the `category_maxes` dictionary dynamically so the total updates automatically when you add new entries.

### Can I add multiple custom categories at once?

Yes. You can add multiple fields to the `Scores` model simultaneously. Ensure each category has a corresponding entry in `category_maxes`, a dedicated section in the prompt template, and handling logic in both [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) and [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py). Test with a sample résumé to verify all categories appear in the JSON response, console output, and CSV export.

### What happens if the LLM returns an invalid score for the new category?

Pydantic will raise a validation error when parsing the LLM response into the `Scores` model. To handle this gracefully, ensure your prompt explicitly defines the score range and JSON format. You can also wrap the parsing logic in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) with try-except blocks to catch validation errors and log the raw LLM output for debugging.