How to Customize or Add New Evaluation Scoring Categories in the Hiring Agent

To add a new evaluation scoring category to the hiring agent, extend the Scores Pydantic model in models.py, update the LLM prompt template to request the new category, and propagate the field through score.py for console display and transform.py for CSV export.

The interviewstreet/hiring-agent is an open-source résumé evaluation tool that scores candidates across four fixed dimensions: open source contributions, self projects, production experience, and technical skills. When your organization needs to assess additional competencies—such as leadership, communication, or domain expertise—you must modify the data models, prompts, and output formatters to recognize and display the new scoring category.

Understanding the Fixed Category Architecture

The hiring agent evaluates résumés using a strict schema defined in models.py. The Scores class (lines 225-230) currently defines four mandatory fields: open_source, self_projects, production, and technical_skills. Each field is a CategoryScore object containing score, max, and evidence attributes. The aggregate EvaluationData model (lines 44-51) collects these category scores alongside bonus points, deductions, strengths, and improvement areas to produce the final evaluation.

Step-by-Step Guide to Adding a New Category

To introduce a custom category such as leadership, you must synchronize changes across the data layer, LLM interface, and output formatters.

1. Extend the Scores Model in models.py

First, add the new category field to the Scores class in models.py. This ensures Pydantic can validate and parse the LLM's JSON response without field errors.

class Scores(BaseModel):
    open_source: CategoryScore
    self_projects: CategoryScore
    production: CategoryScore
    technical_skills: CategoryScore
    # New custom category

    leadership: CategoryScore

2. Update the LLM Prompt Template

The LLM must be instructed to generate scores for your new category. Edit the resume evaluation template (managed by prompts/template_manager.py and rendered in evaluator.py) to include a section defining the new criteria and expected JSON structure:

### Leadership

Assess the candidate's experience leading teams, initiatives, or open-source communities.
Return a JSON block with:
{
  "score": <0-20>,
  "max": 20,
  "evidence": "<justification>"
}

Because evaluator.py parses the LLM response directly into EvaluationData using the updated Scores model, no code changes are required in the evaluator itself once the template is updated.

3. Configure Maximum Score Mapping in score.py

The category_maxes dictionary in score.py caps category scores to prevent outliers. Add your new category with its maximum possible value (see lines 66-68 for the existing logic):

category_maxes = {
    "open_source": 35,
    "self_projects": 30,
    "production": 25,
    "technical_skills": 10,
    "leadership": 20,  # New category cap

}

If your evaluation logic relies on a static total (e.g., 120 points), recalculate the max_possible_score variable to include the new category's maximum.

4. Render the Category in Console Output

Update the display logic in score.py (following the pattern at lines 81-90) to print the new category:

if hasattr(evaluation.scores, "leadership") and evaluation.scores.leadership:
    lead_score = evaluation.scores.leadership
    capped_score = min(lead_score.score, category_maxes["leadership"])
    print(f"🦸 Leadership:          {capped_score}/{lead_score.max}")
    print(f"   Evidence: {lead_score.evidence}\n")

5. Export to CSV via transform.py

Finally, extend the CSV conversion logic in transform.py (compare with lines 676-686) to include the new score columns:

if evaluation and hasattr(evaluation, "scores"):
    scores = evaluation.scores
    # Existing fields...

    csv_row["leadership_score"] = scores.leadership.score
    csv_row["leadership_max"] = scores.leadership.max

Summary

• Data Model: Add the new CategoryScore field to the Scores class in models.py to enable JSON validation.
• LLM Prompt: Extend the evaluation template to request the new category and specify its scoring rubric.
• Score Capping: Update the category_maxes dictionary in score.py to set the category's weight and prevent over-scoring.
• Display: Mirror existing console output patterns in score.py to render the category with formatted headers and evidence.
• Export: Append the new score fields to the CSV row dictionary in transform.py for downstream analytics.

Frequently Asked Questions

Do I need to modify evaluator.py to parse new categories?

No. The evaluator.py script uses Pydantic's EvaluationData model to parse the LLM's JSON response automatically. As long as you added the field to Scores in models.py and updated the prompt template, the evaluator will ingest the new category without additional code changes.

How do I adjust the total maximum score when adding a category?

Locate the max_possible_score calculation in score.py (typically around lines 66-68). Add your new category's maximum to this total, or refactor the logic to sum the category_maxes dictionary dynamically so the total updates automatically when you add new entries.

Can I add multiple custom categories at once?

Yes. You can add multiple fields to the Scores model simultaneously. Ensure each category has a corresponding entry in category_maxes, a dedicated section in the prompt template, and handling logic in both score.py and transform.py. Test with a sample résumé to verify all categories appear in the JSON response, console output, and CSV export.

What happens if the LLM returns an invalid score for the new category?

Pydantic will raise a validation error when parsing the LLM response into the Scores model. To handle this gracefully, ensure your prompt explicitly defines the score range and JSON format. You can also wrap the parsing logic in evaluator.py with try-except blocks to catch validation errors and log the raw LLM output for debugging.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}