What Metrics Does evaluator.py Provide for Candidate Performance?

The evaluator.py module in the interviewstreet/hiring-agent repository provides six categories of performance metrics: category scores (open-source, self-projects, production, technical skills), bonus points, deductions, key strengths, areas for improvement, and score boundaries, all structured within an EvaluationData Pydantic model.

The evaluator.py file serves as the core assessment engine in the interviewstreet/hiring-agent project, orchestrating LLM-based resume evaluations. It transforms unstructured resume text into quantifiable performance data by parsing structured responses into the EvaluationData schema defined in main/models.py.

How evaluator.py Structures Candidate Assessments

The evaluation workflow centers on the ResumeEvaluator class in main/evaluator.py, which sends resume content to a language model and returns a structured EvaluationData object. This object encapsulates both quantitative scoring and qualitative feedback, enabling downstream ranking and automated decision-making.

The metrics system operates with defined mathematical boundaries. According to the source code in main/evaluator.py, the system enforces:

• MAX_FINAL_SCORE = 120
• MIN_FINAL_SCORE = -20
• MAX_BONUS_POINTS = 20

Detailed Performance Metrics Breakdown

Category Scores (Technical Competency)

The scores attribute within EvaluationData contains four distinct category scores defined in main/models.py:

• open_source: Evaluates contributions to open-source projects
• self_projects: Assesses independent project work
• production: Measures production-level system experience
• technical_skills: Rates overall technical capability

Each category includes:

• score: Numeric points achieved
• max: Maximum possible points for that category
• evidence: Textual explanation supporting the rating

Bonus Points and Deductions (Adjustment Factors)

Beyond base category scores, evaluator.py captures adjustment factors through two distinct fields:

Bonus Points (bonus_points):

• total: Aggregated extra points (capped at 20)
• breakdown: Textual description of outstanding resume aspects justifying the bonus

Deductions (deductions):

• total: Positive value subtracted from the final score
• reasons: Detailed explanation of missing or weak elements triggering penalties

Qualitative Feedback Dimensions

The evaluation captures narrative insights through two list-based fields defined in main/models.py:

Key Strengths (key_strengths): A curated list of 1–5 standout qualifications identified by the model.

Areas for Improvement (areas_for_improvement): A targeted list of 1–5 development opportunities or gaps detected in the resume.

Score Boundaries and Validation Constants

The main/evaluator.py file defines hard constraints that govern all calculations:

• Final scores cannot exceed 120 points
• Final scores cannot drop below -20 points
• Bonus points are strictly limited to 20 points maximum

These constants ensure consistent evaluation ranges across all candidate assessments.

Accessing Metrics Programmatically

To retrieve these metrics, instantiate ResumeEvaluator and call evaluate_resume():

from evaluator import ResumeEvaluator

# Initialise the evaluator (defaults to the configured LLM model)

evaluator = ResumeEvaluator()

# Provide a plain-text resume

resume_text = """John Doe
Software Engineer with 5 years of experience...
"""

# Run the evaluation

data = evaluator.evaluate_resume(resume_text)

# Access category scores

print("Open-source score:", data.scores.open_source.score, "/", data.scores.open_source.max)
print("Technical-skills score:", data.scores.technical_skills.score)

# Access adjustment factors

print("Bonus points:", data.bonus_points.total)
print("Deductions:", data.deductions.total)

# Access qualitative feedback

print("Key strengths:", data.key_strengths)
print("Improvement areas:", data.areas_for_improvement)

The EvaluationData Pydantic model validates all fields automatically, ensuring type safety for the scores, bonus_points, and deductions nested objects.

Summary

• evaluator.py drives the resume assessment workflow in interviewstreet/hiring-agent by interfacing with language models and structuring responses into quantifiable metrics.
• Six metric categories comprise the evaluation: category scores (four technical dimensions), bonus points, deductions, key strengths, areas for improvement, and enforced score boundaries.
• Score ranges are strictly bounded between -20 and 120, with bonus points capped at 20.
• Structured schemas in main/models.py define EvaluationData, Scores, BonusPoints, and Deductions to ensure consistent data validation.
• Programmatic access occurs through the ResumeEvaluator.evaluate_resume() method, returning a fully populated EvaluationData instance.

Frequently Asked Questions

How is the final candidate score calculated from the individual metrics?

The final score aggregates the four category scores from the scores object, adds the bonus_points.total, and subtracts the deductions.total. According to main/evaluator.py, this calculated value must fall within the MIN_FINAL_SCORE (-20) and MAX_FINAL_SCORE (120) constants, though the specific aggregation logic may apply these bounds as constraints.

What is the difference between category scores and bonus points in evaluator.py?

Category scores (scores) evaluate specific technical competencies (open-source, self-projects, production, technical skills) against defined maximums, while bonus points recognize exceptional achievements outside standard criteria. Deductions function similarly to negative bonus points, penalizing missing elements. This separation allows hiring teams to distinguish between baseline competency and exceptionalism.

Where are the performance metric schemas defined in the hiring-agent repository?

All Pydantic schemas reside in main/models.py. The EvaluationData class serves as the top-level container, containing nested Scores, BonusPoints, and Deductions models. Each schema field includes type validation and documentation, with key_strengths and areas_for_improvement defined as constrained lists in the EvaluationData definition.

What are the minimum and maximum possible scores in the evaluation system?

The main/evaluator.py file explicitly defines MIN_FINAL_SCORE = -20 and MAX_FINAL_SCORE = 120, creating a 140-point dynamic range. Additionally, the MAX_BONUS_POINTS = 20 constant limits discretionary positive adjustments. These boundaries ensure evaluation consistency regardless of resume quality extremes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}