How Evaluation Data Validation Works with Pydantic in the Hiring-Agent Repository

The hiring-agent repository validates evaluation results using a strict Pydantic EvaluationData model that automatically enforces type safety, numeric constraints, and collection size limits on fields like scores, bonus_points, and key_strengths.

The interviewstreet/hiring-agent repository implements rigorous evaluation data validation to ensure that candidate assessments meet strict structural requirements before processing. By defining a comprehensive EvaluationData schema in main/models.py, the application leverages Pydantic to automatically validate incoming data from LLM responses or external services. This validation layer acts as a gatekeeper that prevents malformed evaluation data from propagating through the hiring pipeline.

The EvaluationData Model Architecture

In main/models.py, the EvaluationData class serves as the root validation schema for all evaluation results. This Pydantic model defines the exact structure expected for candidate assessments, utilizing nested models to validate complex hierarchical data.

Nested Model Definitions

The EvaluationData model relies on several specialized Pydantic models to validate specific components:

• Scores: Contains four required CategoryScore fields (open_source, self_projects, production, technical_skills)
• CategoryScore: Validates individual score entries with numeric bounds and evidence strings
• BonusPoints: Enforces non-negative totals with a maximum cap and required breakdown descriptions
• Deductions: Ensures deduction totals are non-negative and accompanied by explanatory reasons

Validation Constraints and Business Rules

Pydantic automatically validates each field against Python type hints and constraints declared with the Field function. The hiring-agent repository implements specific business logic through these constraint definitions.

Numeric Range Validation

The bonus_points field enforces a total value between 0 and 20 using ge=0 and le=20 constraints. Similarly, the deductions model requires total ≥ 0. These validations ensure that final scores remain within acceptable business ranges and prevent negative adjustments from entering the system.

Collection Size Limits

For qualitative feedback, both key_strengths and areas_for_improvement are validated as List[str] with min_items=1 and max_items=5. This prevents empty feedback submissions while keeping responses concise and focused on the most critical points.

Validation in Practice

When the application receives evaluation data—typically from an LLM response or external service—it instantiates the EvaluationData model directly from the raw JSON payload.

Valid Data Example

from main.models import EvaluationData, Scores, CategoryScore, BonusPoints, Deductions

# Example of building a valid evaluation payload

payload = {
    "scores": {
        "open_source": {"score": 8.5, "max": 10, "evidence": "Contributed to 3 repos"},
        "self_projects": {"score": 7.0, "max": 10, "evidence": "Built a personal API"},
        "production": {"score": 9.0, "max": 10, "evidence": "Deployed at scale"},
        "technical_skills": {"score": 8.0, "max": 10, "evidence": "Strong Python/SQL"},
    },
    "bonus_points": {"total": 5.0, "breakdown": "Open‑source contributions"},
    "deductions": {"total": 2.0, "reasons": "Minor style issues"},
    "key_strengths": ["Problem solving", "Team collaboration"],
    "areas_for_improvement": ["Testing coverage", "Documentation"],
}

# Pydantic validates on instantiation; raises ValidationError on bad data

evaluation = EvaluationData(**payload)
print(evaluation.json(indent=2))

Handling ValidationError Exceptions

When validation fails, Pydantic raises a ValidationError that captures all constraint violations. The calling code in main/evaluator.py can catch these exceptions to prevent malformed data from being stored or processed further.

from pydantic import ValidationError
from main.models import EvaluationData

bad_payload = {
    "scores": {},                     # missing required categories -> ValidationError

    "bonus_points": {"total": -1},    # total < 0 violates ge=0 -> ValidationError

    "deductions": {"total": -5},
    "key_strengths": [],              # empty list fails min_items=1 -> ValidationError

    "areas_for_improvement": ["A"] * 6,  # 6 items exceeds max_items=5 -> ValidationError

}

try:
    EvaluationData(**bad_payload)
except ValidationError as e:
    print(e)

Integration with the Evaluation Pipeline

The main/evaluator.py module consumes validated EvaluationData instances when scoring candidates. By relying on Pydantic's validation guarantees, the evaluator can safely access nested attributes without defensive type checking, knowing that all CategoryScore objects contain valid numeric scores and required evidence strings.

Summary

• The EvaluationData model in main/models.py defines the complete schema for evaluation data validation using Pydantic
• Field constraints using ge, le, min_items, and max_items enforce business rules on numeric ranges and list sizes
• Nested models (Scores, BonusPoints, Deductions) validate complex hierarchical data structures with strict type checking
• Pydantic raises ValidationError immediately on instantiation, preventing malformed evaluation data from entering the processing pipeline
• The main/evaluator.py module relies on these validation guarantees to safely process candidate assessments without additional defensive coding

Frequently Asked Questions

What triggers the validation in the hiring-agent repository?

Validation occurs when the EvaluationData model is instantiated from raw data, typically when receiving JSON responses from LLMs or external evaluation services. Pydantic checks all fields, types, and constraints during this initialization process according to the definitions in main/models.py. Any violation immediately raises a ValidationError before the data reaches business logic.

How does the validation prevent invalid bonus points from being accepted?

The BonusPoints model uses Field(ge=0, le=20) to constrain the total field. If a payload contains a negative value or exceeds 20 points, Pydantic raises a ValidationError immediately, preventing the invalid evaluation data from reaching the scoring logic in main/evaluator.py.

Can the validation handle partial evaluation data or missing fields?

No, the EvaluationData model requires all specified fields including the four category scores in the scores object. Missing required fields trigger validation errors. This strictness ensures that main/evaluator.py always receives complete assessment data with all required CategoryScore entries present.

What happens when list constraints are violated for strengths or improvements?

If key_strengths or areas_for_improvement contain fewer than one item or more than five items, Pydantic raises a ValidationError citing the min_items or max_items constraint violation. This enforces concise feedback while ensuring at least one strength and improvement area is always documented.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}