# How evaluator.py Assesses Resumes: Inside the ResumeEvaluator Logic

> Explore the inner workings of evaluator.py and learn how the ResumeEvaluator class assesses resumes using LLM prompts, Pydantic validation, and a structured output pipeline.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: deep-dive
- Published: 2026-07-15

---

**The `ResumeEvaluator` class in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) orchestrates LLM-based resume assessment by constructing dynamic prompts, enforcing structured JSON output via Pydantic schemas, and validating results through a multi-step pipeline.**

The [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) module in the `interviewstreet/hiring-agent` repository contains the core assessment logic that drives automated candidate screening. This module implements a deterministic pipeline that combines prompt engineering, provider abstraction, and schema validation to produce consistent, type-safe scoring results.

## Prompt Construction and Template Rendering

The assessment logic begins with sophisticated prompt assembly. The `_load_evaluation_prompt` method embeds the full resume text into a criteria prompt template, while a separate system prompt (`resume_evaluation_system_message`) establishes the evaluation context.

According to the source code at lines 40–46 and 53–59, these templates live in [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) and are rendered at runtime using Jinja-style templating. This separation of concerns allows the evaluator to inject candidate-specific content into standardized evaluation frameworks without hardcoding prompts.

## LLM Provider Selection and Initialization

Before making any inference calls, the evaluator initializes the appropriate backend. The `initialize_llm_provider` function (lines 36–39) instantiates the correct LLM client based on the `model_name` parameter passed during `ResumeEvaluator` initialization.

This abstraction layer allows the assessment logic to remain provider-agnostic, whether you are using OpenAI, Anthropic, or other supported backends.

## Structured Chat Payload Assembly

The core inference request is constructed at lines 61–73. The evaluator builds a chat payload containing:

- The selected **model name**
- The **system message** (the rendered `resume_evaluation_system_message`)
- The **user message** (the rendered resume criteria prompt with embedded resume text)
- Generation options including `temperature` and `top_p` (defaulted from `MODEL_PARAMETERS`)

This structured approach ensures that every evaluation request carries consistent generation parameters, reducing variability in scoring.

## JSON Schema Enforcement for Structured Output

A critical component of the assessment logic occurs at lines 75–77. Rather than parsing free-text responses, the evaluator enforces strict output formatting by supplying `EvaluationData.model_json_schema()` as the `format` argument to the LLM provider.

The `EvaluationData` model (defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)) acts as a contract, specifying exactly which fields (scores, comments, criteria assessments) the LLM must return. This schema validation eliminates ambiguity and ensures downstream code receives predictable data structures.

## Response Extraction and Type Validation

After the LLM returns data, the pipeline executes a three-stage validation process (lines 80–86):

1. **Extraction**: `extract_json_from_response` (from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)) cleans the raw LLM output, handling potential markdown code blocks or extraneous text.
2. **Parsing**: The cleaned string passes through `json.loads` to produce a Python dictionary.
3. **Typing**: The dictionary is passed to the `EvaluationData` Pydantic model, which validates field types and constraints, returning a strongly-typed object representing the resume scores.

## Error Handling and Observability

The assessment logic includes comprehensive error handling at lines 89–91. Any exception during prompt loading, LLM inference, or JSON parsing is logged and re-raised, ensuring that failures are visible to calling applications rather than returning silent null values.

## Practical Implementation Examples

### Basic Resume Evaluation

```python
from evaluator import ResumeEvaluator

resume_text = "John Doe\nSoftware Engineer with 5 years of experience..."
evaluator = ResumeEvaluator(model_name="gpt-4o-mini")
result = evaluator.evaluate_resume(resume_text)

print(result)          # Pydantic model with scores, comments, etc.

```

### Customizing Model Parameters

You can override default generation parameters by passing a custom configuration dictionary:

```python
custom_params = {"temperature": 0.2, "top_p": 0.95}
evaluator = ResumeEvaluator(model_name="gpt-4o-mini", model_params=custom_params)

```

## Key Supporting Components

The assessment logic in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) relies on several supporting modules:

- **[`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py)**: Handles loading and rendering of Jinja-style prompt templates (`resume_evaluation_criteria`, `resume_evaluation_system_message`)
- **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)**: Provides `initialize_llm_provider` and `extract_json_from_response` utilities
- **[`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)**: Defines the `EvaluationData` Pydantic model used for schema validation

## Summary

- **evaluator.py** implements a deterministic, multi-step assessment pipeline through the `ResumeEvaluator` class.
- The logic uses **template-based prompt construction** (lines 40–59) to embed resume content into standardized evaluation frameworks.
- **Structured output enforcement** via `EvaluationData.model_json_schema()` (lines 75–77) guarantees type-safe responses.
- The **provider abstraction layer** (`initialize_llm_provider`) makes the system model-agnostic.
- **Three-stage validation** (extraction, parsing, Pydantic validation) ensures data integrity before returning results.

## Frequently Asked Questions

### How does evaluator.py ensure consistent scoring across different LLM providers?

The assessment logic enforces consistency through the `EvaluationData` Pydantic schema, which is passed to every provider as a JSON format requirement. By constraining the LLM to return specific fields with defined types (lines 75–77), the system normalizes outputs regardless of whether you use GPT-4, Claude, or other supported models. Additionally, standardized `temperature` and `top_p` parameters from `MODEL_PARAMETERS` reduce variability in generation.

### What happens if the LLM returns malformed JSON or invalid scores?

The `extract_json_from_response` function in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) first attempts to clean and extract valid JSON from the raw text. If extraction or subsequent `json.loads` parsing fails, or if the data fails Pydantic validation against the `EvaluationData` model (lines 80–91), the exception is logged and re-raised. This fail-fast approach ensures that calling code receives only validated, complete assessment data.

### Can I customize the evaluation criteria in evaluator.py without modifying the source code?

Yes. The evaluation criteria are defined in template files loaded by [`template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/template_manager.py), not hardcoded in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py). The `_load_evaluation_prompt` method references `resume_evaluation_criteria` and `resume_evaluation_system_message` templates. By modifying these template files or extending the template manager to load custom variants, you can adjust scoring criteria without altering the core assessment logic in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py).

### Which model parameters does ResumeEvaluator use by default?

By default, the evaluator inherits parameters from `MODEL_PARAMETERS`, typically including `temperature` and `top_p` settings optimized for consistent evaluation. These are passed during the chat payload construction at lines 61–73. You can override these defaults by passing a custom `model_params` dictionary when instantiating `ResumeEvaluator`, as shown in the implementation examples above.