# Main Functionalities of evaluator.py in the Hiring-Agent Repository

> Explore evaluator.py in hiring-agent: orchestrates LLM resume analysis, initializes providers, crafts prompts, invokes chat completions, and returns structured EvaluationData for efficient hiring.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: deep-dive
- Published: 2026-07-15

---

**The evaluator.py file in the interviewstreet/hiring-agent repository implements the ResumeEvaluator class, which orchestrates LLM-based resume analysis by initializing providers, constructing Jinja prompts, invoking chat completions, and returning structured EvaluationData objects.**

The [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) module serves as the central evaluation engine for the open-source hiring-agent project. It encapsulates the complete pipeline for automated resume screening, from provider selection to structured data extraction, functioning as the primary entry point for AI-driven candidate assessment.

## Core Responsibilities of the ResumeEvaluator Class

The `ResumeEvaluator` class defined in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) handles six critical responsibilities that convert unstructured resume text into machine-readable assessment data.

### LLM Provider Initialization

The class initializes the appropriate LLM backend during instantiation. In the `__init__` method ([lines 24-34](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py#L24-L34)), it stores the model name, loads generation parameters including `temperature` and `top_p`, creates a `TemplateManager` instance, and delegates provider selection to `_initialize_llm_provider`.

The private method `_initialize_llm_provider` ([lines 36-39](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py#L36-L39)) calls `initialize_llm_provider` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), which maps the requested model to either **Ollama** or **Gemini** based on the `MODEL_PROVIDER_MAPPING` configuration.

### Jinja Template-Based Prompt Construction

The evaluator constructs prompts using Jinja2 templates. The `_load_evaluation_prompt` method ([lines 40-46](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py#L40-L46)) renders the `resume_evaluation_criteria.jinja` template, injecting the raw resume text into the evaluation rubric.

Additionally, the system-level instructions are loaded via `template_manager.render_template("resume_evaluation_system_message")`, which references `resume_evaluation_system_message.jinja`. This dual-template approach separates the scoring criteria from behavioral instructions sent to the LLM.

### Structured Chat Invocation

The `evaluate_resume` method ([lines 48-88](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py#L48-L88)) orchestrates the actual LLM call. It builds a `chat_params` dictionary containing generation options and appends a `format` argument specifying `EvaluationData.model_json_schema()`. This schema constraint instructs the LLM to return JSON matching the `EvaluationData` structure defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py).

### Response Parsing and Validation

After receiving the raw LLM response, the evaluator processes it through `extract_json_from_response` (imported from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)), which strips markdown fences and code blocks. The cleaned JSON is parsed with `json.loads` and instantiated as an `EvaluationData` Pydantic model, ensuring type safety and schema validation before returning to callers.

## Step-by-Step Execution Flow in evaluator.py

Understanding the internal workflow clarifies how the repository processes candidate resumes.

### Constructor and Provider Setup

When instantiating `ResumeEvaluator(model_name="llama2")`, the constructor stores configuration parameters and triggers provider initialization. The system supports multiple backends through the abstraction layer in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), allowing seamless switching between local Ollama instances and Google Gemini API endpoints.

### Prompt Generation Pipeline

The evaluation workflow renders two distinct templates:
1. **System message**: Loaded from `resume_evaluation_system_message.jinja` to establish the LLM's role and output constraints
2. **User prompt**: Generated by `_load_evaluation_prompt` using `resume_evaluation_criteria.jinja` combined with the candidate's resume text

### Evaluation Orchestration

The `evaluate_resume` method executes the complete pipeline:
1. Assembles message lists with system and user content
2. Configures generation parameters (temperature, top_p)
3. Attaches the JSON schema format hint
4. Invokes the provider's chat completion interface
5. Processes the response through the JSON extraction utility
6. Returns a validated `EvaluationData` instance

## Practical Usage Example

The following implementation demonstrates how to consume [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) in a production context:

```python
from evaluator import ResumeEvaluator

# Initialize with default or specified model

evaluator = ResumeEvaluator()

# Load raw resume content

with open("candidate_resume.txt", "r") as f:
    raw_resume = f.read()

# Execute evaluation pipeline

try:
    result = evaluator.evaluate_resume(raw_resume)
    print(f"Technical Score: {result.scores.technical}")
    print(f"Key Strengths: {result.key_strengths}")
    print(f"Improvement Areas: {result.areas_for_improvement}")
except Exception as exc:
    print(f"Evaluation failed: {exc}")

```

This pattern returns an `EvaluationData` object containing structured scores, narrative feedback, and quantitative metrics ready for downstream ATS integration or UI rendering.

## Integration with Supporting Components

The [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) module operates within a broader architecture that includes:

- **[`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)**: Defines the `EvaluationData` Pydantic schema ([lines 44-51](https://github.com/interviewstreet/hiring-agent/blob/main/models.py#L44-L51)) that guarantees type-safe evaluation output
- **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)**: Provides `initialize_llm_provider` for backend abstraction and `extract_json_from_response` for response cleaning ([lines 13-37](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py#L13-L37))
- **[`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py)**: Contains `MODEL_PROVIDER_MAPPING` and default model configurations used during initialization
- **[`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py)**: Handles Jinja template loading and rendering logic
- **`prompts/templates/`**: Houses `resume_evaluation_criteria.jinja` and `resume_evaluation_system_message.jinja` used for prompt construction

These dependencies enable [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) to maintain clean separation between orchestration logic, provider implementations, and prompt engineering.

## Summary

- **evaluator.py** implements the `ResumeEvaluator` class, which serves as the primary orchestration engine for resume evaluation in the hiring-agent repository.
- The module handles LLM provider initialization (Ollama or Gemini), Jinja2 template rendering, and JSON-schema-validated response parsing.
- Key methods include `__init__` (lines 24-34), `_initialize_llm_provider` (lines 36-39), `_load_evaluation_prompt` (lines 40-46), and `evaluate_resume` (lines 48-88).
- The class returns strongly-typed `EvaluationData` objects defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), ensuring downstream consumers receive validated structured data.
- Integration with [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) and the templates directory provides the abstraction layers necessary for multi-provider support and prompt management.

## Frequently Asked Questions

### What is the primary role of evaluator.py in the hiring-agent repository?

The file implements the core evaluation logic that transforms unstructured resume text into structured assessment data. It serves as the main entry point for AI-driven resume screening, coordinating between LLM providers, prompt templates, and data validation schemas to produce consistent, machine-readable evaluation reports.

### How does ResumeEvaluator ensure the LLM returns valid JSON?

The `evaluate_resume` method appends `EvaluationData.model_json_schema()` to the chat parameters as a format hint, instructing the LLM to conform to the schema. After receiving the response, it processes the text through `extract_json_from_response` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) to remove markdown wrappers, then validates the parsed dictionary against the `EvaluationData` Pydantic model before returning.

### Which LLM providers are supported by evaluator.py?

The module supports **Ollama** for local model execution and **Gemini** for Google Cloud AI integration. Provider selection occurs in `_initialize_llm_provider` based on the `MODEL_PROVIDER_MAPPING` dictionary defined in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py), enabling seamless switching between local and cloud-based inference depending on the model name specified during instantiation.

### What templates does evaluator.py use to evaluate resumes?

The class utilizes two Jinja templates located in the `prompts/templates/` directory: `resume_evaluation_criteria.jinja` defines the scoring rubric and evaluation criteria, while `resume_evaluation_system_message.jinja` provides system-level instructions that guide the LLM's tone, formatting requirements, and behavioral constraints during the evaluation process.