Main Functionalities of evaluator.py in the Hiring-Agent Repository

The evaluator.py file in the interviewstreet/hiring-agent repository implements the ResumeEvaluator class, which orchestrates LLM-based resume analysis by initializing providers, constructing Jinja prompts, invoking chat completions, and returning structured EvaluationData objects.

The evaluator.py module serves as the central evaluation engine for the open-source hiring-agent project. It encapsulates the complete pipeline for automated resume screening, from provider selection to structured data extraction, functioning as the primary entry point for AI-driven candidate assessment.

Core Responsibilities of the ResumeEvaluator Class

The ResumeEvaluator class defined in evaluator.py handles six critical responsibilities that convert unstructured resume text into machine-readable assessment data.

LLM Provider Initialization

The class initializes the appropriate LLM backend during instantiation. In the __init__ method (lines 24-34), it stores the model name, loads generation parameters including temperature and top_p, creates a TemplateManager instance, and delegates provider selection to _initialize_llm_provider.

The private method _initialize_llm_provider (lines 36-39) calls initialize_llm_provider from llm_utils.py, which maps the requested model to either Ollama or Gemini based on the MODEL_PROVIDER_MAPPING configuration.

Jinja Template-Based Prompt Construction

The evaluator constructs prompts using Jinja2 templates. The _load_evaluation_prompt method (lines 40-46) renders the resume_evaluation_criteria.jinja template, injecting the raw resume text into the evaluation rubric.

Additionally, the system-level instructions are loaded via template_manager.render_template("resume_evaluation_system_message"), which references resume_evaluation_system_message.jinja. This dual-template approach separates the scoring criteria from behavioral instructions sent to the LLM.

Structured Chat Invocation

The evaluate_resume method (lines 48-88) orchestrates the actual LLM call. It builds a chat_params dictionary containing generation options and appends a format argument specifying EvaluationData.model_json_schema(). This schema constraint instructs the LLM to return JSON matching the EvaluationData structure defined in models.py.

Response Parsing and Validation

After receiving the raw LLM response, the evaluator processes it through extract_json_from_response (imported from llm_utils.py), which strips markdown fences and code blocks. The cleaned JSON is parsed with json.loads and instantiated as an EvaluationData Pydantic model, ensuring type safety and schema validation before returning to callers.

Step-by-Step Execution Flow in evaluator.py

Understanding the internal workflow clarifies how the repository processes candidate resumes.

Constructor and Provider Setup

When instantiating ResumeEvaluator(model_name="llama2"), the constructor stores configuration parameters and triggers provider initialization. The system supports multiple backends through the abstraction layer in llm_utils.py, allowing seamless switching between local Ollama instances and Google Gemini API endpoints.

Prompt Generation Pipeline

The evaluation workflow renders two distinct templates:

1. System message: Loaded from resume_evaluation_system_message.jinja to establish the LLM's role and output constraints
2. User prompt: Generated by _load_evaluation_prompt using resume_evaluation_criteria.jinja combined with the candidate's resume text

Evaluation Orchestration

The evaluate_resume method executes the complete pipeline:

1. Assembles message lists with system and user content
2. Configures generation parameters (temperature, top_p)
3. Attaches the JSON schema format hint
4. Invokes the provider's chat completion interface
5. Processes the response through the JSON extraction utility
6. Returns a validated EvaluationData instance

Practical Usage Example

The following implementation demonstrates how to consume evaluator.py in a production context:

from evaluator import ResumeEvaluator

# Initialize with default or specified model

evaluator = ResumeEvaluator()

# Load raw resume content

with open("candidate_resume.txt", "r") as f:
    raw_resume = f.read()

# Execute evaluation pipeline

try:
    result = evaluator.evaluate_resume(raw_resume)
    print(f"Technical Score: {result.scores.technical}")
    print(f"Key Strengths: {result.key_strengths}")
    print(f"Improvement Areas: {result.areas_for_improvement}")
except Exception as exc:
    print(f"Evaluation failed: {exc}")

This pattern returns an EvaluationData object containing structured scores, narrative feedback, and quantitative metrics ready for downstream ATS integration or UI rendering.

Integration with Supporting Components

The evaluator.py module operates within a broader architecture that includes:

• models.py: Defines the EvaluationData Pydantic schema (lines 44-51) that guarantees type-safe evaluation output
• llm_utils.py: Provides initialize_llm_provider for backend abstraction and extract_json_from_response for response cleaning (lines 13-37)
• prompt.py: Contains MODEL_PROVIDER_MAPPING and default model configurations used during initialization
• prompts/template_manager.py: Handles Jinja template loading and rendering logic
• prompts/templates/: Houses resume_evaluation_criteria.jinja and resume_evaluation_system_message.jinja used for prompt construction

These dependencies enable evaluator.py to maintain clean separation between orchestration logic, provider implementations, and prompt engineering.

Summary

• evaluator.py implements the ResumeEvaluator class, which serves as the primary orchestration engine for resume evaluation in the hiring-agent repository.
• The module handles LLM provider initialization (Ollama or Gemini), Jinja2 template rendering, and JSON-schema-validated response parsing.
• Key methods include __init__ (lines 24-34), _initialize_llm_provider (lines 36-39), _load_evaluation_prompt (lines 40-46), and evaluate_resume (lines 48-88).
• The class returns strongly-typed EvaluationData objects defined in models.py, ensuring downstream consumers receive validated structured data.
• Integration with llm_utils.py and the templates directory provides the abstraction layers necessary for multi-provider support and prompt management.

Frequently Asked Questions

What is the primary role of evaluator.py in the hiring-agent repository?

The file implements the core evaluation logic that transforms unstructured resume text into structured assessment data. It serves as the main entry point for AI-driven resume screening, coordinating between LLM providers, prompt templates, and data validation schemas to produce consistent, machine-readable evaluation reports.

How does ResumeEvaluator ensure the LLM returns valid JSON?

The evaluate_resume method appends EvaluationData.model_json_schema() to the chat parameters as a format hint, instructing the LLM to conform to the schema. After receiving the response, it processes the text through extract_json_from_response from llm_utils.py to remove markdown wrappers, then validates the parsed dictionary against the EvaluationData Pydantic model before returning.

Which LLM providers are supported by evaluator.py?

The module supports Ollama for local model execution and Gemini for Google Cloud AI integration. Provider selection occurs in _initialize_llm_provider based on the MODEL_PROVIDER_MAPPING dictionary defined in prompt.py, enabling seamless switching between local and cloud-based inference depending on the model name specified during instantiation.

What templates does evaluator.py use to evaluate resumes?

The class utilizes two Jinja templates located in the prompts/templates/ directory: resume_evaluation_criteria.jinja defines the scoring rubric and evaluation criteria, while resume_evaluation_system_message.jinja provides system-level instructions that guide the LLM's tone, formatting requirements, and behavioral constraints during the evaluation process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}