How evaluator.py Assesses Resumes: Inside the ResumeEvaluator Logic

The ResumeEvaluator class in evaluator.py orchestrates LLM-based resume assessment by constructing dynamic prompts, enforcing structured JSON output via Pydantic schemas, and validating results through a multi-step pipeline.

The evaluator.py module in the interviewstreet/hiring-agent repository contains the core assessment logic that drives automated candidate screening. This module implements a deterministic pipeline that combines prompt engineering, provider abstraction, and schema validation to produce consistent, type-safe scoring results.

Prompt Construction and Template Rendering

The assessment logic begins with sophisticated prompt assembly. The _load_evaluation_prompt method embeds the full resume text into a criteria prompt template, while a separate system prompt (resume_evaluation_system_message) establishes the evaluation context.

According to the source code at lines 40–46 and 53–59, these templates live in prompts/template_manager.py and are rendered at runtime using Jinja-style templating. This separation of concerns allows the evaluator to inject candidate-specific content into standardized evaluation frameworks without hardcoding prompts.

LLM Provider Selection and Initialization

Before making any inference calls, the evaluator initializes the appropriate backend. The initialize_llm_provider function (lines 36–39) instantiates the correct LLM client based on the model_name parameter passed during ResumeEvaluator initialization.

This abstraction layer allows the assessment logic to remain provider-agnostic, whether you are using OpenAI, Anthropic, or other supported backends.

Structured Chat Payload Assembly

The core inference request is constructed at lines 61–73. The evaluator builds a chat payload containing:

• The selected model name
• The system message (the rendered resume_evaluation_system_message)
• The user message (the rendered resume criteria prompt with embedded resume text)
• Generation options including temperature and top_p (defaulted from MODEL_PARAMETERS)

This structured approach ensures that every evaluation request carries consistent generation parameters, reducing variability in scoring.

JSON Schema Enforcement for Structured Output

A critical component of the assessment logic occurs at lines 75–77. Rather than parsing free-text responses, the evaluator enforces strict output formatting by supplying EvaluationData.model_json_schema() as the format argument to the LLM provider.

The EvaluationData model (defined in models.py) acts as a contract, specifying exactly which fields (scores, comments, criteria assessments) the LLM must return. This schema validation eliminates ambiguity and ensures downstream code receives predictable data structures.

Response Extraction and Type Validation

After the LLM returns data, the pipeline executes a three-stage validation process (lines 80–86):

1. Extraction: extract_json_from_response (from llm_utils.py) cleans the raw LLM output, handling potential markdown code blocks or extraneous text.
2. Parsing: The cleaned string passes through json.loads to produce a Python dictionary.
3. Typing: The dictionary is passed to the EvaluationData Pydantic model, which validates field types and constraints, returning a strongly-typed object representing the resume scores.

Error Handling and Observability

The assessment logic includes comprehensive error handling at lines 89–91. Any exception during prompt loading, LLM inference, or JSON parsing is logged and re-raised, ensuring that failures are visible to calling applications rather than returning silent null values.

Practical Implementation Examples

Basic Resume Evaluation

from evaluator import ResumeEvaluator

resume_text = "John Doe\nSoftware Engineer with 5 years of experience..."
evaluator = ResumeEvaluator(model_name="gpt-4o-mini")
result = evaluator.evaluate_resume(resume_text)

print(result)          # Pydantic model with scores, comments, etc.

Customizing Model Parameters

You can override default generation parameters by passing a custom configuration dictionary:

custom_params = {"temperature": 0.2, "top_p": 0.95}
evaluator = ResumeEvaluator(model_name="gpt-4o-mini", model_params=custom_params)

Key Supporting Components

The assessment logic in evaluator.py relies on several supporting modules:

• prompts/template_manager.py: Handles loading and rendering of Jinja-style prompt templates (resume_evaluation_criteria, resume_evaluation_system_message)
• llm_utils.py: Provides initialize_llm_provider and extract_json_from_response utilities
• models.py: Defines the EvaluationData Pydantic model used for schema validation

Summary

• evaluator.py implements a deterministic, multi-step assessment pipeline through the ResumeEvaluator class.
• The logic uses template-based prompt construction (lines 40–59) to embed resume content into standardized evaluation frameworks.
• Structured output enforcement via EvaluationData.model_json_schema() (lines 75–77) guarantees type-safe responses.
• The provider abstraction layer (initialize_llm_provider) makes the system model-agnostic.
• Three-stage validation (extraction, parsing, Pydantic validation) ensures data integrity before returning results.

Frequently Asked Questions

How does evaluator.py ensure consistent scoring across different LLM providers?

The assessment logic enforces consistency through the EvaluationData Pydantic schema, which is passed to every provider as a JSON format requirement. By constraining the LLM to return specific fields with defined types (lines 75–77), the system normalizes outputs regardless of whether you use GPT-4, Claude, or other supported models. Additionally, standardized temperature and top_p parameters from MODEL_PARAMETERS reduce variability in generation.

What happens if the LLM returns malformed JSON or invalid scores?

The extract_json_from_response function in llm_utils.py first attempts to clean and extract valid JSON from the raw text. If extraction or subsequent json.loads parsing fails, or if the data fails Pydantic validation against the EvaluationData model (lines 80–91), the exception is logged and re-raised. This fail-fast approach ensures that calling code receives only validated, complete assessment data.

Can I customize the evaluation criteria in evaluator.py without modifying the source code?

Yes. The evaluation criteria are defined in template files loaded by template_manager.py, not hardcoded in evaluator.py. The _load_evaluation_prompt method references resume_evaluation_criteria and resume_evaluation_system_message templates. By modifying these template files or extending the template manager to load custom variants, you can adjust scoring criteria without altering the core assessment logic in evaluator.py.

Which model parameters does ResumeEvaluator use by default?

By default, the evaluator inherits parameters from MODEL_PARAMETERS, typically including temperature and top_p settings optimized for consistent evaluation. These are passed during the chat payload construction at lines 61–73. You can override these defaults by passing a custom model_params dictionary when instantiating ResumeEvaluator, as shown in the implementation examples above.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}