Does evaluator.py Generate Test Cases or Use Predefined Ones?

evaluator.py does not generate test cases; instead, it uses predefined Jinja templates to evaluate resumes by sending them to an LLM and parsing structured JSON responses.

In the interviewstreet/hiring-agent repository, evaluator.py implements the ResumeEvaluator class to automate resume scoring. Unlike systems that synthesize dynamic test inputs, this module relies entirely on static, pre-written templates and schemas to assess candidate resumes against consistent criteria.

How ResumeEvaluator Processes Resumes

The ResumeEvaluator class implements a five-step pipeline that exclusively uses predefined resources rather than generated test cases.

Step 1: Initialization and Template Setup

In evaluator.py (lines 24-34), the constructor initializes the evaluation environment by accepting a model_name (e.g., "gemini" or "openai"), resolving default parameters via initialize_llm_provider from llm_utils.py, and creating a TemplateManager instance to handle prompt loading.

Step 2: Loading the Evaluation Criteria Template

The _load_evaluation_prompt method (lines 40-46) renders the resume_evaluation_criteria.jinja template, injecting the candidate's resume text into the predefined criteria structure. This ensures every evaluation uses the same static rubric encoded in the template file.

Step 3: Configuring the System Message

Lines 53-58 load an additional template, resume_evaluation_system_message.jinja, which provides the LLM with context and high-level instructions. This system message is predefined and constant across all evaluations.

Step 4: LLM Request with Structured Output

The evaluator constructs a chat payload (lines 61-78) containing both the system message and the rendered criteria prompt. It passes a format argument pointing to the JSON schema of EvaluationData (defined in models.py) to enforce structured output, then calls the provider's chat method to generate the evaluation.

Step 5: Response Parsing and Data Extraction

Lines 80-87 handle the raw LLM response by calling extract_json_from_response (from llm_utils.py) to clean the output, then parsing the JSON into an EvaluationData instance containing scores, comments, and other evaluation metrics.

Key Files in the Evaluation Architecture

The template-driven design relies on these specific files:

• evaluator.py: Contains the ResumeEvaluator class that implements the evaluation logic, template loading, and LLM interaction.
• prompts/templates/resume_evaluation_criteria.jinja: Stores the static evaluation criteria and rubric used for all resume assessments.
• prompts/templates/resume_evaluation_system_message.jinja: Provides the system-level instructions that guide the LLM's evaluation behavior.
• models.py: Defines the EvaluationData Pydantic model used for JSON schema validation and structured output parsing.
• llm_utils.py: Supplies utility functions including initialize_llm_provider and extract_json_from_response.

Practical Usage Examples

To evaluate a resume string using the predefined templates:

from evaluator import ResumeEvaluator

resume_text = """
John Doe
Software Engineer
Experience with Python and distributed systems...
"""

evaluator = ResumeEvaluator(model_name="gemini")  # or any supported model

result = evaluator.evaluate_resume(resume_text)

print(result)  # → EvaluationData instance with scores, comments, etc.

To understand the internal data flow for debugging:

evaluator = ResumeEvaluator()

# Internally, evaluate_resume performs:

# 1. Renders the Jinja templates with the resume text

# 2. Calls provider.chat() with format=EvaluationData.model_json_schema()

# 3. Extracts JSON using extract_json_from_response()

# 4. Returns EvaluationData instance

Summary

• evaluator.py uses predefined Jinja templates, not generated test cases, to evaluate resumes against static criteria.
• The ResumeEvaluator class loads criteria from resume_evaluation_criteria.jinja and system instructions from resume_evaluation_system_message.jinja.
• Evaluation occurs via LLM calls structured by the EvaluationData schema, with responses parsed through extract_json_from_response.
• All evaluation logic is template-driven and deterministic, residing in the interviewstreet/hiring-agent repository.

Frequently Asked Questions

Does evaluator.py create synthetic test data for resume evaluation?

No. The module does not generate synthetic test cases or dynamic inputs. It only accepts an existing resume string and processes it against predefined criteria templates loaded from the prompts/templates/ directory.

What templates does evaluator.py use?

It uses two Jinja templates: resume_evaluation_criteria.jinja for the evaluation rubric and resume_evaluation_system_message.jinja for LLM instructions. Both are rendered with the candidate's resume text to create the final prompt.

How does evaluator.py ensure consistent scoring?

By using static templates and a fixed JSON schema (EvaluationData), the evaluator ensures every resume is judged against identical criteria. The LLM is forced to return structured data via the format parameter in the chat call, eliminating variability in output format.

Can evaluator.py work with different LLM providers?

Yes. The constructor accepts a model_name parameter (e.g., "gemini", "openai") and uses initialize_llm_provider from llm_utils.py to instantiate the appropriate client. The template-based evaluation logic remains provider-agnostic, using the same predefined prompts regardless of the backend model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}