# Does evaluator.py Generate Test Cases or Use Predefined Ones?

> Discover whether evaluator.py generates test cases or uses predefined Jinja templates with an LLM. Get clear insights into resume evaluation.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: internals
- Published: 2026-07-15

---

**evaluator.py does not generate test cases; instead, it uses predefined Jinja templates to evaluate resumes by sending them to an LLM and parsing structured JSON responses.**

In the `interviewstreet/hiring-agent` repository, [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) implements the `ResumeEvaluator` class to automate resume scoring. Unlike systems that synthesize dynamic test inputs, this module relies entirely on static, pre-written templates and schemas to assess candidate resumes against consistent criteria.

## How ResumeEvaluator Processes Resumes

The `ResumeEvaluator` class implements a five-step pipeline that exclusively uses predefined resources rather than generated test cases.

### Step 1: Initialization and Template Setup

In [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) (lines 24-34), the constructor initializes the evaluation environment by accepting a `model_name` (e.g., "gemini" or "openai"), resolving default parameters via `initialize_llm_provider` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), and creating a `TemplateManager` instance to handle prompt loading.

### Step 2: Loading the Evaluation Criteria Template

The `_load_evaluation_prompt` method (lines 40-46) renders the `resume_evaluation_criteria.jinja` template, injecting the candidate's resume text into the predefined criteria structure. This ensures every evaluation uses the same static rubric encoded in the template file.

### Step 3: Configuring the System Message

Lines 53-58 load an additional template, `resume_evaluation_system_message.jinja`, which provides the LLM with context and high-level instructions. This system message is predefined and constant across all evaluations.

### Step 4: LLM Request with Structured Output

The evaluator constructs a chat payload (lines 61-78) containing both the system message and the rendered criteria prompt. It passes a `format` argument pointing to the JSON schema of `EvaluationData` (defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)) to enforce structured output, then calls the provider's `chat` method to generate the evaluation.

### Step 5: Response Parsing and Data Extraction

Lines 80-87 handle the raw LLM response by calling `extract_json_from_response` (from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)) to clean the output, then parsing the JSON into an `EvaluationData` instance containing scores, comments, and other evaluation metrics.

## Key Files in the Evaluation Architecture

The template-driven design relies on these specific files:

- **[`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py)**: Contains the `ResumeEvaluator` class that implements the evaluation logic, template loading, and LLM interaction.
- **`prompts/templates/resume_evaluation_criteria.jinja`**: Stores the static evaluation criteria and rubric used for all resume assessments.
- **`prompts/templates/resume_evaluation_system_message.jinja`**: Provides the system-level instructions that guide the LLM's evaluation behavior.
- **[`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)**: Defines the `EvaluationData` Pydantic model used for JSON schema validation and structured output parsing.
- **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)**: Supplies utility functions including `initialize_llm_provider` and `extract_json_from_response`.

## Practical Usage Examples

To evaluate a resume string using the predefined templates:

```python
from evaluator import ResumeEvaluator

resume_text = """
John Doe
Software Engineer
Experience with Python and distributed systems...
"""

evaluator = ResumeEvaluator(model_name="gemini")  # or any supported model

result = evaluator.evaluate_resume(resume_text)

print(result)  # → EvaluationData instance with scores, comments, etc.

```

To understand the internal data flow for debugging:

```python
evaluator = ResumeEvaluator()

# Internally, evaluate_resume performs:

# 1. Renders the Jinja templates with the resume text

# 2. Calls provider.chat() with format=EvaluationData.model_json_schema()

# 3. Extracts JSON using extract_json_from_response()

# 4. Returns EvaluationData instance

```

## Summary

- **evaluator.py uses predefined Jinja templates**, not generated test cases, to evaluate resumes against static criteria.
- The `ResumeEvaluator` class loads criteria from `resume_evaluation_criteria.jinja` and system instructions from `resume_evaluation_system_message.jinja`.
- Evaluation occurs via LLM calls structured by the `EvaluationData` schema, with responses parsed through `extract_json_from_response`.
- All evaluation logic is template-driven and deterministic, residing in the `interviewstreet/hiring-agent` repository.

## Frequently Asked Questions

### Does evaluator.py create synthetic test data for resume evaluation?

No. The module does not generate synthetic test cases or dynamic inputs. It only accepts an existing resume string and processes it against predefined criteria templates loaded from the `prompts/templates/` directory.

### What templates does evaluator.py use?

It uses two Jinja templates: `resume_evaluation_criteria.jinja` for the evaluation rubric and `resume_evaluation_system_message.jinja` for LLM instructions. Both are rendered with the candidate's resume text to create the final prompt.

### How does evaluator.py ensure consistent scoring?

By using static templates and a fixed JSON schema (`EvaluationData`), the evaluator ensures every resume is judged against identical criteria. The LLM is forced to return structured data via the `format` parameter in the chat call, eliminating variability in output format.

### Can evaluator.py work with different LLM providers?

Yes. The constructor accepts a `model_name` parameter (e.g., "gemini", "openai") and uses `initialize_llm_provider` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) to instantiate the appropriate client. The template-based evaluation logic remains provider-agnostic, using the same predefined prompts regardless of the backend model.