# How Hiring Agent Generates Explainable Evaluation Evidence

> Learn how Hiring Agent creates explainable evaluation evidence by using LLMs to generate scored categories with textual justifications for clear insights.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: internals
- Published: 2026-07-21

---

**Hiring Agent generates evaluation evidence by prompting a large language model to return structured JSON containing scored categories with textual justifications, then parsing and displaying those evidence fields alongside numeric ratings.**

The `interviewstreet/hiring-agent` repository implements an automated résumé evaluation system that produces transparent, auditable assessments. This open-source tool leverages carefully engineered prompts to extract structured reasoning from large language models, making the **evaluation evidence** fully explainable. The implementation centers on a pipeline that transforms raw résumé text into categorized scores with detailed textual support grounded in the source code of [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) and [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py).

## The Evaluation Pipeline: From Resume Text to Structured Evidence

### Prompt Construction and Template Injection

The `ResumeEvaluator` class in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) orchestrates evidence generation by loading Jinja templates that define the evaluation criteria. The system injects the raw résumé text into the `resume_evaluation_criteria` template while using `resume_evaluation_system_message` to instruct the model on output formatting. This dual-template approach ensures the LLM receives both the content to evaluate and strict structural requirements for the response.

### LLM Invocation with Schema Enforcement

When `evaluate_resume` is called, the evaluator constructs a message array combining the system instructions and user content, then invokes `self.provider.chat` with a `format` parameter. This parameter forces the model to output JSON matching the `EvaluationData` schema defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), effectively constraining the LLM to produce structured evidence rather than free-form text.

### Response Parsing and JSON Extraction

After receiving the raw LLM output, the pipeline applies `extract_json_from_response` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) to clean the text and isolate valid JSON. The sanitized response is then deserialized into an `EvaluationData` instance through Pydantic validation, ensuring type safety before evidence extraction occurs.

## How Evidence Is Captured and Structured

### The Scoring Schema and Evidence Fields

Within the parsed `EvaluationData` object, the `scores` field contains nested structures for each evaluation category—including **open_source**, **self_projects**, **production**, and **technical_skills**. Each category object holds three critical fields: `score` (numeric rating), `max` (maximum possible points), and `evidence` (textual justification). This design ensures every numeric assessment carries an accompanying explanation generated by the LLM.

### Evidence Generation Mechanism

The textual evidence itself originates directly from the LLM's reasoning process as guided by the criteria template in `prompts/templates/resume_evaluation_criteria.jinja`. When the model analyzes the résumé content against predefined rubrics, it generates natural language justifications that populate the `evidence` fields of the JSON response. This creates an auditable trail connecting specific résumé content to specific scoring decisions.

## Presenting Explainable Results

### Human-Readable Evidence Display

The `print_evaluation_results` function in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) traverses the `EvaluationData` object and renders each category's evidence alongside its numeric score. This presentation layer transforms the structured JSON into formatted output that hiring managers can review, providing transparency into why a candidate received specific ratings in each competency area.

## Practical Implementation Examples

Running a complete evaluation from the command line:

```bash
python score.py path/to/resume.pdf

```

This command extracts the PDF content, optionally enriches it with GitHub profile data, invokes `ResumeEvaluator.evaluate_resume`, and displays the evidence for each scoring category through `print_evaluation_results`.

Programmatic access to evaluation evidence:

```python
from evaluator import ResumeEvaluator

# Raw résumé text (already extracted from a PDF or other source)

resume_text = "John Doe\nSoftware Engineer…"

evaluator = ResumeEvaluator()
evaluation = evaluator.evaluate_resume(resume_text)

# Access evidence for the “open_source” category

print(evaluation.scores.open_source.evidence)

```

Inspecting the raw LLM JSON response for debugging:

```python
from evaluator import ResumeEvaluator
from llm_utils import extract_json_from_response
from models import EvaluationData

evaluator = ResumeEvaluator()
raw_response = evaluator.provider.chat(
    model="gpt-4o-mini",
    messages=[{"role": "system", "content": "..."},
              {"role": "user", "content": "..."}],
    format=EvaluationData.model_json_schema()
)
json_text = extract_json_from_response(raw_response["message"]["content"])
print(json_text)  # Shows scores, evidence, bonuses, deductions, etc.

```

## Summary

- **Hiring Agent** generates explainable evidence by constraining LLM outputs to a structured JSON schema defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py).
- The `ResumeEvaluator` class in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) handles prompt construction, template injection, and response parsing using Jinja templates stored in `prompts/templates/`.
- Evidence appears as textual justifications within each scoring category (**open_source**, **self_projects**, **production**, **technical_skills**) alongside numeric ratings.
- The [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) module provides both CLI and programmatic interfaces for running evaluations and displaying human-readable evidence trails.
- JSON extraction and cleaning utilities in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) ensure robust parsing of LLM responses before evidence extraction.

## Frequently Asked Questions

### How does Hiring Agent ensure the LLM returns valid JSON?

Hiring Agent enforces output structure by passing `EvaluationData.model_json_schema()` as the `format` parameter during the LLM call. This schema constraint, combined with the system message template in `prompts/templates/resume_evaluation_system_message.jinja`, instructs the model to produce valid JSON. Additionally, the `extract_json_from_response` utility in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) cleans the raw response to handle any markdown code blocks or extraneous text before Pydantic validation occurs.

### What specific categories does the evaluation evidence cover?

The evidence structure covers four primary categories defined in the `EvaluationData` schema: **open_source** (contributions to public repositories), **self_projects** (personal development work), **production** (professional implementation experience), and **technical_skills** (language and framework proficiency). Each category includes a score, maximum value, and textual evidence field providing specific justifications for the rating.

### Can I customize the evaluation criteria or evidence format?

Yes, the evaluation criteria are defined in `prompts/templates/resume_evaluation_criteria.jinja`, which can be modified to adjust the rubrics the LLM uses when generating evidence. The output format is governed by the Pydantic models in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)—modifying `EvaluationData` or its nested score categories will change the JSON schema enforced during LLM generation, though changes must align with the LLM's capabilities to produce structured outputs.

### Which files handle the extraction of JSON from raw LLM responses?

The [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) file contains the `extract_json_from_response` helper function that isolates valid JSON from markdown-formatted or conversational LLM outputs. This function is invoked in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) immediately after receiving the raw response and before Pydantic deserialization into the `EvaluationData` model.