How Hiring Agent Generates Explainable Evaluation Evidence

Hiring Agent generates evaluation evidence by prompting a large language model to return structured JSON containing scored categories with textual justifications, then parsing and displaying those evidence fields alongside numeric ratings.

The interviewstreet/hiring-agent repository implements an automated résumé evaluation system that produces transparent, auditable assessments. This open-source tool leverages carefully engineered prompts to extract structured reasoning from large language models, making the evaluation evidence fully explainable. The implementation centers on a pipeline that transforms raw résumé text into categorized scores with detailed textual support grounded in the source code of evaluator.py and score.py.

The Evaluation Pipeline: From Resume Text to Structured Evidence

Prompt Construction and Template Injection

The ResumeEvaluator class in evaluator.py orchestrates evidence generation by loading Jinja templates that define the evaluation criteria. The system injects the raw résumé text into the resume_evaluation_criteria template while using resume_evaluation_system_message to instruct the model on output formatting. This dual-template approach ensures the LLM receives both the content to evaluate and strict structural requirements for the response.

LLM Invocation with Schema Enforcement

When evaluate_resume is called, the evaluator constructs a message array combining the system instructions and user content, then invokes self.provider.chat with a format parameter. This parameter forces the model to output JSON matching the EvaluationData schema defined in models.py, effectively constraining the LLM to produce structured evidence rather than free-form text.

Response Parsing and JSON Extraction

After receiving the raw LLM output, the pipeline applies extract_json_from_response from llm_utils.py to clean the text and isolate valid JSON. The sanitized response is then deserialized into an EvaluationData instance through Pydantic validation, ensuring type safety before evidence extraction occurs.

How Evidence Is Captured and Structured

The Scoring Schema and Evidence Fields

Within the parsed EvaluationData object, the scores field contains nested structures for each evaluation category—including open_source, self_projects, production, and technical_skills. Each category object holds three critical fields: score (numeric rating), max (maximum possible points), and evidence (textual justification). This design ensures every numeric assessment carries an accompanying explanation generated by the LLM.

Evidence Generation Mechanism

The textual evidence itself originates directly from the LLM's reasoning process as guided by the criteria template in prompts/templates/resume_evaluation_criteria.jinja. When the model analyzes the résumé content against predefined rubrics, it generates natural language justifications that populate the evidence fields of the JSON response. This creates an auditable trail connecting specific résumé content to specific scoring decisions.

Presenting Explainable Results

Human-Readable Evidence Display

The print_evaluation_results function in score.py traverses the EvaluationData object and renders each category's evidence alongside its numeric score. This presentation layer transforms the structured JSON into formatted output that hiring managers can review, providing transparency into why a candidate received specific ratings in each competency area.

Practical Implementation Examples

Running a complete evaluation from the command line:

python score.py path/to/resume.pdf

This command extracts the PDF content, optionally enriches it with GitHub profile data, invokes ResumeEvaluator.evaluate_resume, and displays the evidence for each scoring category through print_evaluation_results.

Programmatic access to evaluation evidence:

from evaluator import ResumeEvaluator

# Raw résumé text (already extracted from a PDF or other source)

resume_text = "John Doe\nSoftware Engineer…"

evaluator = ResumeEvaluator()
evaluation = evaluator.evaluate_resume(resume_text)

# Access evidence for the “open_source” category

print(evaluation.scores.open_source.evidence)

Inspecting the raw LLM JSON response for debugging:

from evaluator import ResumeEvaluator
from llm_utils import extract_json_from_response
from models import EvaluationData

evaluator = ResumeEvaluator()
raw_response = evaluator.provider.chat(
    model="gpt-4o-mini",
    messages=[{"role": "system", "content": "..."},
              {"role": "user", "content": "..."}],
    format=EvaluationData.model_json_schema()
)
json_text = extract_json_from_response(raw_response["message"]["content"])
print(json_text)  # Shows scores, evidence, bonuses, deductions, etc.

Summary

• Hiring Agent generates explainable evidence by constraining LLM outputs to a structured JSON schema defined in models.py.
• The ResumeEvaluator class in evaluator.py handles prompt construction, template injection, and response parsing using Jinja templates stored in prompts/templates/.
• Evidence appears as textual justifications within each scoring category (open_source, self_projects, production, technical_skills) alongside numeric ratings.
• The score.py module provides both CLI and programmatic interfaces for running evaluations and displaying human-readable evidence trails.
• JSON extraction and cleaning utilities in llm_utils.py ensure robust parsing of LLM responses before evidence extraction.

Frequently Asked Questions

How does Hiring Agent ensure the LLM returns valid JSON?

Hiring Agent enforces output structure by passing EvaluationData.model_json_schema() as the format parameter during the LLM call. This schema constraint, combined with the system message template in prompts/templates/resume_evaluation_system_message.jinja, instructs the model to produce valid JSON. Additionally, the extract_json_from_response utility in llm_utils.py cleans the raw response to handle any markdown code blocks or extraneous text before Pydantic validation occurs.

What specific categories does the evaluation evidence cover?

The evidence structure covers four primary categories defined in the EvaluationData schema: open_source (contributions to public repositories), self_projects (personal development work), production (professional implementation experience), and technical_skills (language and framework proficiency). Each category includes a score, maximum value, and textual evidence field providing specific justifications for the rating.

Can I customize the evaluation criteria or evidence format?

Yes, the evaluation criteria are defined in prompts/templates/resume_evaluation_criteria.jinja, which can be modified to adjust the rubrics the LLM uses when generating evidence. The output format is governed by the Pydantic models in models.py—modifying EvaluationData or its nested score categories will change the JSON schema enforced during LLM generation, though changes must align with the LLM's capabilities to produce structured outputs.

Which files handle the extraction of JSON from raw LLM responses?

The llm_utils.py file contains the extract_json_from_response helper function that isolates valid JSON from markdown-formatted or conversational LLM outputs. This function is invoked in evaluator.py immediately after receiving the raw response and before Pydantic deserialization into the EvaluationData model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}