Fairness Constraints in Hiring Agent's Evaluation: Technical Implementation Guide

Hiring Agent enforces strict fairness constraints through prompt-engineered LLM instructions that prohibit demographic bias, mandate technical merit scoring, enforce category point caps, and validate structured output via Pydantic schemas.

The interviewstreet/hiring-agent repository implements a bias-resistant resume evaluation system that relies on explicit fairness constraints embedded in LLM prompts. These constraints ensure that candidate assessments depend solely on technical ability rather than demographic characteristics. Understanding how these safeguards are implemented helps developers audit and maintain equitable AI-driven hiring tools.

Core Fairness Rules in Resume Evaluation

Demographic Blindness Requirements

The evaluation criteria explicitly prohibit the LLM from considering personal characteristics. According to the resume_evaluation_criteria.jinja template, scores must never depend on a candidate's name, gender, age, ethnicity, college name, GPA, city, or any attribute unrelated to technical capability.

Technical Merit Scoring Focus

Evaluations must derive solely from six specific technical areas:

• Technical skills and programming languages
• Project complexity and real-world impact
• Open-source contributions and community involvement
• Production-level work experience
• Technical communication and documentation
• Problem-solving demonstrated in projects

Program Distinction Safeguards

The system treats "Google Summer of Code (GSoC)" and "Girl Script Summer of Code" as distinct programs within the prompt template. This prevents scoring shortcuts that could create bias between similarly structured programs with different names.

Hard Score Limits and Ceilings

The EvaluationData schema in models.py enforces strict numeric boundaries:

• Open-source: 0-35 points
• Self-projects: 0-30 points
• Production: 0-25 points
• Technical skills: 0-10 points
• Bonus points: Maximum 20 total
• Overall score: Maximum 120 points (categories + bonus - deductions)

Additionally, every score field must contain non-empty evidence strings, and all category scores must be ≥ 0 with deductions captured separately.

How Constraints Are Enforced at Runtime

Prompt Construction

The ResumeEvaluator._load_evaluation_prompt method renders the Jinja template containing fairness rules. This prepares the system message that instructs the LLM on acceptable evaluation criteria.

LLM Provider Interaction

The composed prompt is sent to the configured provider, either models.OllamaProvider or models.GeminiProvider, depending on deployment configuration.

Response Validation

Raw LLM output passes through extract_json_from_response and undergoes strict validation against the EvaluationData Pydantic model. Any deviation from the schema—such as missing evidence fields or scores exceeding defined maximums—raises validation errors that halt processing.

Post-Processing Guards

Validation failures force developers to adjust either the prompt engineering or the LLM behavior, creating a feedback loop that maintains constraint integrity across evaluations.

Implementation in the Source Code

The fairness architecture spans multiple files in the repository:

• prompts/templates/resume_evaluation_criteria.jinja: Contains the explicit fairness rules and scoring criteria embedded in the LLM prompt.
• evaluator.py: Houses the ResumeEvaluator class with the _load_evaluation_prompt method that orchestrates prompt assembly.
• models.py: Defines the EvaluationData Pydantic schema that enforces field presence and numeric constraints at runtime.
• prompt.py and llm_utils.py: Provide provider selection and response parsing utilities.
• score.py: CLI entry point that orchestrates end-to-end execution from PDF ingestion to final evaluation.

Practical Code Example

The following demonstrates how fairness constraints are automatically enforced during evaluation:

from evaluator import ResumeEvaluator

# Load resume text (already extracted from PDF)

with open("sample_resume.md") as f:
    resume_text = f.read()

evaluator = ResumeEvaluator()                # Uses DEFAULT_MODEL from config

evaluation = evaluator.evaluate_resume(resume_text)

print(evaluation.json(indent=2))

The returned evaluation object contains:

• Four mandatory score categories respecting their maximum point caps
• A bonus_points object with total ≤ 20
• A deductions object for penalties
• Required evidence strings for every scored field
• key_strengths and areas_for_improvement limited to required counts

If the LLM attempts to assign demographic-based scores or exceed point limits, the EvaluationData validation raises an immediate error, preventing biased outputs from reaching downstream systems.

Summary

• Hiring Agent's fairness constraints are embedded directly in the LLM prompt template at prompts/templates/resume_evaluation_criteria.jinja.
• Demographic characteristics including name, gender, age, ethnicity, and education institution are explicitly excluded from scoring considerations.
• Technical merit evaluation focuses on six specific competency areas with strict evidentiary requirements.
• Pydantic schema validation in models.py enforces hard numeric ceilings on all score categories and bonus points.
• The four-stage pipeline (prompt construction, LLM invocation, JSON extraction, and schema validation) ensures constraints apply consistently across OllamaProvider and GeminiProvider backends.

Frequently Asked Questions

How does Hiring Agent prevent demographic bias in resume scoring?

The system embeds explicit prohibitions in the evaluation criteria template, instructing the LLM to ignore names, gender, age, ethnicity, college names, GPA, and geographic location. These instructions are enforced at the prompt level and validated through the EvaluationData schema, ensuring scores derive solely from demonstrated technical abilities.

What are the maximum point allocations for each evaluation category?

According to the schema defined in models.py, the category limits are: Open-source (35 points), Self-projects (30 points), Production (25 points), and Technical skills (10 points). Bonus points cannot exceed 20, and the total combined score has a hard ceiling of 120 points.

Which source files contain the fairness constraint implementation?

The constraints are defined in prompts/templates/resume_evaluation_criteria.jinja, loaded by ResumeEvaluator._load_evaluation_prompt in evaluator.py, and enforced through the EvaluationData Pydantic model in models.py. Response parsing utilities in llm_utils.py handle the extraction and initial formatting of LLM outputs before validation.

What happens if the LLM generates scores that violate fairness constraints?

The EvaluationData schema validation catches violations such as missing evidence strings, scores exceeding category maximums, or negative values. These validation errors force immediate correction by the developer, either through prompt refinement or LLM behavior adjustment, preventing non-compliant evaluations from being processed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}