How Scores Are Generated for Candidate Evaluations in Hiring Agent: A Technical Deep Dive

Hiring Agent generates candidate evaluation scores by parsing PDF résumés into structured data, prompting an LLM to assess four achievement categories, then aggregating capped category scores with bonus points and deductions to produce a final clamped numeric result.

Hiring Agent is an open-source automated résumé evaluator from InterviewStreet that uses Large Language Models (LLMs) to screen engineering candidates. Understanding exactly how it calculates evaluation scores enables teams to customize scoring weights and interpret candidate rankings accurately.

Three-Stage Evaluation Pipeline

The scoring system operates through a deterministic pipeline that transforms raw PDF documents into structured numeric ratings.

Stage 1: PDF Parsing and Enrichment

The process begins in pdf.py where the PDFHandler class extracts structured data from candidate résumés.

from pdf import PDFHandler

pdf_handler = PDFHandler()
resume_data = pdf_handler.extract_json_from_pdf(pdf_path)

This produces a JSONResume Pydantic model containing sections like basics, work, education, skills, and projects. If the candidate includes a GitHub profile, the system optionally enriches the data by fetching public repositories via fetch_and_display_github_info in github.py, merging additional context into the résumé text before evaluation.

Stage 2: LLM-Driven Assessment

The core scoring logic resides in evaluator.py. The ResumeEvaluator.evaluate_resume method (lines 48-87) constructs a detailed prompt using the resume_evaluation_criteria template and sends it to the configured LLM provider (Ollama or Gemini).

from evaluator import ResumeEvaluator

evaluator = ResumeEvaluator()
evaluation = evaluator.evaluate_resume(resume_text)  # Returns EvaluationData

The LLM receives the complete résumé text plus a JSON schema defining the expected output structure. It returns an EvaluationData object (defined in models.py, lines 18-50) containing four categorized scores:

• Open Source (open_source)
• Self Projects (self_projects)
• Production Experience (production)
• Technical Skills (technical_skills)

Each category is a CategoryScore containing a raw score, a max ceiling, and an evidence string justifying the rating.

Stage 3: Score Aggregation and Clamping

Final score calculation occurs in score.py, primarily within the print_evaluation_results function. The algorithm implements three critical safeguards to prevent score inflation:

1. Per-category capping: Each score is clamped to its configured maximum (e.g., Open Source capped at 35 points)
2. Bonus and deduction handling: Adds points from BonusPoints and subtracts Deductions
3. Global maximum enforcement: Clamps the total to category_maxes + 20 (where 20 represents the maximum possible bonus points)

# Logic derived from score.py lines 41-73 and 78-86

total_score = 0
max_score = 0

for cat, data in evaluation.scores.model_dump().items():
    cat_score = min(data["score"], data["max"])  # Enforce per-category cap

    total_score += cat_score
    max_score += data["max"]

# Apply bonuses and deductions

total_score += evaluation.bonus_points.total
total_score -= evaluation.deductions.total

# Global clamp

max_possible = max_score + 20
final_score = min(total_score, max_possible)

Core Data Models

The scoring system relies on strict Pydantic models defined in models.py (lines 18-50) to ensure type safety and structured LLM outputs.

CategoryScore defines individual rating components:

class CategoryScore(BaseModel):
    score: float
    max: float
    evidence: str

EvaluationData serves as the top-level schema containing all scoring dimensions:

class Scores(BaseModel):
    open_source: CategoryScore
    self_projects: CategoryScore
    production: CategoryScore
    technical_skills: CategoryScore

class EvaluationData(BaseModel):
    scores: Scores
    bonus_points: BonusPoints
    deductions: Deductions
    key_strengths: List[str]
    areas_for_improvement: List[str]

Customizing the Scoring Algorithm

Teams can modify scoring behavior by adjusting category maximums or bonus calculations in score.py. The category maximums are defined at lines 80-85, while the aggregation logic resides in the print_evaluation_results function.

Manual score aggregation (mirroring the production implementation):

def calculate_final_score(evaluation: EvaluationData) -> float:
    """Replicate Hiring Agent's scoring logic programmatically."""
    total, max_total = 0.0, 0.0
    
    for cat in evaluation.scores.model_dump().values():
        total += min(cat["score"], cat["max"])
        max_total += cat["max"]
    
    total += evaluation.bonus_points.total
    total -= evaluation.deductions.total
    
    return min(total, max_total + 20)  # 20 = MAX_BONUS_POINTS

Running Evaluations

Execute the full pipeline from the command line:

python score.py path/to/candidate_resume.pdf

This invokes the main() function in score.py, which orchestrates PDF parsing, optional GitHub enrichment, LLM evaluation, and final score computation.

Summary

• Hiring Agent uses a three-stage pipeline: PDF parsing → LLM evaluation → score aggregation.
• Per-category caps prevent individual dimensions from exceeding predefined limits (e.g., 35 points for Open Source).
• Bonus points (maximum 20) and deductions adjust the final total before clamping to the global maximum.
• Pydantic models (EvaluationData, CategoryScore) enforce structured LLM outputs and type safety throughout the system.
• Source files: evaluator.py handles LLM prompting, score.py manages aggregation logic, and models.py defines the data schemas.

Frequently Asked Questions

How does Hiring Agent prevent LLMs from generating inflated scores?

The system implements strict output validation using Pydantic schemas defined in models.py. Even if the LLM returns high raw values, the aggregation logic in score.py enforces per-category maximums and a global cap of category_maxes + 20, ensuring scores remain within configured boundaries.

Can I modify the category weights or maximum point values?

Yes. Edit the category maximums defined in score.py (lines 80-85) and adjust the MAX_BONUS_POINTS constant (set to 20) to change the total score range. The system automatically recalculates the global clamp based on these configuration values.

What happens if a candidate has no GitHub profile?

The evaluation proceeds normally. The find_profile function in score.py detects missing profiles, and the pipeline skips the GitHub enrichment step in github.py. The LLM evaluates the candidate solely based on the PDF résumé content, though bonus points typically associated with open-source contributions may not apply.

Which LLM providers does Hiring Agent support?

According to the source code in evaluator.py, the system supports Ollama (for local models) and Gemini (Google's API). The initialize_llm_provider function selects the appropriate client based on environment configuration, allowing teams to run evaluations offline or via cloud APIs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}