How Hiring-Agent Evaluates Candidate Submissions: A Technical Deep Dive

Hiring-Agent evaluates candidate submissions by converting PDF résumés into structured JSON, enriching them with GitHub profile data, and orchestrating an LLM-driven pipeline that produces structured, fairness-aware scores across four key categories.

The interviewstreet/hiring-agent repository provides an open-source framework for automated résumé evaluation. Understanding how it processes candidate submissions reveals a sophisticated pipeline that combines document parsing, external API integration, and structured LLM prompting to generate objective hiring assessments.

The Five-Stage Evaluation Pipeline

The evaluation logic follows a sequential pipeline that transforms raw PDF documents into scored evaluations. Each stage handles a specific concern, from document extraction to final score calculation.

Stage 1: PDF Parsing and Structured Extraction

Located in pdf.py, the PDFHandler class initiates the process by reading résumé PDFs using PyMuPDF. It converts each page to Markdown-like text, then prompts an LLM through Jinja templates to produce a JSONResume object (defined in models.py). When DEVELOPMENT_MODE is enabled, results are cached in cache/resumecache_*.json to avoid redundant API calls during iterative development.

Stage 2: GitHub Profile Enrichment

If the extracted JSON contains a GitHub profile URL, github.py fetches the candidate's repository metadata and classification data. This enrichment step aggregates public code contributions as additional evaluation signals, with cached results stored in cache/githubcache_*.json.

Stage 3: Resume Text Assembly

The transform.py module converts structured data back into plain text through helper functions like convert_json_resume_to_text and convert_github_data_to_text. This concatenated string—containing the résumé content plus GitHub contributions—forms the enhanced resume input for LLM evaluation.

Stage 4: LLM-Driven Structured Evaluation

The ResumeEvaluator class in evaluator.py orchestrates the core assessment:

• Uses TemplateManager (from prompts/template_manager.py) to render resume_evaluation_system_message.jinja and resume_evaluation_criteria.jinja
• Calls initialize_llm_provider from llm_utils.py to instantiate either Ollama or Gemini clients
• Enforces structured output by passing EvaluationData.model_json_schema() to constrain the LLM response
• Sanitizes responses using extract_json_from_response before parsing into the Pydantic EvaluationData model

Stage 5: Scoring and Presentation

The entry point score.py aggregates evaluation results through _evaluate_resume, then calculates:

• Category scores for Open Source, Self Projects, Production, and Technical Skills (respecting per-category maximums)
• Bonus points (capped at 20) minus deductions
• Final total capped at 120 maximum points

Results display via print_evaluation_results, with DEVELOPMENT_MODE enabling CSV export to resume_evaluations.csv for analytics.

Practical Implementation Examples

Direct programmatic usage bypassing PDF handling:

from evaluator import ResumeEvaluator
from models import EvaluationData

resume_text = """
John Doe
Software Engineer
GitHub: https://github.com/johndoe
"""

evaluator = ResumeEvaluator()
evaluation: EvaluationData = evaluator.evaluate_resume(resume_text)
print(evaluation)

Command-line execution for full pipeline:

python score.py path/to/resume.pdf

This CLI invocation triggers the complete flow: PDF extraction, GitHub enrichment, LLM evaluation, and scored report generation.

Summary

• Multi-stage pipeline: hiring-agent processes submissions through extraction, enrichment, transformation, LLM evaluation, and scoring phases
• Structured data flow: PDFs convert to JSONResume objects before text reassembly for LLM consumption
• External signal integration: GitHub profiles augment evaluation context when present in résumés
• Provider-agnostic LLM support: Supports both Ollama and Gemini via llm_utils.py configuration
• Schema-constrained outputs: Uses Pydantic schemas to enforce structured EvaluationData responses from the LLM
• Configurable caps: Final scores respect maximum limits (120 total, 20 bonus points) across four evaluation categories

Frequently Asked Questions

How does hiring-agent handle PDF documents that contain complex formatting?

The PDFHandler in pdf.py leverages PyMuPDF to extract text while preserving document structure as Markdown-like content. It then uses LLM prompting with Jinja templates to normalize this content into the structured JSONResume schema, effectively handling varied formatting through semantic extraction rather than rigid parsing rules.

What LLM providers does hiring-agent support?

According to llm_utils.py, the system supports Ollama (for local inference) and Gemini (Google's API). The initialize_llm_provider function configures the appropriate client based on environment settings, allowing flexibility between local development and production deployment.

Where does hiring-agent store intermediate evaluation results?

When DEVELOPMENT_MODE is enabled, the system caches extracted résumé data in cache/resumecache_*.json and GitHub enrichment data in cache/githubcache_*.json. Final evaluation results can be exported to resume_evaluations.csv for batch analysis and record keeping.

How is the final candidate score calculated in hiring-agent?

The score.py module calculates category scores for Open Source, Self Projects, Production experience, and Technical Skills, each with defined maximums. It adds bonus points (capped at 20) and subtracts deductions, then applies a hard cap of 120 points to the final total, ensuring consistent scoring normalization across all candidate submissions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}