# How Hiring-Agent Evaluates Candidate Submissions: A Technical Deep Dive

> Discover how Hiring-Agent evaluates candidate submissions. Learn about PDF to JSON conversion, GitHub enrichment, and LLM-driven scoring for fairness and accuracy.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: deep-dive
- Published: 2026-07-15

---

**Hiring-Agent evaluates candidate submissions by converting PDF résumés into structured JSON, enriching them with GitHub profile data, and orchestrating an LLM-driven pipeline that produces structured, fairness-aware scores across four key categories.**

The `interviewstreet/hiring-agent` repository provides an open-source framework for automated résumé evaluation. Understanding how it processes candidate submissions reveals a sophisticated pipeline that combines document parsing, external API integration, and structured LLM prompting to generate objective hiring assessments.

## The Five-Stage Evaluation Pipeline

The evaluation logic follows a sequential pipeline that transforms raw PDF documents into scored evaluations. Each stage handles a specific concern, from document extraction to final score calculation.

### Stage 1: PDF Parsing and Structured Extraction

Located in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), the `PDFHandler` class initiates the process by reading résumé PDFs using **PyMuPDF**. It converts each page to Markdown-like text, then prompts an LLM through Jinja templates to produce a `JSONResume` object (defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)). When `DEVELOPMENT_MODE` is enabled, results are cached in `cache/resumecache_*.json` to avoid redundant API calls during iterative development.

### Stage 2: GitHub Profile Enrichment

If the extracted JSON contains a GitHub profile URL, [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) fetches the candidate's repository metadata and classification data. This enrichment step aggregates public code contributions as additional evaluation signals, with cached results stored in `cache/githubcache_*.json`.

### Stage 3: Resume Text Assembly

The [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) module converts structured data back into plain text through helper functions like `convert_json_resume_to_text` and `convert_github_data_to_text`. This concatenated string—containing the résumé content plus GitHub contributions—forms the *enhanced resume* input for LLM evaluation.

### Stage 4: LLM-Driven Structured Evaluation

The `ResumeEvaluator` class in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) orchestrates the core assessment:

- Uses `TemplateManager` (from [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py)) to render `resume_evaluation_system_message.jinja` and `resume_evaluation_criteria.jinja`
- Calls `initialize_llm_provider` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) to instantiate either **Ollama** or **Gemini** clients
- Enforces structured output by passing `EvaluationData.model_json_schema()` to constrain the LLM response
- Sanitizes responses using `extract_json_from_response` before parsing into the Pydantic `EvaluationData` model

### Stage 5: Scoring and Presentation

The entry point [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) aggregates evaluation results through `_evaluate_resume`, then calculates:

- Category scores for Open Source, Self Projects, Production, and Technical Skills (respecting per-category maximums)
- Bonus points (capped at 20) minus deductions
- Final total capped at 120 maximum points

Results display via `print_evaluation_results`, with `DEVELOPMENT_MODE` enabling CSV export to `resume_evaluations.csv` for analytics.

## Practical Implementation Examples

Direct programmatic usage bypassing PDF handling:

```python
from evaluator import ResumeEvaluator
from models import EvaluationData

resume_text = """
John Doe
Software Engineer
GitHub: https://github.com/johndoe
"""

evaluator = ResumeEvaluator()
evaluation: EvaluationData = evaluator.evaluate_resume(resume_text)
print(evaluation)

```

Command-line execution for full pipeline:

```bash
python score.py path/to/resume.pdf

```

This CLI invocation triggers the complete flow: PDF extraction, GitHub enrichment, LLM evaluation, and scored report generation.

## Summary

- **Multi-stage pipeline**: hiring-agent processes submissions through extraction, enrichment, transformation, LLM evaluation, and scoring phases
- **Structured data flow**: PDFs convert to `JSONResume` objects before text reassembly for LLM consumption
- **External signal integration**: GitHub profiles augment evaluation context when present in résumés
- **Provider-agnostic LLM support**: Supports both Ollama and Gemini via [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) configuration
- **Schema-constrained outputs**: Uses Pydantic schemas to enforce structured `EvaluationData` responses from the LLM
- **Configurable caps**: Final scores respect maximum limits (120 total, 20 bonus points) across four evaluation categories

## Frequently Asked Questions

### How does hiring-agent handle PDF documents that contain complex formatting?

The `PDFHandler` in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) leverages PyMuPDF to extract text while preserving document structure as Markdown-like content. It then uses LLM prompting with Jinja templates to normalize this content into the structured `JSONResume` schema, effectively handling varied formatting through semantic extraction rather than rigid parsing rules.

### What LLM providers does hiring-agent support?

According to [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), the system supports **Ollama** (for local inference) and **Gemini** (Google's API). The `initialize_llm_provider` function configures the appropriate client based on environment settings, allowing flexibility between local development and production deployment.

### Where does hiring-agent store intermediate evaluation results?

When `DEVELOPMENT_MODE` is enabled, the system caches extracted résumé data in `cache/resumecache_*.json` and GitHub enrichment data in `cache/githubcache_*.json`. Final evaluation results can be exported to `resume_evaluations.csv` for batch analysis and record keeping.

### How is the final candidate score calculated in hiring-agent?

The [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) module calculates category scores for Open Source, Self Projects, Production experience, and Technical Skills, each with defined maximums. It adds bonus points (capped at 20) and subtracts deductions, then applies a hard cap of 120 points to the final total, ensuring consistent scoring normalization across all candidate submissions.