# How to Extend Evaluation Criteria in Hiring-Agent: Adding Custom Scoring Rules

> Learn how to extend evaluation criteria in Hiring-Agent by adding custom scoring rules. Easily update resume evaluation templates and Pydantic models for advanced candidate assessment.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: how-to-guide
- Published: 2026-07-05

---

**You can extend the evaluation criteria by editing the `resume_evaluation_criteria.jinja` template to describe new rules to the LLM, and optionally updating the `EvaluationData` Pydantic model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) if you require new top-level score fields.**

The interviewstreet/hiring-agent repository uses a prompt-driven scoring architecture where the `ResumeEvaluator` class processes candidate resumes through a structured Jinja template. Extending evaluation criteria requires modifying the LLM instructions in the template file and, for new scoring dimensions, adjusting the validation schema to match the expected JSON output.

## Understanding the Evaluation Architecture

The hiring-agent pipeline relies on three core components working in sequence. First, the **Jinja template** at `prompts/templates/resume_evaluation_criteria.jinja` contains the human-readable scoring instructions that guide the LLM. Second, the **Pydantic models** in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) (specifically `EvaluationData` and `Scores`) validate the structured JSON response returned by the LLM. Third, the **evaluator constants** in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) define scoring limits such as `MAX_BONUS_POINTS` and `MAX_FINAL_SCORE` that constrain the final calculation.

The `ResumeEvaluator.evaluate_resume()` method orchestrates this flow by rendering the template, sending it to the LLM, and deserializing the response into the `EvaluationData` model. This design separates the scoring logic (defined in prompts) from the data validation (defined in Python), making the system highly adaptable.

## Three Levels of Extension

You can customize scoring rules at three distinct levels depending on your requirements.

### Prompt Level: New Rules in the Jinja Template

The most straightforward approach involves editing the Jinja template to append new rule blocks. The template already contains sections for **Open Source**, **Self Projects**, **Production**, and **Technical Skills** with specific bonus and deduction criteria. By adding new category descriptions—such as "Community Engagement" or "Technical Writing"—you provide the LLM with instructions to generate additional scores without modifying any Python code. The LLM will include these new evaluations in its JSON response as long as the instructions are clear.

### Model Level: New Pydantic Fields

If your new rule introduces a **new top-level score category** rather than modifying existing logic, you must extend the `Scores` model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py). Add a new `CategoryScore` field to capture the structured output. The `ResumeEvaluator` automatically deserializes the LLM response into this updated schema, populating the new field without requiring changes to the evaluation logic itself.

### Constants Level: Scoring Limits and Post-Processing

When new rules change the maximum achievable points, adjust the constants defined in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py). Values such as `MAX_BONUS_POINTS`, `MIN_FINAL_SCORE`, and `MAX_FINAL_SCORE` constrain the final calculation. Update these thresholds to maintain coherent scoring boundaries. Downstream components like [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) and CSV export utilities will automatically respect these new limits.

## Practical Example: Adding a Community Engagement Category

Follow this concrete example to add a "Community Engagement" category worth up to 15 points.

### Step 1: Edit the Jinja Template

Open `prompts/templates/resume_evaluation_criteria.jinja` and append a new section following the existing format.

```jinja

## Community Engagement (0-15 points)

- Evaluate participation in developer communities (e.g., Stack Overflow, Reddit, Discord).
- **HIGH SCORES (12-15 points):** Regular contributions (≥ 50 answers/up-votes), moderation roles, or leadership in open-source communities.
- **MEDIUM SCORES (6-11 points):** Occasional helpful answers, speaking at meet-ups, or active membership in relevant forums.
- **LOW SCORES (0-5 points):** Minimal or no community activity.

### BONUS POINTS

- +2 points for mentoring junior developers (verified via LinkedIn recommendations or personal statements).

```

This block instructs the LLM to generate a `community_engagement` entry in the JSON output.

### Step 2: Update the Pydantic Schema (Optional)

If you need `community_engagement` as a distinct top-level field in the results, modify [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) to extend the `Scores` class.

```python
from pydantic import BaseModel

class CategoryScore(BaseModel):
    score: int
    max: int
    evidence: str

class Scores(BaseModel):
    open_source: CategoryScore
    self_projects: CategoryScore
    production: CategoryScore
    technical_skills: CategoryScore
    community_engagement: CategoryScore  # New field added

```

The `ResumeEvaluator` automatically populates this field when parsing the LLM response.

### Step 3: Adjust Scoring Limits (Optional)

If the new category increases the theoretical maximum score, update the constants in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py).

```python

# evaluator.py

MAX_BONUS_POINTS = 25  # Increased from 20 to accommodate community bonuses

MAX_FINAL_SCORE = 135  # Adjust if the new category pushes the total maximum higher

```

### Step 4: Test the Pipeline

Run the evaluation pipeline to verify the integration works end-to-end.

```bash
python score.py examples/resume.pdf

```

Verify that the output JSON contains the new field with valid data:

```json
{
  "scores": {
    "open_source": {"score": 30, "max": 35, "evidence": "..."},
    "self_projects": {"score": 22, "max": 30, "evidence": "..."},
    "production": {"score": 18, "max": 25, "evidence": "..."},
    "technical_skills": {"score": 9, "max": 10, "evidence": "..."},
    "community_engagement": {"score": 13, "max": 15, "evidence": "Active Stack Overflow contributor with 500+ reputation"}
  },
  "bonus_points": {"total": 6, "breakdown": "..."},
  "deductions": {"total": 2, "reasons": "..."}
}

```

If the field is missing or null, verify that the Jinja template indentation is correct and that the JSON schema in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) matches the field name expected by the prompt.

## Summary

- **Modify the Jinja template** at `prompts/templates/resume_evaluation_criteria.jinja` to add new scoring rules for the LLM to evaluate.
- **Extend the Pydantic model** in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) only when introducing new top-level score categories that require structured storage.
- **Update constants** in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) such as `MAX_BONUS_POINTS` and `MAX_FINAL_SCORE` if new rules alter the scoring boundaries.
- **Test with [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py)** to ensure the LLM returns valid JSON matching your updated schema before deploying changes.

## Frequently Asked Questions

### Do I need to modify Python code to add new scoring rules?

Not necessarily. If you are adjusting existing categories or adding bonus/deduction rules that fit within the current `Scores` structure, you only need to edit the Jinja template. The LLM will follow your new instructions and return the appropriate JSON. You only need to modify [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) when adding entirely new top-level score fields that do not exist in the current schema.

### How does the ResumeEvaluator handle new fields in the LLM response?

The `ResumeEvaluator.evaluate_resume()` method uses the `EvaluationData` Pydantic model to parse and validate the LLM's JSON output. If you add a field to the `Scores` model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), Pydantic automatically extracts and validates the corresponding data from the response. The evaluator requires no additional logic changes because it treats the model as the source of truth for the expected structure.

### What happens if the LLM returns a score outside the defined range?

The Pydantic `CategoryScore` model validates that scores are integers, but the range constraints (e.g., 0-15 points) are enforced primarily through the prompt instructions in the Jinja template. If the LLM returns a value exceeding your specified maximum, the current implementation in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) may still process it unless you add explicit validation logic. For strict enforcement, add Pydantic validators to the `CategoryScore` class or clamp values during post-processing in the evaluator.

### Can I add deduction rules instead of bonus points using the same method?

Yes. The Jinja template already contains a deductions section. You can append new deduction rules—such as penalties for employment gaps or missing documentation—directly to the template. Ensure you update the `deductions` field structure in the response handling if you introduce new deduction categories beyond the simple total, or describe them clearly in the prompt so the LLM includes them in the existing deductions breakdown.