# How to Test Evaluation Logic Without Processing Real Resumes in Hiring-Agent

> Easily test hiring-agent evaluation logic without real resumes. Inject mock LLM providers and use minimal resume text strings for fast, accurate validation. Avoid live model calls.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: testing
- Published: 2026-07-22

---

**You can validate changes to the interviewstreet/hiring-agent evaluation pipeline by injecting a mock LLM provider into `ResumeEvaluator` and passing minimal résumé text strings, eliminating the need for real PDF files or live model calls.**

The hiring-agent system turns candidate résumés into plain text, forwards that text to a language model, and parses the JSON response into a strongly-typed `EvaluationData` object. If you are iterating on scoring rules or prompt logic, you do not need to run the full PDF extraction stack. Instead, you can test evaluation logic without processing real resumes by isolating the `ResumeEvaluator` class and substituting the LLM backend with a deterministic stub.

## How the Evaluation Pipeline Is Structured

The core flow is implemented across three key files. In [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py), the `ResumeEvaluator` class builds the evaluation prompt and calls the LLM provider. In [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), the `EvaluationData` Pydantic model enforces the schema for scores, bonus points, deductions, and qualitative feedback. The provider itself is wired through [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) via `initialize_llm_provider`, which selects between Ollama and Gemini based on `MODEL_PROVIDER_MAPPING`.

Key implementation points from the source code include:

- `ResumeEvaluator.__init__` loads the prompt template through `TemplateManager` and stores the selected provider.
- `evaluate_resume` assembles the chat payload, invokes `self.provider.chat`, extracts the JSON block, and validates it against `EvaluationData`.
- `initialize_llm_provider` returns a concrete client with a `chat` method, making the system easy to swap for testing.

Because the provider is injected rather than hard-coded, you can replace the real network client with a lightweight stub that returns a fixed JSON payload.

## Unit Testing `ResumeEvaluator` with a Mock Provider

To test the evaluation step directly, instantiate `ResumeEvaluator` (or a subclass), override the `provider` attribute with a fake object that mirrors the `chat` signature, and call `evaluate_resume` with a minimal string. This approach skips PDF parsing, network traffic, and rate limits.

The fake provider below subclasses `OllamaProvider` and returns a deterministic assistant message that serializes to valid `EvaluationData`.

```python
import json
from evaluator import ResumeEvaluator
from llm_utils import OllamaProvider

class FakeProvider(OllamaProvider):
    """A stub that returns a fixed JSON payload instead of calling a real LLM."""
    def chat(self, *_, **__) -> dict:
        fake_response = {
            "message": {
                "role": "assistant",
                "content": json.dumps({
                    "scores": {
                        "open_source": {"score": 30, "max": 35, "evidence": "Contributed to 5 repos"},
                        "self_projects": {"score": 25, "max": 30, "evidence": "Built 3 apps"},
                        "production": {"score": 20, "max": 25, "evidence": "2 years at XYZ"},
                        "technical_skills": {"score": 8, "max": 10, "evidence": "Python, SQL"}
                    },
                    "bonus_points": {"total": 15, "breakdown": "Open‑source leadership"},
                    "deductions": {"total": 0, "reasons": ""},
                    "key_strengths": ["Team player", "Quick learner"],
                    "areas_for_improvement": ["Public speaking"]
                })
            }
        }
        return fake_response

def test_evaluate_resume_returns_structured_data():
    # Arrange: a tiny résumé text that satisfies the prompt template

    sample_resume = "John Doe – Software Engineer with 2 years experience. Open‑source contributor."
    
    # Inject the fake provider

    evaluator = ResumeEvaluator(model_name="llama3", model_params={"temperature": 0})
    evaluator.provider = FakeProvider()   # replace the real LLM client

    
    # Act

    result = evaluator.evaluate_resume(sample_resume)
    
    # Assert: the result respects the EvaluationData schema

    assert result.scores.open_source.score == 30
    assert result.bonus_points.total == 15
    assert "Team player" in result.key_strengths

```

This pattern demonstrates how to test evaluation logic without processing real resumes. You assert on concrete fields inside the returned `EvaluationData` instance—such as `scores.open_source.score`, `bonus_points.total`, and `key_strengths`—while the mock guarantees the same payload every run.

## Testing Higher-Level Flows in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py)

The same substitution strategy works for orchestration code. In [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py), functions such as `_evaluate_resume` accept a `JSONResume` object, extract or enrich its text, and delegate to `ResumeEvaluator`. You can exercise this layer by building a fabricated `JSONResume` instance from [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) and monkeypatching the evaluator class.

```python
from score import _evaluate_resume
from models import JSONResume

def test__evaluate_resume_uses_mock_evaluator(monkeypatch):
    # Build a minimal JSONResume instance (only what the function inspects)

    resume = JSONResume(basics=None, work=None, education=None, skills=None, projects=None)
    
    # Patch the ResumeEvaluator to use our FakeProvider (same as above)

    from evaluator import ResumeEvaluator
    class FakeEvaluator(ResumeEvaluator):
        def __init__(self, *_, **__):
            self.provider = FakeProvider()
        def evaluate_resume(self, *_, **__):
            return super().evaluate_resume("dummy text")
    
    monkeypatch.setattr('score.ResumeEvaluator', FakeEvaluator)
    
    # Call the function – it will hit the mock and return a deterministic EvaluationData

    evaluation = _evaluate_resume(resume)
    
    assert evaluation is not None
    assert evaluation.deductions.total == 0

```

By intercepting `ResumeEvaluator` at the module boundary, you verify that CSV formatting, printing, and other downstream consumers in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) behave correctly when given a controlled `EvaluationData` object.

## Key Files Involved in Evaluation Testing

Understanding the following files helps you decide where to place mocks and assertions:

- [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) — Houses `ResumeEvaluator.__init__` and `evaluate_resume`, the primary methods to target during unit testing.
- [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) — Defines `EvaluationData` and `JSONResume`, the Pydantic schemas that enforce data contracts.
- [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) — Contains `initialize_llm_provider`, the helper that selects Ollama or Gemini; mimicking its output is the fastest way to remove external dependencies.
- [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) — Orchestrates PDF extraction, optional enrichment, and evaluator invocation; patch here for integration-style tests.
- [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) — Loads Jinja templates used by `ResumeEvaluator`; relevant if you are testing prompt changes.
- [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py) — Stores default model names and `MODEL_PROVIDER_MAPPING`.

## Summary

- **Inject a mock LLM provider** to bypass live network calls and rate limits when testing `ResumeEvaluator` in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py).
- **Use minimal résumé strings** instead of real PDFs; the prompt only requires plaintext that satisfies the template variables.
- **Assert on `EvaluationData` fields** such as `scores`, `bonus_points`, `deductions`, and `key_strengths` to confirm parsing and schema validation.
- **Monkeypatch `ResumeEvaluator` in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py)** to test higher-level orchestration with fabricated `JSONResume` objects.
- **Avoid the PDF pipeline** entirely because `evaluate_resume` and `_evaluate_resume` operate on strings or structured models already.

## Frequently Asked Questions

### How do I avoid PDF extraction when testing evaluation logic?

The `evaluate_resume` method in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) accepts a plain string. Pass a short résumé paragraph directly and skip the PDF-to-text stage. For `_evaluate_resume` in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py), construct a lightweight `JSONResume` from [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) so the function never invokes the file parser.

### Can I test prompt changes without calling a real LLM?

Yes. Create a `FakeProvider` that returns a JSON string matching the fields in `EvaluationData`. Update the fake payload whenever you add new scoring rubric fields, then run assertions against those fields. This lets you iterate on prompts and schema in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) without incurring inference costs.

### What is the fastest way to mock the LLM provider?

Subclass `OllamaProvider` (or match the `chat` protocol from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)) and override `chat` to return a dictionary with a serialized assistant message. Assign the mock instance to `evaluator.provider` after construction. This replaces the provider selected by `initialize_llm_provider` without changing upstream configuration code.

### How do I test the full flow from [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) without real resumes?

Use `pytest`'s `monkeypatch` to replace `score.ResumeEvaluator` with a fake subclass that pre-configures `FakeProvider`. Feed the patched flow a minimal `JSONResume` object. The function will execute enrichment and output formatting logic while still returning deterministic `EvaluationData`.