How to Test Evaluation Logic Without Processing Real Resumes in Hiring-Agent

You can validate changes to the interviewstreet/hiring-agent evaluation pipeline by injecting a mock LLM provider into ResumeEvaluator and passing minimal résumé text strings, eliminating the need for real PDF files or live model calls.

The hiring-agent system turns candidate résumés into plain text, forwards that text to a language model, and parses the JSON response into a strongly-typed EvaluationData object. If you are iterating on scoring rules or prompt logic, you do not need to run the full PDF extraction stack. Instead, you can test evaluation logic without processing real resumes by isolating the ResumeEvaluator class and substituting the LLM backend with a deterministic stub.

How the Evaluation Pipeline Is Structured

The core flow is implemented across three key files. In evaluator.py, the ResumeEvaluator class builds the evaluation prompt and calls the LLM provider. In models.py, the EvaluationData Pydantic model enforces the schema for scores, bonus points, deductions, and qualitative feedback. The provider itself is wired through llm_utils.py via initialize_llm_provider, which selects between Ollama and Gemini based on MODEL_PROVIDER_MAPPING.

Key implementation points from the source code include:

• ResumeEvaluator.__init__ loads the prompt template through TemplateManager and stores the selected provider.
• evaluate_resume assembles the chat payload, invokes self.provider.chat, extracts the JSON block, and validates it against EvaluationData.
• initialize_llm_provider returns a concrete client with a chat method, making the system easy to swap for testing.

Because the provider is injected rather than hard-coded, you can replace the real network client with a lightweight stub that returns a fixed JSON payload.

Unit Testing ResumeEvaluator with a Mock Provider

To test the evaluation step directly, instantiate ResumeEvaluator (or a subclass), override the provider attribute with a fake object that mirrors the chat signature, and call evaluate_resume with a minimal string. This approach skips PDF parsing, network traffic, and rate limits.

The fake provider below subclasses OllamaProvider and returns a deterministic assistant message that serializes to valid EvaluationData.

import json
from evaluator import ResumeEvaluator
from llm_utils import OllamaProvider

class FakeProvider(OllamaProvider):
    """A stub that returns a fixed JSON payload instead of calling a real LLM."""
    def chat(self, *_, **__) -> dict:
        fake_response = {
            "message": {
                "role": "assistant",
                "content": json.dumps({
                    "scores": {
                        "open_source": {"score": 30, "max": 35, "evidence": "Contributed to 5 repos"},
                        "self_projects": {"score": 25, "max": 30, "evidence": "Built 3 apps"},
                        "production": {"score": 20, "max": 25, "evidence": "2 years at XYZ"},
                        "technical_skills": {"score": 8, "max": 10, "evidence": "Python, SQL"}
                    },
                    "bonus_points": {"total": 15, "breakdown": "Open‑source leadership"},
                    "deductions": {"total": 0, "reasons": ""},
                    "key_strengths": ["Team player", "Quick learner"],
                    "areas_for_improvement": ["Public speaking"]
                })
            }
        }
        return fake_response

def test_evaluate_resume_returns_structured_data():
    # Arrange: a tiny résumé text that satisfies the prompt template

    sample_resume = "John Doe – Software Engineer with 2 years experience. Open‑source contributor."
    
    # Inject the fake provider

    evaluator = ResumeEvaluator(model_name="llama3", model_params={"temperature": 0})
    evaluator.provider = FakeProvider()   # replace the real LLM client

    
    # Act

    result = evaluator.evaluate_resume(sample_resume)
    
    # Assert: the result respects the EvaluationData schema

    assert result.scores.open_source.score == 30
    assert result.bonus_points.total == 15
    assert "Team player" in result.key_strengths

This pattern demonstrates how to test evaluation logic without processing real resumes. You assert on concrete fields inside the returned EvaluationData instance—such as scores.open_source.score, bonus_points.total, and key_strengths—while the mock guarantees the same payload every run.

Testing Higher-Level Flows in score.py

The same substitution strategy works for orchestration code. In score.py, functions such as _evaluate_resume accept a JSONResume object, extract or enrich its text, and delegate to ResumeEvaluator. You can exercise this layer by building a fabricated JSONResume instance from models.py and monkeypatching the evaluator class.

from score import _evaluate_resume
from models import JSONResume

def test__evaluate_resume_uses_mock_evaluator(monkeypatch):
    # Build a minimal JSONResume instance (only what the function inspects)

    resume = JSONResume(basics=None, work=None, education=None, skills=None, projects=None)
    
    # Patch the ResumeEvaluator to use our FakeProvider (same as above)

    from evaluator import ResumeEvaluator
    class FakeEvaluator(ResumeEvaluator):
        def __init__(self, *_, **__):
            self.provider = FakeProvider()
        def evaluate_resume(self, *_, **__):
            return super().evaluate_resume("dummy text")
    
    monkeypatch.setattr('score.ResumeEvaluator', FakeEvaluator)
    
    # Call the function – it will hit the mock and return a deterministic EvaluationData

    evaluation = _evaluate_resume(resume)
    
    assert evaluation is not None
    assert evaluation.deductions.total == 0

By intercepting ResumeEvaluator at the module boundary, you verify that CSV formatting, printing, and other downstream consumers in score.py behave correctly when given a controlled EvaluationData object.

Key Files Involved in Evaluation Testing

Understanding the following files helps you decide where to place mocks and assertions:

• evaluator.py — Houses ResumeEvaluator.__init__ and evaluate_resume, the primary methods to target during unit testing.
• models.py — Defines EvaluationData and JSONResume, the Pydantic schemas that enforce data contracts.
• llm_utils.py — Contains initialize_llm_provider, the helper that selects Ollama or Gemini; mimicking its output is the fastest way to remove external dependencies.
• score.py — Orchestrates PDF extraction, optional enrichment, and evaluator invocation; patch here for integration-style tests.
• prompts/template_manager.py — Loads Jinja templates used by ResumeEvaluator; relevant if you are testing prompt changes.
• prompt.py — Stores default model names and MODEL_PROVIDER_MAPPING.

Summary

• Inject a mock LLM provider to bypass live network calls and rate limits when testing ResumeEvaluator in evaluator.py.
• Use minimal résumé strings instead of real PDFs; the prompt only requires plaintext that satisfies the template variables.
• Assert on EvaluationData fields such as scores, bonus_points, deductions, and key_strengths to confirm parsing and schema validation.
• Monkeypatch ResumeEvaluator in score.py to test higher-level orchestration with fabricated JSONResume objects.
• Avoid the PDF pipeline entirely because evaluate_resume and _evaluate_resume operate on strings or structured models already.

Frequently Asked Questions

How do I avoid PDF extraction when testing evaluation logic?

The evaluate_resume method in evaluator.py accepts a plain string. Pass a short résumé paragraph directly and skip the PDF-to-text stage. For _evaluate_resume in score.py, construct a lightweight JSONResume from models.py so the function never invokes the file parser.

Can I test prompt changes without calling a real LLM?

Yes. Create a FakeProvider that returns a JSON string matching the fields in EvaluationData. Update the fake payload whenever you add new scoring rubric fields, then run assertions against those fields. This lets you iterate on prompts and schema in evaluator.py without incurring inference costs.

What is the fastest way to mock the LLM provider?

Subclass OllamaProvider (or match the chat protocol from llm_utils.py) and override chat to return a dictionary with a serialized assistant message. Assign the mock instance to evaluator.provider after construction. This replaces the provider selected by initialize_llm_provider without changing upstream configuration code.

How do I test the full flow from score.py without real resumes?

Use pytest's monkeypatch to replace score.ResumeEvaluator with a fake subclass that pre-configures FakeProvider. Feed the patched flow a minimal JSONResume object. The function will execute enrichment and output formatting logic while still returning deterministic EvaluationData.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}