# How to Test the Evaluation Pipeline with Sample Resumes in Hiring-Agent > Test the hiring agent evaluation pipeline with sample resumes. Run python score.py to execute the full flow from extraction to scoring and cache intermediate results for easy inspection. - Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent) - Tags: testing - Published: 2026-06-27 --- **You can test the hiring-agent evaluation pipeline by running `python score.py ` against any sample PDF, which executes the full flow from PDF extraction through GitHub enrichment to final scoring while caching intermediate results for inspection.** The evaluation pipeline in the `interviewstreet/hiring-agent` repository is a fully automated end-to-end system that transforms raw PDF resumes into structured, scored evaluations. Understanding how to test this pipeline with sample resumes allows you to validate each processing stage—from initial text extraction to LLM-based scoring—without relying on production data. ## Understanding the Pipeline Architecture Before testing, you should understand the seven-stage flow implemented in the source code: 1. **PDF Ingestion** - [`pymupdf_rag.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pymupdf_rag.py) uses PyMuPDF to convert each page into a Markdown-ish string. 2. **Structured Parsing** - [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) sends the Markdown to an LLM using Jinja templates stored in `prompts/templates/*.jinja`, returning JSON-Resume style objects. 3. **GitHub Enrichment** - [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) fetches profile and repository data when a GitHub URL is detected, classifying projects and selecting the top 7. 4. **Evaluation Scoring** - [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) loads the resume evaluation criteria template, builds a chat prompt, and parses the structured `EvaluationData` response using `extract_json_from_response`. 5. **Orchestration** - [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) wires all stages together, handling caching, CSV export (when `DEVELOPMENT_MODE=True`), and pretty printing. 6. **Model abstraction** - [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) defines Pydantic schemas (`JSONResume`, `EvaluationData`) and provider wrappers. 7. **Data transformation** - [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) converts raw JSON into printable text and CSV rows. ## Prerequisites and Environment Setup First, clone the repository and install dependencies: ```bash git clone https://github.com/interviewstreet/hiring-agent cd hiring-agent python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements.txt ``` Configure your environment by copying the example file: ```bash cp .env.example .env ``` Edit `.env` to select your LLM provider. For local testing without external APIs, use Ollama: ```bash # .env LLM_PROVIDER=ollama DEFAULT_MODEL=gemma3:4b DEVELOPMENT_MODE=True ``` Set `DEVELOPMENT_MODE=True` to enable caching and CSV export during testing. ## Creating a Sample Resume PDF You need a PDF file to test the pipeline. Create a minimal test resume using Python's ReportLab: ```python # create_test_resume.py from reportlab.lib.pagesizes import LETTER from reportlab.pdfgen import canvas c = canvas.Canvas("test_resume.pdf", pagesize=LETTER) c.setFont("Helvetica", 12) c.drawString(72, 720, "Jane Smith") c.drawString(72, 700, "Senior Software Engineer") c.drawString(72, 680, "Email: jane.smith@example.com") c.drawString(72, 660, "GitHub: https://github.com/janesmith") c.drawString(72, 640, "Skills: Python, Rust, Kubernetes") c.drawString(72, 620, "Experience:") c.drawString(90, 600, "TechCorp – Staff Engineer (2019-2024)") c.showPage() c.save() ``` Run the script to generate your test file: ```bash python create_test_resume.py ``` This creates `test_resume.pdf`, which contains structured data including a GitHub URL to trigger the enrichment stage in [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py). ## Running the Full Evaluation Pipeline Execute the pipeline using the CLI entry point in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py): ```bash python score.py test_resume.pdf ``` The orchestration flow in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) performs the following actions: - Extracts text via [`pymupdf_rag.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pymupdf_rag.py) and caches the result to [`cache/resumecache_test_resume.json`](https://github.com/interviewstreet/hiring-agent/blob/main/cache/resumecache_test_resume.json) - Detects the GitHub URL and fetches profile data via [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py), caching to [`cache/githubcache_test_resume.json`](https://github.com/interviewstreet/hiring-agent/blob/main/cache/githubcache_test_resume.json) - Builds evaluation prompts using [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) and the templates in `prompts/templates/resume_evaluation_criteria.jinja` - Outputs scores to the console and appends results to `resume_evaluations.csv` (because `DEVELOPMENT_MODE=True`) ## Inspecting Intermediate Artifacts Because the pipeline caches intermediate results, you can inspect the JSON structures at each stage without re-running the LLM calls. View the parsed resume structure: ```bash cat cache/resumecache_test_resume.json | jq . ``` View the fetched GitHub data: ```bash cat cache/githubcache_test_resume.json | jq . ``` To re-run the full pipeline fresh, delete the cache files: ```bash rm cache/resumecache_test_resume.json cache/githubcache_test_resume.json ``` Subsequent runs without deletion will use cached data, making them nearly instant. ## Configuring Alternative LLM Providers While Ollama works offline, you can test with Google's Gemini for comparison. Update your `.env`: ```bash LLM_PROVIDER=gemini DEFAULT_MODEL=gemini-2.5-pro GEMINI_API_KEY=YOUR_KEY_HERE ``` Run the pipeline again: ```bash python score.py test_resume.pdf ``` The provider logic in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py) automatically routes requests to the appropriate backend based on `LLM_PROVIDER` and `DEFAULT_MODEL` environment variables. ## Summary Testing the hiring-agent evaluation pipeline requires only a sample PDF and the [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) CLI command: - **Environment**: Set `DEVELOPMENT_MODE=True` to enable caching and CSV output in [`config.py`](https://github.com/interviewstreet/hiring-agent/blob/main/config.py). - **Sample Data**: Create a PDF with ReportLab or use any existing resume containing a GitHub URL to trigger full enrichment. - **Execution**: Run `python score.py ` to trigger the chain through [`pymupdf_rag.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pymupdf_rag.py), [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py), and [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py). - **Validation**: Inspect `cache/resumecache_*.json` and `cache/githubcache_*.json` to verify extraction and enrichment stages. - **Iteration**: Delete cache files to force re-processing, or rely on caching for rapid iterative testing. ## Frequently Asked Questions ### How do I verify that the GitHub enrichment stage is working? Check for the existence of `cache/githubcache_.json` after running [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py). If this file contains repository data and profile information, the [`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py) module successfully extracted the URL from your PDF and queried the GitHub API. If the file is missing or empty, verify that your sample resume includes a valid `https://github.com/` URL format. ### Can I test the pipeline without an internet connection? Yes. Set `LLM_PROVIDER=ollama` in your `.env` file and ensure you have the Ollama server running locally with your chosen model (e.g., `gemma3:4b`). According to the [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) implementation, the Ollama provider routes requests to `localhost:11434`, requiring no external API calls. However, GitHub enrichment requires internet access unless you manually create a cached `githubcache_*.json` file before running. ### What file handles the final scoring output format? The [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) file builds the evaluation prompt and parses the LLM response into an `EvaluationData` Pydantic model. The [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py) module then converts this structured data into both console-friendly text and CSV rows. When `DEVELOPMENT_MODE=True`, [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) appends results to `resume_evaluations.csv` using the helpers in [`transform.py`](https://github.com/interviewstreet/hiring-agent/blob/main/transform.py). ### How do I debug failures in the resume parsing stage? Inspect the intermediate output in `cache/resumecache_.json` to see the raw JSON produced by the LLM in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py). If this file is malformed or missing sections, check the Jinja templates in `prompts/templates/` to ensure they match the expected schema defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) for `JSONResume`. You can also enable verbose logging or modify [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) to print the raw Markdown output from [`pymupdf_rag.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pymupdf_rag.py) before it reaches the LLM.