Troubleshooting Common hiring-agent Issues: 5-Stage Pipeline Fixes

Most hiring-agent failures stem from PDF extraction errors, LLM misconfiguration, or stale development caches in the five-stage pipeline spanning pymupdf_rag.py to score.py.

The interviewstreet/hiring-agent repository transforms resume PDFs into structured evaluations through a multi-stage LLM pipeline. When the pipeline breaks, symptoms often appear far from the root cause in pdf.py or evaluator.py. Understanding the architectural flow—from PyMuPDF extraction to fairness-aware scoring—lets you isolate failures quickly.

PDF Extraction Failures in pymupdf_rag.py

The pipeline begins in pymupdf_rag.py, where the PDFExtractor class converts documents to Markdown-like text. Failures here cascade downstream, causing malformed section parsing or empty LLM prompts.

Corrupt or Unsupported PDF Formats

If you encounter RuntimeError: cannot open file in pymupdf_rag.py, the PDF is likely corrupted or uses an unsupported spec. Verify the file opens in a standard viewer, then convert it to a stable PDF/A format using pdftk or similar tools. Ensure the file path is absolute and the process has read permissions.

Missing Text from Custom Fonts

When headings or tables disappear from the generated Markdown, the PDF likely uses custom fonts or image-based text that PyMuPDF cannot interpret. Enable image extraction and OCR fallback:

from pymupdf_rag import PDFExtractor

pdf_path = "resume.pdf"
try:
    extractor = PDFExtractor(pdf_path, extract_images=True)
    markdown = extractor.to_markdown()
    print(f"Extracted {len(markdown)} characters")
except Exception as e:
    raise RuntimeError(f"Failed to open PDF: {e}")

Adjust the to_markdown helper in pymupdf_rag.py to treat non-standard heading markers as level-2 headings if the document uses unusual typography.

LLM Provider Configuration Errors

The LLM provider layer in models.py and llm_utils.py bridges the pipeline to Gemini, Ollama, or other backends. Misconfiguration here produces AuthenticationError or empty JSON responses that break the pdf.py section parser.

Authentication Failures with Gemini or Ollama

Check that .env contains valid variables before activating the virtual environment:

# .env

LLM_PROVIDER=gemini
DEFAULT_MODEL=gemini-1.5-flash
GEMINI_API_KEY=your_key_here

Reload the environment with source .venv/bin/activate and verify connectivity:

from llm_utils import get_provider
provider = get_provider()
print(f"Using provider: {provider.__class__.__name__}")

Malformed JSON from Template Mismatches

When pdf.py returns unexpected JSON structures, the Jinja templates in prompts/templates/ likely diverge from the LLM's expected schema. Reinstall dependencies to ensure Jinja2 compatibility, then diff your local templates against the repository's README examples.

GitHub Enrichment Failures in github.py

The GitHubClient class in github.py fetches profile data and selects the top 7 repositories via LLM-driven analysis. This stage fails silently or with KeyError when API limits or schema changes occur.

Rate Limiting and Missing Tokens

If no repositories appear despite a valid username in the resume, check for a missing GITHUB_TOKEN or rate-limit exhaustion. Create a personal access token with repo scope, export it in .env, and wait for GitHub's hourly rate limit reset if you've exceeded 60 unauthenticated requests.

API Response Structure Changes

A KeyError: 'login' indicates GitHub modified their API response format. Update github.py to use the new field name or fallback to html_url:

from github import GitHubClient

client = GitHubClient()
repos = client.get_user_repos("octocat")
print(f"Fetched {len(repos)} repositories")

Run the repository-fetching unit tests in tests/ to confirm the response shape matches the expected Pydantic models.

Evaluation and Scoring Anomalies in evaluator.py

The Evaluator class in evaluator.py applies fairness-aware scoring rules using LLM-generated evidence. Template corruption or logic drift causes missing categories or incorrect calculations.

Missing Scoring Categories

If "open_source" or "production" categories disappear from the output, the Jinja template (resume_evaluation_criteria.jinja) was edited incorrectly. Restore the original template from version control and run a dry evaluation:

from evaluator import Evaluator

evaluator = Evaluator()
result = evaluator.evaluate(json_resume, dry_run=True)
print(result.scores)

Incorrect Bonus Calculations

When bonus points or deductions seem mathematically wrong, compare the current evaluator.py against the main branch. The fairness constraints for bonus_points and deductions may have been altered, requiring you to re-apply the original weighting logic.

Development Mode Caching Issues

When DEVELOPMENT_MODE=True in config.py, the pipeline caches intermediate JSON and appends results to resume_evaluations.csv via score.py. This creates two distinct failure modes.

Stale Cache Artifacts

If results appear outdated despite PDF changes, delete cached files to force reprocessing:

import glob, os

for path in glob.glob("cache/resumecache_*.json"):
    os.remove(path)
print("Cache cleared")

Set DEVELOPMENT_MODE=False in config.py for production runs that bypass the cache entirely.

CSV Export Permission Errors

A PermissionError when writing to resume_evaluations.csv indicates file-system restrictions. Change permissions with chmod 664 resume_evaluations.csv or execute the script with a user that has write access to the working directory.

Summary

• PDF extraction fails in pymupdf_rag.py due to corrupted files or non-standard fonts—verify files and enable OCR when needed.
• LLM configuration requires correct .env variables (LLM_PROVIDER, GEMINI_API_KEY) and matching Jinja templates in prompts/templates/.
• GitHub enrichment needs a valid GITHUB_TOKEN to avoid rate limits and may require field name updates if the API changes.
• Evaluation anomalies in evaluator.py stem from corrupted templates or modified fairness logic—use dry_run=True to debug.
• Development mode caches results in cache/ and writes to CSV; clear caches and check permissions when results seem stale.

Frequently Asked Questions

Why does hiring-agent return empty JSON for some resume sections?

Empty JSON usually indicates the LLM provider timed out or the prompt template in prompts/templates/ references a deprecated schema. Verify your LLM_PROVIDER and DEFAULT_MODEL environment variables, then compare your local templates against the repository's README examples to ensure field alignment.

How do I fix GitHub API rate limits when processing multiple candidates?

Create a personal access token with repo scope and set it as GITHUB_TOKEN in your .env file. Without authentication, GitHub limits you to 60 requests per hour; with a token, the limit increases to 5,000 requests per hour. If you hit the limit, wait for the hourly reset or implement exponential backoff in github.py.

What causes incorrect scoring calculations in the final evaluation?

Incorrect calculations typically result from manual edits to evaluator.py that altered the bonus_points or deductions logic, or from a corrupted resume_evaluation_criteria.jinja template. Run evaluator.py --dry-run (or call evaluate() with dry_run=True) to inspect intermediate scores, then restore the original template from the main branch if categories are missing.

How do I clear the development cache to force a fresh resume analysis?

Delete the JSON files in the cache/ directory or set DEVELOPMENT_MODE=False in config.py. For a one-time reset, run rm cache/*.json before executing score.py. This ensures PyMuPDF re-extracts the PDF and the LLM reprocesses all sections rather than loading stale intermediate results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}