What Causes Resume Score Variance Across Multiple Runs in Hiring Agent

Score variance across multiple runs of the same resume stems primarily from LLM non-determinism and hidden PDF content, with secondary contributions from stochastic GitHub project selection and caching behavior.

When using the interviewstreet/hiring-agent pipeline to evaluate candidate résumés, you may notice the final score fluctuating between executions despite using identical input files. This instability originates from specific implementation details in the extraction and evaluation modules. Understanding these root causes allows you to configure the pipeline for deterministic, reproducible results.

Root Causes of Score Variance

The pipeline introduces variability through four main mechanisms located in the core orchestration and parsing modules.

LLM Non-Determinism

The large language model responsible for extracting résumé sections, selecting GitHub projects, and applying the scoring rubric operates with default sampling parameters. In models.py, both OllamaProvider and GeminiProvider call their respective APIs without setting temperature to 0, allowing the model to sample probabilistically from the output distribution. Each invocation can therefore return slightly different extractions or evaluations, directly propagating to score differences in evaluator.py.

Hidden PDF Content

The PDF-to-Markdown conversion in pymupdf_rag.py captures invisible or zero-width text embedded within résumé files. When this stray content—such as hidden skill keywords or obfuscated project names—is parsed as valid data, the LLM may artificially inflate category scores. Because the invisible text extraction is non-deterministic across runs, the interpreted content varies, causing score fluctuations.

GitHub Project Selection Variability

In github.py, the LLM is prompted to select exactly seven projects from a candidate's repository list. Since this selection relies on the same non-deterministic model calls, the specific set of projects chosen (and their associated commit counts or complexity metrics) can differ between runs, altering the final calculation in score.py.

Development Mode Caching

When DEVELOPMENT_MODE=True, intermediate JSON results are cached to disk. If the cache is cleared or modified between executions, the pipeline re-processes the PDF and GitHub data, triggering fresh LLM calls and generating new scores that diverge from previous cached results.

How to Eliminate Score Variance

You can enforce deterministic behavior by modifying three specific components of the codebase.

Force Deterministic LLM Output

Set the temperature parameter to 0 in both provider implementations within models.py to remove sampling randomness.

For the Ollama provider:

# models.py – OllamaProvider

class OllamaProvider(BaseProvider):
    def _call(self, messages: List[Dict[str, str]]) -> str:
        payload = {
            "model": os.getenv("DEFAULT_MODEL"),
            "messages": messages,
            "temperature": 0,  # Deterministic sampling

        }
        # Existing request logic

For the Gemini provider:

# models.py – GeminiProvider

class GeminiProvider(BaseProvider):
    def _call(self, messages: List[Dict[str, str]]) -> str:
        request = {
            "model": os.getenv("DEFAULT_MODEL"),
            "temperature": 0,  # Deterministic sampling

            "messages": messages,
        }
        # Send request logic

Sanitize PDF Input

Add a post-processing step in pdf.py to strip invisible Unicode characters after extraction. This prevents zero-width spaces and hidden markers from influencing the LLM's interpretation.

# pdf.py – after converting pages to markdown

def _clean_text(text: str) -> str:
    # Remove zero-width spaces, joiners, and byte order marks

    invisible = ["\u200B", "\u200C", "\u200D", "\uFEFF"]
    for char in invisible:
        text = text.replace(char, "")
    return text

# In PDFHandler.to_markdown(...)

raw_md = self._to_markdown(page)
clean_md = _clean_text(raw_md)

Stabilize Project Selection

Modify github.py to sort projects by a stable metric before slicing to the top seven, removing the dependency on LLM ordering.

# github.py – after LLM returns candidate projects

def _rank_projects(projects: List[Dict]) -> List[Dict]:
    # Sort by commit count descending, then by name ascending

    return sorted(projects, key=lambda p: (-p.get("commit_count", 0), p["name"]))

# When selecting the final 7:

candidates = self._rank_projects(llm_response)
selected = candidates[:7]  # Deterministic top-7

Summary

• LLM temperature settings in models.py cause probabilistic output variations that directly impact scoring consistency.
• Invisible PDF content parsed by pymupdf_rag.py introduces unpredictable data points that can inflate scores arbitrarily.
• Stochastic project selection in github.py changes the evaluation subset between runs when the LLM chooses different repositories.
• Setting temperature=0, sanitizing extracted text for Unicode characters \u200B through \uFEFF, and sorting projects by commit count before selection will eliminate virtually all score variance.

Frequently Asked Questions

Why does the same resume get different scores each time I run it?

The variance occurs because the Hiring Agent pipeline uses non-deterministic LLM sampling by default, allowing the model to generate slightly different extractions and evaluations on each call. Additionally, hidden text in PDF source files may be parsed inconsistently across runs, and the GitHub project selector may choose different repositories due to the same probabilistic behavior.

Which file controls the LLM temperature settings?

The temperature parameter is configured in models.py within the provider classes OllamaProvider and GeminiProvider. By default, these classes do not explicitly set temperature=0, which allows the underlying API to use its default stochastic sampling settings.

Can invisible text in PDFs really affect scoring?

Yes. The pymupdf_rag.py converter captures all embedded text including zero-width spaces and hidden layers. When this invisible content contains skill keywords or project references, the LLM in evaluator.py interprets them as valid candidate attributes, potentially increasing category scores artificially. Since extraction of these hidden characters varies between runs, scores fluctuate accordingly.

How do I make the GitHub project selection deterministic?

Modify github.py to sort the LLM-returned project list by objective metrics—such as commit count and repository name—before selecting the top seven entries. This replaces the stochastic LLM ordering with a stable, reproducible ranking that produces identical selections across multiple executions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}