# How LLM Non‑Determinism Affects Resume Scoring Consistency in Hiring‑Agent

> Learn how LLM non-determinism impacts resume scoring consistency. Discover how fluctuating scores and rankings arise from identical prompts in the hiring agent. Understand the technical causes.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: performance
- Published: 2026-07-22

---

**LLM non‑determinism introduces score variance by causing the language model to generate different token sequences for identical prompts, which propagates through the JSON parsing logic in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) to produce fluctuating category scores and inconsistent candidate rankings.**

The interviewstreet/hiring-agent repository implements an AI‑powered resume evaluation system that relies on large language models to score candidate qualifications across categories like technical skills and open‑source contributions. Because the system parses structured JSON responses from the LLM to calculate final scores in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py), any stochastic variation in model output directly impacts resume scoring consistency. Understanding how temperature settings and sampling parameters influence this pipeline is essential for maintaining reliable hiring decisions.

## The Mechanism of LLM Non‑Determinism in evaluator.py

In [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py), the `ResumeEvaluator` class constructs chat parameters that control the LLM's randomness during the evaluation request:

```python
chat_params = {
    "model": self.model_name,
    "messages": [
        {"role": "system", "content": system_message},
        {"role": "user",   "content": full_prompt},
    ],
    "options": {
        "stream": False,
        "temperature": self.model_params.get("temperature", 0.5),   # ← temperature

        "top_p":       self.model_params.get("top_p", 0.9),        # ← top‑p

    },
}

```

([source line 66‑71](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py#L66))

The **temperature** parameter determines whether the model selects the highest‑probability token (temperature `0.0`) or samples from a distribution of possible tokens (temperature `> 0.0`). When temperature exceeds zero, random sampling introduces different text sequences across identical prompts, directly affecting the structured `EvaluationData` returned to the system.

## Default Temperature Configuration by Model

Default stochasticity levels vary by model configuration defined in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py):

```python
MODEL_PARAMETERS = {
    "qwen3:1.7b": {"temperature": 0.0, "top_p": 0.9},
    "gemma3:4b":  {"temperature": 0.1, "top_p": 0.9},
    "gemini-2.5-flash": {"temperature": 0.1, "top_p": 0.9},
    # …

}

```

([source line 27‑44](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py#L27))

The repository configures `qwen3:1.7b` with a deterministic temperature of `0.0`, while `gemma3:4b` and `gemini-2.5-flash` default to `0.1`. This means switching between models without adjusting parameters automatically changes resume scoring consistency, with higher temperatures producing greater variance in category scores such as technical skills or open‑source contributions.

## Propagation of Variance to Final Scores

The final numeric score calculation in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) aggregates individual category scores from the LLM‑generated JSON:

```python
total_score = 0
if hasattr(evaluation, "scores"):
    for cat, data in evaluation.scores.model_dump().items():
        total_score += min(data["score"], data["max"])

```

([source line 52‑56](https://github.com/interviewstreet/hiring-agent/blob/main/score.py#L52))

Because `evaluation.scores` derives directly from the LLM's parsed response, any stochastic variation in the model's JSON output causes immediate fluctuations in `total_score`. A candidate might receive **7.5** for technical skills on one evaluation and **8.2** on another, solely due to non‑deterministic token sampling rather than actual resume content differences.

## Practical Impact on Resume Evaluation

When LLM non‑determinism affects the evaluation pipeline, organizations encounter three primary consistency issues:

- **Fluctuating raw scores** – Individual category scores vary between evaluation runs for identical resume text, causing the "Technical Skills" or "Production" scores to shift randomly.
- **Inconsistent bonus handling** – The evaluator may add or subtract bonus points based on wording variations that shift with random sampling, even when the underlying experience remains constant.
- **Ranking instability** – Small stochastic differences across thousands of resumes can reorder candidate rankings, potentially changing who advances in the hiring process.

## Code Examples: Controlling Scoring Consistency

### Running with Default Stochastic Parameters

When using the default configuration, expect variance between runs:

```python
from evaluator import ResumeEvaluator
from pdf import PDFExtractor

pdf_path = "resume/sample.pdf"
pdf_extractor = PDFExtractor()
resume_text = pdf_extractor.extract_text_from_pdf(pdf_path)

# Uses default model (gemma3:4b, temperature 0.1)

evaluator = ResumeEvaluator()
evaluation = evaluator.evaluate_resume(resume_text)

print(evaluation.scores.technical_skills.score)   # May differ on each run

```

### Enforcing Deterministic Scoring

To eliminate variance and ensure resume scoring consistency, override the temperature parameter:

```python
from evaluator import ResumeEvaluator

# Override temperature to 0.0 for deterministic output

custom_params = {"temperature": 0.0, "top_p": 0.9}
evaluator = ResumeEvaluator(model_name="gemma3:4b", model_params=custom_params)

evaluation = evaluator.evaluate_resume(resume_text)
print(evaluation.scores.technical_skills.score)   # Same value on every run

```

### Debugging Raw LLM Responses

Inspect the raw JSON to identify variance sources:

```python
evaluator = ResumeEvaluator()
response = evaluator.provider.chat(
    model="gemma3:4b",
    messages=[{"role": "system", "content": "..."},
              {"role": "user",   "content": "..."}],
    options={"temperature": 0.1, "top_p": 0.9},
    format=evaluator.provider.EvaluationData.model_json_schema()
)
print(response["message"]["content"])   # Raw JSON string from LLM

```

## Architectural Factors Amplifying Variability

Several design decisions in the hiring‑agent codebase interact with LLM non‑determinism to affect resume scoring consistency:

**Prompt Templating** – The `TemplateManager` class injects resume text into static evaluation criteria templates. Minor whitespace variations in the input combined with stochastic sampling can produce divergent semantic interpretations.

**Structured Output Formatting** – While the system requests JSON using `EvaluationData.model_json_schema()`, some providers inject formatting quirks before the JSON block, which may affect the parser in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) and consequently the scores in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py).

**No Explicit Seed Control** – The current implementation does not expose a random‑seed parameter; therefore, even with a fixed temperature, the underlying service may use internal randomness that prevents perfect reproducibility.

## Summary

- LLM non‑determinism originates from **temperature** and **top_p** parameters configured in [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) and defined per‑model in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py) (lines 27‑44).
- Default temperature varies by model (**0.0** for qwen3:1.7b, **0.1** for gemma3:4b), directly impacting the variance of JSON evaluation outputs.
- The aggregation logic in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) (lines 52‑56) propagates any JSON output variations into final numeric scores without normalization.
- Setting temperature to **0.0** enforces deterministic token selection and ensures consistent resume scoring across multiple runs.
- Lack of seed control and model‑specific defaults require careful configuration management to maintain evaluation reliability when switching between LLM providers.

## Frequently Asked Questions

### Why do I get different scores when evaluating the same resume twice?

When the configured **temperature** is greater than zero (such as the default `0.1` for `gemma3:4b`), the LLM samples tokens probabilistically rather than selecting the highest‑probability option. This causes the model to generate slightly different JSON responses for identical prompts, which [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) then parses into different numeric scores. To eliminate this variance, set `temperature` to `0.0` in the model parameters.

### Which temperature setting guarantees completely deterministic resume scoring?

A **temperature of 0.0** forces the model to select the highest‑probability token at each generation step, producing quasi‑deterministic output. However, complete determinism also depends on the specific LLM provider implementation, as some services may introduce internal randomness or hardware‑dependent floating‑point variations even at zero temperature.

### How can I verify that my LLM configuration is causing score variance?

Compare the raw JSON responses from multiple evaluation runs by logging the output of `evaluator.provider.chat()` before parsing. If the `message.content` strings differ between runs for identical resume text, the variance originates from LLM non‑determinism. If the JSON content is identical but scores differ, investigate the parsing logic in [`score.py`](https://github.com/interviewstreet/hiring-agent/blob/main/score.py) or [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) instead.

### Does switching from gemma3:4b to qwen3:1.7b improve scoring consistency?

Yes, according to the `MODEL_PARAMETERS` definition in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py), `qwen3:1.7b` defaults to **temperature 0.0** while `gemma3:4b` uses **0.1**. Switching to `qwen3:1.7b` without overriding parameters will immediately reduce stochastic variance because the model generates tokens deterministically by default. However, always verify the current configuration in [`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py) as defaults may change across repository versions.