How LLM Non‑Determinism Affects Resume Scoring Consistency in Hiring‑Agent

LLM non‑determinism introduces score variance by causing the language model to generate different token sequences for identical prompts, which propagates through the JSON parsing logic in score.py to produce fluctuating category scores and inconsistent candidate rankings.

The interviewstreet/hiring-agent repository implements an AI‑powered resume evaluation system that relies on large language models to score candidate qualifications across categories like technical skills and open‑source contributions. Because the system parses structured JSON responses from the LLM to calculate final scores in score.py, any stochastic variation in model output directly impacts resume scoring consistency. Understanding how temperature settings and sampling parameters influence this pipeline is essential for maintaining reliable hiring decisions.

The Mechanism of LLM Non‑Determinism in evaluator.py

In evaluator.py, the ResumeEvaluator class constructs chat parameters that control the LLM's randomness during the evaluation request:

chat_params = {
    "model": self.model_name,
    "messages": [
        {"role": "system", "content": system_message},
        {"role": "user",   "content": full_prompt},
    ],
    "options": {
        "stream": False,
        "temperature": self.model_params.get("temperature", 0.5),   # ← temperature

        "top_p":       self.model_params.get("top_p", 0.9),        # ← top‑p

    },
}

(source line 66‑71)

The temperature parameter determines whether the model selects the highest‑probability token (temperature 0.0) or samples from a distribution of possible tokens (temperature > 0.0). When temperature exceeds zero, random sampling introduces different text sequences across identical prompts, directly affecting the structured EvaluationData returned to the system.

Default Temperature Configuration by Model

Default stochasticity levels vary by model configuration defined in prompt.py:

MODEL_PARAMETERS = {
    "qwen3:1.7b": {"temperature": 0.0, "top_p": 0.9},
    "gemma3:4b":  {"temperature": 0.1, "top_p": 0.9},
    "gemini-2.5-flash": {"temperature": 0.1, "top_p": 0.9},
    # …

}

(source line 27‑44)

The repository configures qwen3:1.7b with a deterministic temperature of 0.0, while gemma3:4b and gemini-2.5-flash default to 0.1. This means switching between models without adjusting parameters automatically changes resume scoring consistency, with higher temperatures producing greater variance in category scores such as technical skills or open‑source contributions.

Propagation of Variance to Final Scores

The final numeric score calculation in score.py aggregates individual category scores from the LLM‑generated JSON:

total_score = 0
if hasattr(evaluation, "scores"):
    for cat, data in evaluation.scores.model_dump().items():
        total_score += min(data["score"], data["max"])

(source line 52‑56)

Because evaluation.scores derives directly from the LLM's parsed response, any stochastic variation in the model's JSON output causes immediate fluctuations in total_score. A candidate might receive 7.5 for technical skills on one evaluation and 8.2 on another, solely due to non‑deterministic token sampling rather than actual resume content differences.

Practical Impact on Resume Evaluation

When LLM non‑determinism affects the evaluation pipeline, organizations encounter three primary consistency issues:

• Fluctuating raw scores – Individual category scores vary between evaluation runs for identical resume text, causing the "Technical Skills" or "Production" scores to shift randomly.
• Inconsistent bonus handling – The evaluator may add or subtract bonus points based on wording variations that shift with random sampling, even when the underlying experience remains constant.
• Ranking instability – Small stochastic differences across thousands of resumes can reorder candidate rankings, potentially changing who advances in the hiring process.

Code Examples: Controlling Scoring Consistency

Running with Default Stochastic Parameters

When using the default configuration, expect variance between runs:

from evaluator import ResumeEvaluator
from pdf import PDFExtractor

pdf_path = "resume/sample.pdf"
pdf_extractor = PDFExtractor()
resume_text = pdf_extractor.extract_text_from_pdf(pdf_path)

# Uses default model (gemma3:4b, temperature 0.1)

evaluator = ResumeEvaluator()
evaluation = evaluator.evaluate_resume(resume_text)

print(evaluation.scores.technical_skills.score)   # May differ on each run

Enforcing Deterministic Scoring

To eliminate variance and ensure resume scoring consistency, override the temperature parameter:

from evaluator import ResumeEvaluator

# Override temperature to 0.0 for deterministic output

custom_params = {"temperature": 0.0, "top_p": 0.9}
evaluator = ResumeEvaluator(model_name="gemma3:4b", model_params=custom_params)

evaluation = evaluator.evaluate_resume(resume_text)
print(evaluation.scores.technical_skills.score)   # Same value on every run

Debugging Raw LLM Responses

Inspect the raw JSON to identify variance sources:

evaluator = ResumeEvaluator()
response = evaluator.provider.chat(
    model="gemma3:4b",
    messages=[{"role": "system", "content": "..."},
              {"role": "user",   "content": "..."}],
    options={"temperature": 0.1, "top_p": 0.9},
    format=evaluator.provider.EvaluationData.model_json_schema()
)
print(response["message"]["content"])   # Raw JSON string from LLM

Architectural Factors Amplifying Variability

Several design decisions in the hiring‑agent codebase interact with LLM non‑determinism to affect resume scoring consistency:

Prompt Templating – The TemplateManager class injects resume text into static evaluation criteria templates. Minor whitespace variations in the input combined with stochastic sampling can produce divergent semantic interpretations.

Structured Output Formatting – While the system requests JSON using EvaluationData.model_json_schema(), some providers inject formatting quirks before the JSON block, which may affect the parser in evaluator.py and consequently the scores in score.py.

No Explicit Seed Control – The current implementation does not expose a random‑seed parameter; therefore, even with a fixed temperature, the underlying service may use internal randomness that prevents perfect reproducibility.

Summary

• LLM non‑determinism originates from temperature and top_p parameters configured in evaluator.py and defined per‑model in prompt.py (lines 27‑44).
• Default temperature varies by model (0.0 for qwen3:1.7b, 0.1 for gemma3:4b), directly impacting the variance of JSON evaluation outputs.
• The aggregation logic in score.py (lines 52‑56) propagates any JSON output variations into final numeric scores without normalization.
• Setting temperature to 0.0 enforces deterministic token selection and ensures consistent resume scoring across multiple runs.
• Lack of seed control and model‑specific defaults require careful configuration management to maintain evaluation reliability when switching between LLM providers.

Frequently Asked Questions

Why do I get different scores when evaluating the same resume twice?

When the configured temperature is greater than zero (such as the default 0.1 for gemma3:4b), the LLM samples tokens probabilistically rather than selecting the highest‑probability option. This causes the model to generate slightly different JSON responses for identical prompts, which score.py then parses into different numeric scores. To eliminate this variance, set temperature to 0.0 in the model parameters.

Which temperature setting guarantees completely deterministic resume scoring?

A temperature of 0.0 forces the model to select the highest‑probability token at each generation step, producing quasi‑deterministic output. However, complete determinism also depends on the specific LLM provider implementation, as some services may introduce internal randomness or hardware‑dependent floating‑point variations even at zero temperature.

How can I verify that my LLM configuration is causing score variance?

Compare the raw JSON responses from multiple evaluation runs by logging the output of evaluator.provider.chat() before parsing. If the message.content strings differ between runs for identical resume text, the variance originates from LLM non‑determinism. If the JSON content is identical but scores differ, investigate the parsing logic in score.py or evaluator.py instead.

Does switching from gemma3:4b to qwen3:1.7b improve scoring consistency?

Yes, according to the MODEL_PARAMETERS definition in prompt.py, qwen3:1.7b defaults to temperature 0.0 while gemma3:4b uses 0.1. Switching to qwen3:1.7b without overriding parameters will immediately reduce stochastic variance because the model generates tokens deterministically by default. However, always verify the current configuration in prompt.py as defaults may change across repository versions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}