Common Causes of LLM Extraction Failures and How to Debug Them in Hiring-Agent

LLM extraction failures typically occur when raw model output contains markdown wrappers, XML artifacts,_truncated JSON, or schema mismatches that break the extract_json_from_response helper in llm_utils.py.

The interviewstreet/hiring-agent repository relies heavily on structured JSON extraction from LLM responses to power its PDF parsing, GitHub project selection, and resume evaluation pipelines. When the extract_json_from_response function fails to parse the raw text, the entire pipeline aborts with a JSON decode error. Understanding the specific failure modes in the source code and implementing systematic debugging techniques will make your LLM integrations resilient.

Root Causes of LLM Extraction Failures

Markdown Code Block Wrappers

The most frequent failure occurs when the LLM surrounds JSON with markdown code fences like ```json ... ```. The extract_json_from_response function in llm_utils.py only strips a leading ```json marker and a trailing ```. If the model includes additional text like "Here is the JSON:" or uses different backtick counts, stray characters remain in the payload and cause json.loads to raise a JSONDecodeError.

XML and HTML Artifacts

Some providers embed XML tags (e.g., <thought> or <answer>) within the response content. The extract_json_from_response helper only strips specific markers, so any unparsed XML tags remain inside the string intended for JSON parsing. This commonly appears when system messages encourage chain-of-thought reasoning before the final JSON output.

Truncated or Partial JSON

Token limits or streaming interruptions can cut the LLM output mid-object, resulting in missing closing braces or incomplete string values. This appears frequently in pdf.py during section extraction and in github.py during project selection, where large contexts push against model limits. The json.loads call then fails because the string terminates prematurely.

Schema Mismatches and Hallucinated Keys

The downstream code in evaluator.py uses Pydantic models like EvaluationData(**evaluation_dict) to validate extracted data. When the LLM invents fields not defined in the schema or omits required keys, validation errors surface as generic extraction failures. This happens even when the JSON is syntactically valid but semantically incorrect for the expected model.

Provider-Specific Formatting Quirks

Different providers return responses in incompatible formats. Gemini sometimes returns quoted strings without code blocks, while Ollama may embed extra newline characters. The generic cleaner in llm_utils.py does not handle both uniformly, causing the initialize_llm_provider logic and provider wrappers in models.py to pass malformed text to the extractor.

Ambiguous Prompt Instructions

If the Jinja templates under prompts/templates/ (such as basics.jinja or github_project_selection.jinja) lack explicit instructions to output only JSON, the model adds explanatory text before or after the structured data. Without the directive "Respond with a valid JSON object ONLY, without any surrounding text or markdown," the parser receives conversational content mixed with JSON.

Debugging LLM Extraction Failures

Log Raw LLM Output

Always capture the unmodified response before any post-processing. In evaluator.py around line 80, you can see the pattern for logging the raw content:

response_text = response["message"]["content"]
logger.debug(f"RAW LLM output: {response_text}")

This reveals whether the failure stems from the provider's output format or the cleaning logic.

Verify Cleaning Steps

After calling extract_json_from_response, log the cleaned candidate to confirm the helper function successfully removed all wrappers:

cleaned = extract_json_from_response(response_text)
logger.debug(f"CLEANED for JSON parsing: {cleaned}")

If you see remaining markdown backticks or XML tags, you need to extend the cleaning logic or tighten the prompt.

Implement Re-ask Loops

Add a retry mechanism that modifies the prompt to be stricter when parsing fails:

MAX_RETRIES = 2
for attempt in range(MAX_RETRIES):
    response = provider.chat(**chat_params)
    cleaned = extract_json_from_response(response["message"]["content"])
    try:
        payload = json.loads(cleaned)
        return payload
    except json.JSONDecodeError:
        chat_params["messages"][-1]["content"] += (
            "\nPlease reply **only** with a valid JSON object, no extra text."
        )

This pattern appears in the robust implementation below and prevents single-point failures from aborting the pipeline.

Enforce JSON Mode at Provider Level

For providers that support it, pass the Pydantic schema to enforce JSON generation at the API level. In evaluator.py, the code already demonstrates this for Gemini:

kwargs = {"format": EvaluationData.model_json_schema()}
response = provider.chat(**chat_params, **kwargs)

This leverages the provider's native JSON constraints rather than relying solely on post-processing.

Robust Implementation Pattern

Drop this reusable utility into your project to handle extraction with automatic retries and comprehensive logging:

# utils/debug_llm.py

import json
import logging
from llm_utils import extract_json_from_response

LOGGER = logging.getLogger(__name__)

def safe_llm_call(provider, chat_params, schema=None, max_retries=2):
    """
    Calls an LLM provider, cleans the response, validates JSON,
    and optionally forces schema validation via the provider.
    """
    for attempt in range(1, max_retries + 1):
        # Attach schema if the provider supports it (Gemini)

        kwargs = {"format": schema} if schema else {}
        response = provider.chat(**chat_params, **kwargs)
        raw = response["message"]["content"]
        LOGGER.debug(f"[Attempt {attempt}] RAW LLM output: {raw}")

        cleaned = extract_json_from_response(raw)
        LOGGER.debug(f"CLEANED JSON candidate: {cleaned}")

        try:
            payload = json.loads(cleaned)
            return payload
        except json.JSONDecodeError as exc:
            LOGGER.error(f"JSON decode error on attempt {attempt}: {exc}")
            # Tighten the prompt for the next try

            chat_params["messages"][-1]["content"] += (
                "\nPlease reply **only** with a valid JSON object, no extra text."
            )
    raise RuntimeError("Failed to obtain parsable JSON after retries")

Usage in pdf.py for section extraction:

# inside PDFHandler._extract_section(...)

chat_params = {...}  # existing parameters

parsed = safe_llm_call(
    self.provider, 
    chat_params, 
    return_model.model_json_schema()
)
transformed = transform_parsed_data(parsed)

Summary

• Markdown wrappers in llm_utils.py require exact matches; any deviation leaves parse-breaking artifacts.
• Truncated JSON from token limits in pdf.py and github.py requires larger context windows or chunking strategies.
• Schema mismatches in evaluator.py fail when EvaluationData receives unexpected keys from hallucinated model output.
• Provider quirks between Ollama and Gemini necessitate provider-specific handling in models.py.
• Debug by logging raw output before extract_json_from_response runs to isolate the failure point.
• Implement re-ask loops with stricter prompts to recover from transient formatting errors.

Frequently Asked Questions

Why does json.loads fail even when the LLM output looks like valid JSON?

The visible output often contains hidden characters such as markdown backticks, XML tags like <thought>, or leading/trailing whitespace that extract_json_from_response failed to strip. Log the raw output with repr() to reveal non-printable characters, and check that your cleaning logic handles the specific formatting quirks of your chosen provider in llm_utils.py.

How do I handle extraction differently for Gemini versus Ollama?

Pass the Pydantic schema directly to Gemini using the format parameter as shown in evaluator.py (kwargs = {"format": EvaluationData.model_json_schema()}) to let the API enforce JSON. For Ollama, increase the system prompt strictness and manually validate the cleaned output, as Ollama may inject newlines or explanatory text that Gemini omits.

What prompt changes prevent LLM extraction failures?

Update every Jinja template in prompts/templates/ to end with explicit instructions: "Respond with a valid JSON object ONLY, without any surrounding text or markdown." This reduces the likelihood of conversational preamble or markdown wrappers that break the extract_json_from_response parser.

When should I retry versus fail immediately on JSONDecodeError?

Implement a retry loop with a maximum of 2 attempts when the error stems from formatting noise or ambiguous instructions. If the second attempt with a tightened prompt fails, the issue is likely schema incompatibility or truncation, which requires human intervention or larger context windows rather than additional retries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}