How to Debug LLM JSON Parsing Failures in the Hiring Agent

Debug LLM JSON parsing failures by inspecting raw responses in pdf.py, testing the extract_json_from_response sanitizer in isolation, and implementing detailed JSONDecodeError logging to isolate markdown fences or truncated content.

The interviewstreet/hiring-agent repository extracts structured résumé data from PDFs using a section-by-section LLM extraction pipeline. When the model wraps output in markdown code blocks or returns truncated JSON, the pipeline fails with decoding errors that require systematic debugging across pdf.py, llm_utils.py, and evaluator.py to identify whether the issue stems from prompt formatting, response sanitization, or schema validation.

Understanding the Extraction Architecture

The Hiring Agent splits extraction into three discrete modules to enable independent debugging:

Module	Role	Key Entry Points
pdf.py	Orchestrates PDF → text → LLM → JSON extraction. Calls the LLM, cleans the raw reply, and runs json.loads.	extract_json_from_pdf (lines 111‑116) calls the LLM and then runs extract_json_from_response.
llm_utils.py	Contains shared helper utilities. The critical routine strips markdown fences and returns pure JSON.	extract_json_from_response (lines 13‑33) sanitizes LLM output.
evaluator.py	Requests structured résumé evaluations using the same utilities.	evaluate_resume (lines 25‑48) → extract_json_from_response.

This functional separation allows you to log or unit-test each stage independently: raw LLM reply → sanitized JSON string → parsed Python object.

Common Causes of JSON Parsing Failures

Three specific issues account for most JSONDecodeError exceptions in the Hiring Agent:

1. Markdown code fences – The LLM often wraps JSON in ```json … ``` blocks or adds explanatory text before/after the payload. If extract_json_from_response fails to strip these wrappers, json.loads crashes.
2. Truncated responses – Large PDFs or timeout conditions cause the LLM to return mid-object JSON (e.g., {"work": [{"company": "Acme"), creating unclosed strings or brackets.
3. Template mismatches – Errors in prompts/templates/ send section names that do not match PDF headings, causing the LLM to return natural language ("I couldn't find the information") instead of valid JSON.

Step-by-Step Debugging Techniques

Capture Raw LLM Output

Before the sanitizer runs, inspect the exact string returned by the model. In pdf.py, the _call_llm_for_section method captures the raw response before any processing:

from hiring_agent.pdf import PDFHandler

handler = PDFHandler()

# Force extraction on a single section to see raw output

raw = handler._call_llm_for_section(
    resume_text="John Doe …",  # abbreviated resume text

    prompt="Extract the work-experience section as JSON."
)

print("=== RAW LLM RESPONSE ===")
print(raw)

This reveals whether the model is adding markdown fences, conversational text, or truncating mid-response.

Test the JSON Sanitizer in Isolation

The extract_json_from_response function in llm_utils.py (lines 13‑33) uses regex to strip markdown fences. Test it independently with known problematic inputs:

from hiring_agent.llm_utils import extract_json_from_response

bad_reply = """Here is the result:\n```json
{
  "work": [
    {"company": "Acme", "title": "Engineer"}
  ]

```"""

clean_json = extract_json_from_response(bad_reply)
print("=== CLEAN JSON ===")
print(clean_json)  # Should be a plain JSON string without fences

If the output still contains stray characters, adjust the regular expression logic in llm_utils.py (lines 31‑33).

Implement Detailed Error Logging

Replace the existing json.loads block in pdf.py with a helper that captures the exact failure point. The error log line in pdf.py at line 127 surfaces exceptions; enhance it with detailed diagnostics:

import json
from hiring_agent.llm_utils import extract_json_from_response

def safe_parse(reply: str):
    try:
        clean = extract_json_from_response(reply)
        return json.loads(clean)
    except json.JSONDecodeError as exc:
        print("❌ JSON parsing failed:")
        print("Original reply:", reply)
        print("After cleaning :", clean)
        print("Error details   :", exc)
        raise

# Usage in extraction workflow

reply = handler._call_llm_for_section(...)
parsed = safe_parse(reply)

This pattern identifies whether the failure occurs during fence stripping or actual JSON parsing.

Validate Against the Pydantic Schema

After successful json.loads, validate the structure against the JSONResume model in models.py to catch field mismatches or missing keys:

from hiring_agent.models import JSONResume

def validate_resume(data: dict) -> JSONResume:
    try:
        return JSONResume(**data)
    except Exception as e:
        print("⚠️ Validation error:", e)
        raise

# After json.loads(...)

resume_obj = validate_resume(parsed)

Running this in a test harness reveals schema violations that indicate prompt template issues or LLM hallucinations.

Critical Source Files for Debugging

File	Why It Matters
pdf.py	Core extraction logic, LLM calls, and error handling at line 127 where JSONDecodeError is caught.
llm_utils.py	Contains extract_json_from_response (lines 13‑33) which sanitizes LLM output.
evaluator.py	Mirrors the PDF extraction flow for résumé evaluation; useful for comparing error patterns.
github.py	Also uses extract_json_from_response; validates consistency across modules.
prompts/template_manager.py	Holds Jinja templates that feed the LLM; template errors surface as extraction failures.
models.py	Defines the JSONResume pydantic schema that specifies expected JSON shapes.

Summary

• Inspect raw LLM responses using _call_llm_for_section in pdf.py before any sanitization occurs.
• Test sanitization logic by running extract_json_from_response from llm_utils.py in isolation with malformed inputs.
• Catch JSONDecodeError explicitly to log the original reply, cleaned output, and error details, particularly around pdf.py line 127.
• Validate parsed data against the JSONResume pydantic model in models.py to detect schema mismatches.
• Check prompt templates in prompts/template_manager.py when the LLM returns natural language instead of JSON.

Frequently Asked Questions

Why does the LLM return markdown fences around JSON?

The LLM returns markdown fences (```json ... ```) because base instructions or prompt templates often request "JSON format" without explicitly prohibiting markdown wrapping. The extract_json_from_response function in llm_utils.py (lines 13‑33) specifically targets these fences with regex patterns to extract the raw JSON string before parsing.

How can I identify if a parsing error is due to truncation?

Truncation produces incomplete JSON objects such as unclosed braces or unterminated strings. When you catch JSONDecodeError in pdf.py, compare the original reply length against expected token counts. If the error message mentions "Unexpected end of JSON input" or the cleaned string ends mid-object, the LLM likely timed out or hit length limits during the _call_llm_for_section execution.

What is the role of extract_json_from_response in the pipeline?

extract_json_from_response in llm_utils.py (lines 13‑33) serves as the sanitization barrier between the LLM and the JSON parser. It strips markdown code fences, removes conversational text, and returns a pure JSON string. This function is called by both pdf.py (line 116) and evaluator.py (line 48), making it the central point to fix formatting issues that affect the entire hiring pipeline.

How do prompt templates affect JSON parsing reliability?

Prompt templates in prompts/template_manager.py determine whether the LLM returns structured JSON or conversational responses. If a template contains ambiguous instructions or references non-existent section names, the model may return "I could not find this section" instead of valid JSON. Ensuring templates explicitly demand raw JSON without markdown fences, and validating section names against the PDF content, prevents these parsing failures before they reach json.loads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}