# How to Debug LLM JSON Parsing Failures in the Hiring Agent

> Debug LLM JSON parsing failures in the hiring agent. Inspect raw responses, test the extract JSON sanitizer, and add detailed JSONDecodeError logging for swift resolution.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: how-to-guide
- Published: 2026-07-06

---

**Debug LLM JSON parsing failures by inspecting raw responses in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), testing the `extract_json_from_response` sanitizer in isolation, and implementing detailed `JSONDecodeError` logging to isolate markdown fences or truncated content.**

The `interviewstreet/hiring-agent` repository extracts structured résumé data from PDFs using a section-by-section LLM extraction pipeline. When the model wraps output in markdown code blocks or returns truncated JSON, the pipeline fails with decoding errors that require systematic debugging across [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py), and [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) to identify whether the issue stems from prompt formatting, response sanitization, or schema validation.

## Understanding the Extraction Architecture

The Hiring Agent splits extraction into three discrete modules to enable independent debugging:

| Module | Role | Key Entry Points |
|--------|------|------------------|
| **[`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py)** | Orchestrates PDF → text → LLM → JSON extraction. Calls the LLM, cleans the raw reply, and runs `json.loads`. | `extract_json_from_pdf` (lines 111‑116) calls the LLM and then runs `extract_json_from_response`. |
| **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)** | Contains shared helper utilities. The critical routine strips markdown fences and returns pure JSON. | `extract_json_from_response` (lines 13‑33) sanitizes LLM output. |
| **[`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py)** | Requests structured résumé evaluations using the same utilities. | `evaluate_resume` (lines 25‑48) → `extract_json_from_response`. |

This functional separation allows you to log or unit-test each stage independently: raw LLM reply → sanitized JSON string → parsed Python object.

## Common Causes of JSON Parsing Failures

Three specific issues account for most `JSONDecodeError` exceptions in the Hiring Agent:

1. **Markdown code fences** – The LLM often wraps JSON in ` ```json … ``` ` blocks or adds explanatory text before/after the payload. If `extract_json_from_response` fails to strip these wrappers, `json.loads` crashes.
2. **Truncated responses** – Large PDFs or timeout conditions cause the LLM to return mid-object JSON (e.g., `{"work": [{"company": "Acme"`), creating unclosed strings or brackets.
3. **Template mismatches** – Errors in `prompts/templates/` send section names that do not match PDF headings, causing the LLM to return natural language ("I couldn't find the information") instead of valid JSON.

## Step-by-Step Debugging Techniques

### Capture Raw LLM Output

Before the sanitizer runs, inspect the exact string returned by the model. In [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), the `_call_llm_for_section` method captures the raw response before any processing:

```python
from hiring_agent.pdf import PDFHandler

handler = PDFHandler()

# Force extraction on a single section to see raw output

raw = handler._call_llm_for_section(
    resume_text="John Doe …",  # abbreviated resume text

    prompt="Extract the work-experience section as JSON."
)

print("=== RAW LLM RESPONSE ===")
print(raw)

```

This reveals whether the model is adding markdown fences, conversational text, or truncating mid-response.

### Test the JSON Sanitizer in Isolation

The `extract_json_from_response` function in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) (lines 13‑33) uses regex to strip markdown fences. Test it independently with known problematic inputs:

```python
from hiring_agent.llm_utils import extract_json_from_response

bad_reply = """Here is the result:\n```json
{
  "work": [
    {"company": "Acme", "title": "Engineer"}
  ]

```"""

clean_json = extract_json_from_response(bad_reply)
print("=== CLEAN JSON ===")
print(clean_json)  # Should be a plain JSON string without fences

```

If the output still contains stray characters, adjust the regular expression logic in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) (lines 31‑33).

### Implement Detailed Error Logging

Replace the existing `json.loads` block in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) with a helper that captures the exact failure point. The error log line in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) at line 127 surfaces exceptions; enhance it with detailed diagnostics:

```python
import json
from hiring_agent.llm_utils import extract_json_from_response

def safe_parse(reply: str):
    try:
        clean = extract_json_from_response(reply)
        return json.loads(clean)
    except json.JSONDecodeError as exc:
        print("❌ JSON parsing failed:")
        print("Original reply:", reply)
        print("After cleaning :", clean)
        print("Error details   :", exc)
        raise

# Usage in extraction workflow

reply = handler._call_llm_for_section(...)
parsed = safe_parse(reply)

```

This pattern identifies whether the failure occurs during fence stripping or actual JSON parsing.

### Validate Against the Pydantic Schema

After successful `json.loads`, validate the structure against the `JSONResume` model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) to catch field mismatches or missing keys:

```python
from hiring_agent.models import JSONResume

def validate_resume(data: dict) -> JSONResume:
    try:
        return JSONResume(**data)
    except Exception as e:
        print("⚠️ Validation error:", e)
        raise

# After json.loads(...)

resume_obj = validate_resume(parsed)

```

Running this in a test harness reveals schema violations that indicate prompt template issues or LLM hallucinations.

## Critical Source Files for Debugging

| File | Why It Matters |
|------|----------------|
| **[`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py)** | Core extraction logic, LLM calls, and error handling at line 127 where `JSONDecodeError` is caught. |
| **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)** | Contains `extract_json_from_response` (lines 13‑33) which sanitizes LLM output. |
| **[`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py)** | Mirrors the PDF extraction flow for résumé evaluation; useful for comparing error patterns. |
| **[`github.py`](https://github.com/interviewstreet/hiring-agent/blob/main/github.py)** | Also uses `extract_json_from_response`; validates consistency across modules. |
| **[`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py)** | Holds Jinja templates that feed the LLM; template errors surface as extraction failures. |
| **[`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)** | Defines the `JSONResume` pydantic schema that specifies expected JSON shapes. |

## Summary

- **Inspect raw LLM responses** using `_call_llm_for_section` in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) before any sanitization occurs.
- **Test sanitization logic** by running `extract_json_from_response` from [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) in isolation with malformed inputs.
- **Catch `JSONDecodeError`** explicitly to log the original reply, cleaned output, and error details, particularly around [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) line 127.
- **Validate parsed data** against the `JSONResume` pydantic model in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) to detect schema mismatches.
- **Check prompt templates** in [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) when the LLM returns natural language instead of JSON.

## Frequently Asked Questions

### Why does the LLM return markdown fences around JSON?

The LLM returns markdown fences (`` ```json ... ``` ``) because base instructions or prompt templates often request "JSON format" without explicitly prohibiting markdown wrapping. The `extract_json_from_response` function in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) (lines 13‑33) specifically targets these fences with regex patterns to extract the raw JSON string before parsing.

### How can I identify if a parsing error is due to truncation?

Truncation produces incomplete JSON objects such as unclosed braces or unterminated strings. When you catch `JSONDecodeError` in [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py), compare the original reply length against expected token counts. If the error message mentions "Unexpected end of JSON input" or the cleaned string ends mid-object, the LLM likely timed out or hit length limits during the `_call_llm_for_section` execution.

### What is the role of `extract_json_from_response` in the pipeline?

`extract_json_from_response` in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) (lines 13‑33) serves as the sanitization barrier between the LLM and the JSON parser. It strips markdown code fences, removes conversational text, and returns a pure JSON string. This function is called by both [`pdf.py`](https://github.com/interviewstreet/hiring-agent/blob/main/pdf.py) (line 116) and [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) (line 48), making it the central point to fix formatting issues that affect the entire hiring pipeline.

### How do prompt templates affect JSON parsing reliability?

Prompt templates in [`prompts/template_manager.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompts/template_manager.py) determine whether the LLM returns structured JSON or conversational responses. If a template contains ambiguous instructions or references non-existent section names, the model may return "I could not find this section" instead of valid JSON. Ensuring templates explicitly demand raw JSON without markdown fences, and validating section names against the PDF content, prevents these parsing failures before they reach `json.loads`.