How to Handle LLM Parsing Failures in Project Selection: Fallback Mechanisms in the Hiring Agent

When the LLM fails to parse project selection responses, the hiring-agent implements a two-layered fallback strategy that returns the first 7 projects from the original list to ensure pipeline continuity.

The interviewstreet/hiring-agent repository relies on large language models to rank and select top GitHub projects for technical interviews. When LLM parsing failures occur during project selection, the system cannot afford to crash or return empty results, so it employs defensive programming patterns in github.py to guarantee a deterministic output of exactly 7 unique projects even when the model returns malformed JSON or encounters network errors.

Two-Layered Fallback Strategy for LLM Parsing Errors

The codebase implements a defensive hierarchy that catches failures at different levels of the LLM invocation stack. Both layers converge on the same safe default: returning the first 7 projects from the original input list.

JSON Parsing Fallback (github.py lines 31–35)

After receiving the LLM response, the code attempts to clean the text and parse it as structured JSON using json.loads(). If the model outputs malformed JSON—such as missing brackets, trailing commas, or explanatory text outside the JSON structure—the json.JSONDecodeError exception triggers an immediate fallback.

In this exception block, the system logs the parsing error and returns projects[:7], preserving the downstream contract while discarding the unparsable LLM output. This ensures that even when the LLM fails to parse project selection responses into valid JSON, the hiring workflow continues uninterrupted.

General Exception Fallback (github.py lines 38–41)

Beyond JSON parsing errors, a broader except clause catches any other failure modes including network timeouts, authentication errors with the LLM provider, or unexpected response formats. This outer exception handler serves as the ultimate safety net.

When triggered, it logs the specific error details and similarly returns the first 7 projects from the original list. This dual-layer approach ensures that both data-format failures and infrastructure failures result in graceful degradation rather than pipeline termination.

Implementation Details in github.py

The core selection logic resides in github.py, where the select_top_projects() function orchestrates the LLM call and fallback handling. The function first prepares the chat parameters using templates from prompts/template_manager.py, then invokes the provider through utilities defined in llm_utils.py.

Here is the simplified flow demonstrating the exception handling:

def select_top_projects(projects):
    try:
        # Prepare prompt and call LLM via provider

        response = provider.chat(**chat_params)
        response_text = response["message"]["content"]
        
        # Attempt to extract and parse JSON from LLM output

        response_text = extract_json_from_response(response_text.strip())
        selected = json.loads(response_text)  # May raise JSONDecodeError

        # ... deduplicate and validate exactly 7 unique items ...

        
    except json.JSONDecodeError as e:
        # JSON could not be parsed → fallback to first 7 original projects

        print(f"ERROR: {e}")
        return projects[:7]
        
    except Exception as e:
        # Network errors, timeouts, or other failures → same fallback

        print(f"Error using LLM: {e}")
        return projects[:7]

This implementation guarantees that the function always returns a list object, preventing None or empty results from propagating to downstream evaluation stages.

The Seven-Project Contract

The fallback mechanism hardcodes a return of exactly 7 unique projects because the downstream evaluation pipeline expects this specific cardinality. By slicing projects[:7], the system maintains interface consistency regardless of LLM availability or output quality.

When the LLM succeeds, the code deduplicates its selections and validates the count; when it fails, the deterministic slice ensures the contract holds. This design choice reflects a resilient architecture that prioritizes workflow completion over perfect LLM execution.

Supporting Files and Infrastructure

While github.py contains the fallback logic, related components handle the LLM interaction setup:

• prompts/template_manager.py – Loads the github_project_selection.jinja template that structures the prompt asking the LLM to rank candidate projects. Poorly formatted LLM outputs often stem from template ambiguity, making this file relevant to parsing failure prevention.

• llm_utils.py – Provides initialize_llm_provider() and model-parameter utilities that the selection step relies on. Configuration errors in this module could trigger the general exception fallback in github.py.

Summary

• Two-layered defense: The system catches JSONDecodeError specifically, then uses a general except clause for all other failures.
• Deterministic fallback: Both error paths return projects[:7] from the original list, ensuring exactly 7 projects are always available.
• Source location: All fallback logic is implemented in github.py between lines 31 and 41.
• Contract preservation: The 7-project guarantee prevents downstream pipeline failures regardless of LLM reliability.

Frequently Asked Questions

What happens when the LLM returns malformed JSON during project selection?

The code catches the JSONDecodeError in the exception block at lines 31–35 of github.py, logs the error, and immediately returns the first 7 projects from the original input list. This prevents the malformed output from crashing the selection pipeline.

How does the hiring-agent handle network timeouts or LLM provider failures?

Network timeouts and provider errors are caught by the general except clause at lines 38–41 of github.py. Like the JSON parsing fallback, this handler logs the error and returns the first 7 projects, ensuring the hiring workflow continues even when the LLM service is unavailable.

Why does the fallback return exactly 7 projects?

The downstream evaluation pipeline requires exactly 7 unique projects to maintain interface contracts. Returning projects[:7] provides a deterministic, safe default that satisfies this requirement without requiring additional validation logic or empty-state handling in subsequent processing steps.

Which file manages the prompts that might cause parsing errors?

The prompts/template_manager.py file loads the github_project_selection.jinja template used to request ranked projects from the LLM. While this file doesn't handle the fallback logic directly, poorly structured prompts can increase the likelihood of receiving unparsable responses, making template management critical to reducing fallback triggers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}