# How to Debug Heurist Mesh Agent Failure Modes: 7 Common Issues and Fixes

> Debug Heurist Mesh agent failures by understanding 7 common issues like network errors, rate limits, and timeouts. Learn quick fixes and troubleshooting tips for smooth operation.

- Repository: [Heurist/heurist-agent-framework](https://github.com/heurist-network/heurist-agent-framework)
- Tags: how-to-guide
- Published: 2026-03-03

---

**Heurist Mesh agent failures typically stem from network errors, rate limits, tool execution exceptions, timeouts, missing fallbacks, LLM parsing issues, or resource leaks, all of which can be diagnosed through structured loguru logs and specific debugging patterns in the MeshAgent base class.**

The heurist-network/heurist-agent-framework provides a robust architecture for building AI agents around the abstract `MeshAgent` base class. When agents misbehave in production or development, understanding the seven primary failure layers and their corresponding debug signals in the source code allows for rapid root-cause analysis.

## Understanding the Error Handling Architecture

The agent execution flow follows a strict pipeline that emits structured logs at every stage. According to the source code in [`mesh/mesh_agent.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/mesh_agent.py), the entry point `call_agent` receives requests and executes `_before_handle_message`, followed by `handle_message` which routes to either direct tool calls or LLM-driven queries. The `_execute_tool_with_policy` method (lines ~79-94) wraps agent-specific logic with timeout handling, while the `@with_retry` decorator in [`decorators.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/decorators.py) (lines 33-50) provides exponential backoff for transient failures. When timeouts occur, `_invoke_fallback_agent` (lines ~96-108) attempts to delegate to backup agents before returning an error payload.

## Common Failure Modes and Debugging Strategies

### Network and API Errors (HTTP 4xx/5xx)

Network failures originate in `_api_request` within [`mesh/mesh_agent.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/mesh_agent.py) (lines ~636-638), where the method inspects HTTP responses and logs API request errors. Symptoms include payloads containing `{"error": "...", "status": "error"}` or log entries stating "API request error".

Enable **loguru** debug level by setting `export LOGURU_LEVEL=DEBUG` before running your agent. Examine the `logger.error` calls inside `_api_request` to view the exact HTTP status codes and response bodies. Verify endpoint accessibility using external tools like `curl` to isolate whether the issue resides in the agent configuration or the upstream service.

### Rate Limiting and Proxy Fallback (429 Errors)

When remote APIs return HTTP 429, the `_api_request` method (lines ~998-1024) checks `supports_proxy_fallback()` and attempts to route through `proxy_client.forward_request` in [`mesh/utils/proxy_client.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/utils/proxy_client.py). You will observe log entries reading "Rate limit exceeded for …" followed by either a successful proxy result or `{"error":"Proxy fallback failed"}`.

Check whether your concrete agent implementation overrides `supports_proxy_fallback()` to return `True` (default is `False`). Inspect the `PROXY_FALLBACK` environment variables and confirm the proxy server list configuration. The proxy client logs will indicate whether the fallback chain succeeded or exhausted all available endpoints.

### Tool Execution Exceptions and Retry Exhaustion

Any agent tool decorated with `@with_retry` (lines 33-50 in [`decorators.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/decorators.py)) will log warning messages like "Retry X/Y for …" when exceptions occur. If all retries fail, the exception bubbles up as "All retries failed …" with the original stack trace preserved in the final `logger.error` output.

To debug, locate the warning "Retry X/Y for …" in your logs and examine the stack trace that follows the final retry attempt. Add temporary `logger.debug` statements inside the specific tool implementation to surface input parameters and intermediate states before the exception occurs.

### Timeout Errors in Tool Execution

Timeouts manifest in `_execute_tool_with_policy` (lines ~79-94) when tool execution exceeds the value returned by `get_default_timeout_seconds()` or `get_tool_timeout_seconds()`. The returned payload contains `{ "status":"error", "error":"Tool '<name>' timed out after N s" }`, accompanied by a warning log with the same message.

Override `get_default_timeout_seconds()` in your agent subclass to increase the threshold, or profile the underlying API call to identify latency bottlenecks. The timeout value is configurable per tool, allowing fine-grained control over long-running operations without affecting the entire agent.

### Missing or Broken Fallback Agents

When timeouts occur but no fallback executes, or when you encounter `ValueError: Fallback spec must include 'module' and 'class'`, the issue resides in `_invoke_fallback_agent` (lines ~96-108). This method expects `get_fallback_for_tool` to return a dictionary containing `module` and `class` keys specifying the fallback agent.

Ensure your agent implements `get_fallback_for_tool` correctly, returning a valid spec such as `{"module": "mesh.agents.trending_token_agent", "class": "TrendingTokenAgent", "input": {...}}`. Inspect the logs emitted inside `_invoke_fallback_agent` to verify that the fallback module loads successfully and receives the forwarded payload.

### LLM-Related Failures (No Tool Call or Malformed Response)

LLM parsing errors occur in `handle_message` (lines ~61-73) when the model returns empty `tool_calls` lists or malformed JSON. The symptom is a response containing `{"error":"Failed to process query"}` or missing tool invocations despite valid user queries.

Enable DEBUG logging for the Gemini helpers (`call_gemini_async`) and add `logger.debug(response)` immediately before the `if not response` check in `handle_message`. This reveals the raw LLM output, allowing you to identify whether the issue stems from prompt engineering, model selection, or response parsing logic.

### Resource Cleanup Problems (Dangling Sessions)

Resource leaks appear during interpreter shutdown as warnings stating "Cleanup failed …". These originate in `__del__` and `cleanup` methods (lines ~545-558 in [`mesh/mesh_agent.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/mesh_agent.py)) when `aiohttp` sessions remain open.

Verify that `await agent_instance.cleanup()` is invoked in every exit path, including within `_invoke_fallback_agent`. Run the agent inside an `async with` block to guarantee proper context manager exit, ensuring that network connections and file descriptors are released deterministically.

## Step-by-Step Debugging Workflow

When encountering an unidentified failure, follow this diagnostic sequence:

1. **Enable comprehensive logging** – Set `LOGURU_LEVEL=DEBUG` to capture all messages from `with_retry`, `monitor_execution`, and `_api_request`.

2. **Identify the failure layer** – Search logs for the specific warning patterns: "API request error" for network issues, "Retry X/Y" for transient failures, "Tool timed out" for execution limits, or "Failed to process query" for LLM errors.

3. **Isolate the component** – Use the file paths and line numbers provided above to locate the exact source code emitting the error message.

4. **Inject diagnostic code** – Add `logger.debug()` statements or temporary print calls inside the suspected method (`_handle_tool_logic`, `get_fallback_for_tool`, etc.) to capture input parameters and intermediate states.

5. **Verify fallbacks and retries** – Confirm that `@with_retry` decorators wrap the correct methods and that `get_fallback_for_tool` returns valid specifications when timeouts are expected.

## Practical Debugging Code Examples

### Enable Detailed Logging for Root Cause Analysis

```python
import os
import sys
from loguru import logger

# Show everything from DEBUG upwards

logger.remove()
logger.add(sys.stderr, level="DEBUG")

# Now import and run an agent

from mesh.agents.trending_token_agent import TrendingTokenAgent
import asyncio

async def main():
    agent = TrendingTokenAgent()
    resp = await agent.call_agent({"query": "Top trending crypto tokens today"})
    print(resp)

asyncio.run(main())

```

This configuration surfaces all log messages from `with_retry`, `monitor_execution`, and `_api_request`, pinpointing exact retry attempts, HTTP error codes, and timeout events.

### Inspect Retry Behavior for Specific Tools

```python
from mesh.agents.unifai_token_analysis_agent import UnifaiTokenAnalysisAgent
import asyncio

async def run():
    agent = UnifaiTokenAnalysisAgent()
    # The tool `search_token` is decorated with @with_retry(max_retries=3)

    result = await agent._execute_tool_with_policy(
        tool_name="search_token",
        function_args={"token": "NON_EXISTENT"},
        session_context={},
        original_params={},
    )
    print(result)

asyncio.run(run())

```

If the remote API fails, the console will display three sequential warnings (`Retry X/3`) followed by either a successful payload or an error dictionary containing the final exception details.

### Force Timeout to Validate Fallback Logic

```python
import asyncio
from mesh.mesh_agent import MeshAgent

class SlowAgent(MeshAgent):
    async def _handle_tool_logic(self, tool_name, function_args, session_context=None):
        await asyncio.sleep(10)      # Simulate a long-running operation

        return {"status": "success", "data": "done"}

    def get_tool_schemas(self):
        return []   # not needed for this demo

    def get_system_prompt(self):
        return "You are a dummy agent."

    def get_default_timeout_seconds(self):
        return 2          # 2-second timeout

    async def get_fallback_for_tool(self, tool_name, function_args, original_params):
        return {
            "module": "mesh.agents.trending_token_agent",
            "class": "TrendingTokenAgent",
            "input": {"query": "fallback data"},
        }

# Run

asyncio.run(SlowAgent().call_agent({"tool": "slow_tool", "tool_arguments": {}}))

```

The log output will contain a warning about the 2-second timeout followed by a successful fallback response from `TrendingTokenAgent`, verifying that your fallback chain is properly configured.

## Key Source Files for Debugging

Understanding these specific files accelerates diagnosis:

- **[`decorators.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/decorators.py)** (lines 33-52, 58-71) – Implements `with_retry` with exponential backoff and `monitor_execution` for execution-time logging. Review this when investigating retry patterns or performance bottlenecks.

- **[`mesh/mesh_agent.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/mesh_agent.py)** (lines 79-138, 545-558) – Contains `_execute_tool_with_policy`, `_invoke_fallback_agent`, and resource cleanup logic. This is the primary location for timeout and fallback debugging.

- **[`mesh/utils/proxy_client.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/utils/proxy_client.py)** – Handles 429 rate-limit redirection to configured proxy servers. Examine this when rate limiting persists despite enabling `supports_proxy_fallback()`.

- **[`mesh/tests/_test_agents.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/tests/_test_agents.py)** – Demonstrates how the framework captures and reports errors during automated testing, providing patterns for your own test harnesses.

## Summary

- **Enable DEBUG logging** via `LOGURU_LEVEL=DEBUG` to expose all retry attempts, HTTP errors, and timeout events across the agent lifecycle.
- **Network and rate-limit errors** surface in `_api_request` (lines ~636-638 and ~998-1024) and require verification of endpoints and proxy configurations.
- **Tool execution failures** are wrapped by `@with_retry` in [`decorators.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/decorators.py) (lines 33-50), with specific retry counts visible in warning logs.
- **Timeouts** are enforced by `_execute_tool_with_policy` (lines ~79-94) and can be adjusted by overriding `get_default_timeout_seconds()` in your agent subclass.
- **Fallback agents** must be specified via `get_fallback_for_tool` returning a dict with `module` and `class` keys to avoid `ValueError` exceptions in `_invoke_fallback_agent`.
- **LLM parsing errors** appear in `handle_message` (lines ~61-73) and require inspection of raw model responses to diagnose prompt or parsing issues.
- **Resource leaks** are prevented by ensuring `await agent.cleanup()` is called in all exit paths, particularly when using fallback agents.

## Frequently Asked Questions

### How do I enable debug logging to trace agent failures?

Set the environment variable `export LOGURU_LEVEL=DEBUG` before importing the agent framework. This configures loguru to emit DEBUG-level messages from `with_retry`, `monitor_execution`, and `_api_request`, revealing exact HTTP status codes, retry attempts, and timeout durations in the console output.

### What should I do when my agent hits rate limits (429 errors) repeatedly?

First, verify that your agent class overrides `supports_proxy_fallback()` to return `True`. Then check the `PROXY_FALLBACK` environment variables and ensure [`mesh/utils/proxy_client.py`](https://github.com/heurist-network/heurist-agent-framework/blob/main/mesh/utils/proxy_client.py) can reach the configured proxy servers. If proxy fallback is disabled or misconfigured, the agent will return `{"error":"Proxy fallback failed"}` immediately upon receiving HTTP 429 responses.

### How do I implement a fallback agent when tools timeout?

Override the `get_fallback_for_tool` method in your `MeshAgent` subclass to return a dictionary specifying the fallback agent module and class: `{"module": "mesh.agents.trending_token_agent", "class": "TrendingTokenAgent", "input": {...}}`. Ensure the module path is importable and the class inherits from `MeshAgent`. The `_invoke_fallback_agent` method (lines ~96-108) will automatically instantiate and delegate to this agent when `_execute_tool_with_policy` detects a timeout.

### Why is my agent timing out even though the API seems fast?

The default timeout is controlled by `get_default_timeout_seconds()`, which defaults to a conservative value. Override this method in your agent to return a higher integer value, or implement `get_tool_timeout_seconds(tool_name)` for tool-specific limits. Additionally, verify that the timeout is occurring in your tool logic rather than in the LLM processing phase by checking whether the log warning mentions "Tool timed out" (tool layer) versus "Failed to process query" (LLM layer).