# Retry Mechanisms for LLM API Calls in Mirofish: A Deep Dive into Resilient AI Integration

> Discover Mirofish's robust LLM API retry mechanisms including exponential backoff with jitter, async/sync decorators, and JSON recovery. Ensure resilient AI integration in your backend.

- Repository: [BaiFu/mirofish](https://github.com/666ghj/mirofish)
- Tags: deep-dive
- Published: 2026-02-23

---

**Mirofish implements exponential backoff retry logic with jitter, async/sync decorators, and JSON recovery helpers to ensure robust LLM API calls across the backend.**

The `666ghj/mirofish` repository handles LLM API calls through a layered resilience strategy that combines generic retry utilities with service-specific recovery logic. By implementing retry mechanisms for LLM API calls at multiple levels—from low-level HTTP wrappers to high-level configuration generators—the codebase ensures fault tolerance against transient network errors, rate limits, and malformed responses.

## Core Retry Infrastructure in [`retry.py`](https://github.com/666ghj/mirofish/blob/main/retry.py)

The foundation of Mirofish's resilience strategy lives in [`backend/app/utils/retry.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/utils/retry.py), which provides both decorator-based and class-based retry mechanisms.

### Synchronous Retry Decorator (`retry_with_backoff`)

The `retry_with_backoff` decorator (lines 15-38) wraps any synchronous function to provide exponential backoff retry logic. It accepts parameters including `max_retries` (default 3), `initial_delay`, `backoff_factor`, and optional `jitter` to prevent thundering-herd effects. The decorator logs each retry attempt and re-raises the final exception after exhausting all retries.

```python
from backend.app.utils.retry import retry_with_backoff
from backend.app.utils.llm_client import LLMClient

@retry_with_backoff(max_retries=4, initial_delay=2.0, jitter=True)
def summarize_text(text: str) -> str:
    client = LLMClient()
    response = client.chat(
        messages=[{"role": "user", "content": f"Summarize:\n{text}"}],
        temperature=0.5,
        max_tokens=256,
    )
    return response

```

### Asynchronous Retry Decorator (`retry_with_backoff_async`)

For non-blocking operations, `retry_with_backoff_async` (lines 80-94) provides identical exponential backoff logic for `async def` functions using `await asyncio.sleep`. This ensures that async services can call LLM endpoints without blocking the event loop while maintaining the same resilience guarantees.

```python
from backend.app.utils.retry import retry_with_backoff_async

@retry_with_backoff_async(max_retries=3, initial_delay=1.5)
async def async_chat(messages: list) -> str:
    client = LLMClient()
    resp = await client.client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0.6,
    )
    return resp.choices[0].message.content

```

### Class-Based Retry Client (`RetryableAPIClient`)

The `RetryableAPIClient` class (lines 149-176) offers a programmatic approach for fine-grained control. Its `call_with_retry` method accepts a callable and executes it with exponential backoff, returning the result directly. This pattern allows services to pass custom exception tuples and retry parameters at runtime.

```python
from backend.app.utils.retry import RetryableAPIClient
from backend.app.utils.llm_client import LLMClient

class InsightService:
    def __init__(self):
        self.llm = LLMClient()
        self.retry_client = RetryableAPIClient(max_retries=3, initial_delay=1.0)

    def generate_insight(self, prompt: str) -> dict:
        def call():
            return self.llm.chat_json(
                messages=[{"role": "user", "content": prompt}],
                temperature=0.0,
            )
        return self.retry_client.call_with_retry(call, exceptions=(Exception,))

```

## Service-Level LLM Retry Strategies

Beyond generic utilities, Mirofish implements domain-specific retry mechanisms for LLM API calls that include response validation and recovery logic.

### Configuration Generation with Temperature Decay

In [`backend/app/services/simulation_config_generator.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/services/simulation_config_generator.py) (lines 433-461), the `_call_llm_with_retry` method implements a sophisticated three-attempt loop specifically for generating simulation configurations. This mechanism lowers the `temperature` parameter on each retry to produce more deterministic output, detects truncated responses via `finish_reason == 'length'`, and attempts JSON repair with regex-based cleanup before raising exceptions. It uses linear backoff (`time.sleep(2 * (attempt + 1))`) between attempts.

```python

# Example invocation from simulation_config_generator.py

generator = SimulationConfigGenerator(...)
config = generator._call_llm_with_retry(
    prompt="Generate a time-config JSON for a 48-hour crisis simulation.",
    system_prompt="You are a social-media simulation expert. Return pure JSON.",
)

```

### Zep Graph API Resilience

For external knowledge-graph operations, [`backend/app/services/zep_tools.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/services/zep_tools.py) (lines 440-466) and [`backend/app/services/zep_entity_reader.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/services/zep_entity_reader.py) (lines 87-105) implement `_call_with_retry` methods with simple exponential backoff. These wrap Zep Graph API calls with delay doubling (default 2 seconds) and structured logging, ensuring that network glitches during knowledge-graph queries do not break the simulation pipeline.

## Summary

Mirofish implements a comprehensive, multi-layered approach to retry mechanisms for LLM API calls:

- **Generic decorators** (`retry_with_backoff`, `retry_with_backoff_async`) in [`backend/app/utils/retry.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/utils/retry.py) provide exponential backoff with optional jitter for both sync and async functions.
- **Programmatic retry client** (`RetryableAPIClient`) offers fine-grained control over exception handling and retry parameters.
- **Domain-specific retry logic** in [`simulation_config_generator.py`](https://github.com/666ghj/mirofish/blob/main/simulation_config_generator.py) combines temperature decay, JSON repair, and linear backoff for robust configuration generation.
- **External API resilience** for Zep Graph operations ensures knowledge-graph queries survive transient network failures.

These mechanisms collectively provide automatic retry on transient errors, intelligent back-off to avoid rate limits, and graceful degradation when all attempts exhaust.

## Frequently Asked Questions

### What is the default number of retries for LLM API calls in Mirofish?

The default number of retries varies by implementation. The generic decorators in [`backend/app/utils/retry.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/utils/retry.py) default to 3 attempts, while the service-specific `_call_llm_with_retry` in [`simulation_config_generator.py`](https://github.com/666ghj/mirofish/blob/main/simulation_config_generator.py) implements a hardcoded 3-attempt loop. The `RetryableAPIClient` accepts configurable `max_retries` at initialization.

### How does Mirofish handle rate limiting when retrying LLM requests?

Mirofish handles rate limiting through exponential backoff strategies that increase the delay between retry attempts. The `retry_with_backoff` decorator doubles the wait time after each failure (configurable via `backoff_factor`), while optional jitter adds randomness to prevent synchronized retry storms. The [`simulation_config_generator.py`](https://github.com/666ghj/mirofish/blob/main/simulation_config_generator.py) uses linear backoff (`time.sleep(2 * (attempt + 1))`) specifically for configuration generation tasks.

### What happens when an LLM returns malformed JSON after multiple retries?

When the LLM returns malformed JSON, the `_call_llm_with_retry` method in [`simulation_config_generator.py`](https://github.com/666ghj/mirofish/blob/main/simulation_config_generator.py) attempts regex-based JSON repair and cleanup before raising an exception. If repair fails after three attempts (with decreasing temperature for more deterministic output), the method raises the final exception, allowing the calling workflow to fall back to default configurations via `_get_default_time_config` or similar methods.

### Does Mirofish support asynchronous retry mechanisms for LLM calls?

Yes, Mirofish provides `retry_with_backoff_async` in [`backend/app/utils/retry.py`](https://github.com/666ghj/mirofish/blob/main/backend/app/utils/retry.py) specifically for asynchronous functions. This decorator uses `await asyncio.sleep` instead of blocking `time.sleep`, ensuring that async services can retry LLM API calls without blocking the event loop. The async decorator maintains the same exponential backoff, jitter, and logging capabilities as its synchronous counterpart.