Retry Mechanisms for LLM API Calls in Mirofish: A Deep Dive into Resilient AI Integration

Mirofish implements exponential backoff retry logic with jitter, async/sync decorators, and JSON recovery helpers to ensure robust LLM API calls across the backend.

The 666ghj/mirofish repository handles LLM API calls through a layered resilience strategy that combines generic retry utilities with service-specific recovery logic. By implementing retry mechanisms for LLM API calls at multiple levels—from low-level HTTP wrappers to high-level configuration generators—the codebase ensures fault tolerance against transient network errors, rate limits, and malformed responses.

Core Retry Infrastructure in retry.py

The foundation of Mirofish's resilience strategy lives in backend/app/utils/retry.py, which provides both decorator-based and class-based retry mechanisms.

Synchronous Retry Decorator (retry_with_backoff)

The retry_with_backoff decorator (lines 15-38) wraps any synchronous function to provide exponential backoff retry logic. It accepts parameters including max_retries (default 3), initial_delay, backoff_factor, and optional jitter to prevent thundering-herd effects. The decorator logs each retry attempt and re-raises the final exception after exhausting all retries.

from backend.app.utils.retry import retry_with_backoff
from backend.app.utils.llm_client import LLMClient

@retry_with_backoff(max_retries=4, initial_delay=2.0, jitter=True)
def summarize_text(text: str) -> str:
    client = LLMClient()
    response = client.chat(
        messages=[{"role": "user", "content": f"Summarize:\n{text}"}],
        temperature=0.5,
        max_tokens=256,
    )
    return response

Asynchronous Retry Decorator (retry_with_backoff_async)

For non-blocking operations, retry_with_backoff_async (lines 80-94) provides identical exponential backoff logic for async def functions using await asyncio.sleep. This ensures that async services can call LLM endpoints without blocking the event loop while maintaining the same resilience guarantees.

from backend.app.utils.retry import retry_with_backoff_async

@retry_with_backoff_async(max_retries=3, initial_delay=1.5)
async def async_chat(messages: list) -> str:
    client = LLMClient()
    resp = await client.client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0.6,
    )
    return resp.choices[0].message.content

Class-Based Retry Client (RetryableAPIClient)

The RetryableAPIClient class (lines 149-176) offers a programmatic approach for fine-grained control. Its call_with_retry method accepts a callable and executes it with exponential backoff, returning the result directly. This pattern allows services to pass custom exception tuples and retry parameters at runtime.

from backend.app.utils.retry import RetryableAPIClient
from backend.app.utils.llm_client import LLMClient

class InsightService:
    def __init__(self):
        self.llm = LLMClient()
        self.retry_client = RetryableAPIClient(max_retries=3, initial_delay=1.0)

    def generate_insight(self, prompt: str) -> dict:
        def call():
            return self.llm.chat_json(
                messages=[{"role": "user", "content": prompt}],
                temperature=0.0,
            )
        return self.retry_client.call_with_retry(call, exceptions=(Exception,))

Service-Level LLM Retry Strategies

Beyond generic utilities, Mirofish implements domain-specific retry mechanisms for LLM API calls that include response validation and recovery logic.

Configuration Generation with Temperature Decay

In backend/app/services/simulation_config_generator.py (lines 433-461), the _call_llm_with_retry method implements a sophisticated three-attempt loop specifically for generating simulation configurations. This mechanism lowers the temperature parameter on each retry to produce more deterministic output, detects truncated responses via finish_reason == 'length', and attempts JSON repair with regex-based cleanup before raising exceptions. It uses linear backoff (time.sleep(2 * (attempt + 1))) between attempts.

# Example invocation from simulation_config_generator.py

generator = SimulationConfigGenerator(...)
config = generator._call_llm_with_retry(
    prompt="Generate a time-config JSON for a 48-hour crisis simulation.",
    system_prompt="You are a social-media simulation expert. Return pure JSON.",
)

Zep Graph API Resilience

For external knowledge-graph operations, backend/app/services/zep_tools.py (lines 440-466) and backend/app/services/zep_entity_reader.py (lines 87-105) implement _call_with_retry methods with simple exponential backoff. These wrap Zep Graph API calls with delay doubling (default 2 seconds) and structured logging, ensuring that network glitches during knowledge-graph queries do not break the simulation pipeline.

Summary

Mirofish implements a comprehensive, multi-layered approach to retry mechanisms for LLM API calls:

• Generic decorators (retry_with_backoff, retry_with_backoff_async) in backend/app/utils/retry.py provide exponential backoff with optional jitter for both sync and async functions.
• Programmatic retry client (RetryableAPIClient) offers fine-grained control over exception handling and retry parameters.
• Domain-specific retry logic in simulation_config_generator.py combines temperature decay, JSON repair, and linear backoff for robust configuration generation.
• External API resilience for Zep Graph operations ensures knowledge-graph queries survive transient network failures.

These mechanisms collectively provide automatic retry on transient errors, intelligent back-off to avoid rate limits, and graceful degradation when all attempts exhaust.

Frequently Asked Questions

What is the default number of retries for LLM API calls in Mirofish?

The default number of retries varies by implementation. The generic decorators in backend/app/utils/retry.py default to 3 attempts, while the service-specific _call_llm_with_retry in simulation_config_generator.py implements a hardcoded 3-attempt loop. The RetryableAPIClient accepts configurable max_retries at initialization.

How does Mirofish handle rate limiting when retrying LLM requests?

Mirofish handles rate limiting through exponential backoff strategies that increase the delay between retry attempts. The retry_with_backoff decorator doubles the wait time after each failure (configurable via backoff_factor), while optional jitter adds randomness to prevent synchronized retry storms. The simulation_config_generator.py uses linear backoff (time.sleep(2 * (attempt + 1))) specifically for configuration generation tasks.

What happens when an LLM returns malformed JSON after multiple retries?

When the LLM returns malformed JSON, the _call_llm_with_retry method in simulation_config_generator.py attempts regex-based JSON repair and cleanup before raising an exception. If repair fails after three attempts (with decreasing temperature for more deterministic output), the method raises the final exception, allowing the calling workflow to fall back to default configurations via _get_default_time_config or similar methods.

Does Mirofish support asynchronous retry mechanisms for LLM calls?

Yes, Mirofish provides retry_with_backoff_async in backend/app/utils/retry.py specifically for asynchronous functions. This decorator uses await asyncio.sleep instead of blocking time.sleep, ensuring that async services can retry LLM API calls without blocking the event loop. The async decorator maintains the same exponential backoff, jitter, and logging capabilities as its synchronous counterpart.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}