Celery Task Retry Policy and Error Handling for Failed LLM API Calls in LMForge

LMForge implements a strict no-retry Celery execution policy coupled with a graceful fallback mechanism that substitutes a default DeepSeek Chat model whenever external LLM API calls fail, ensuring workflow continuity without automatic retry loops.

LMForge is an open-source end-to-end LLMOps platform designed for orchestrating multi-model agent workflows. Understanding its Celery task retry policy and error handling for failed LLM API calls reveals a deliberate architectural choice: the platform prioritizes immediate service degradation over retry storms, using Celery solely as a lightweight asynchronous runner while embedding resilience directly into the LLM service layer.

Celery Configuration: No Automatic Task Retries

In api/config/config.py, the Celery configuration dictionary explicitly omits task-execution retry policies. The implementation only enables connection-level resilience through the broker_connection_retry_on_startup flag, meaning tasks execute exactly once and fail fast on exceptions.

# api/config/config.py

self.CELERY = {
    "broker_url": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_BROKER_DB'))}",
    "result_backend": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_RESULT_BACKEND_DB'))}",
    "task_ignore_result": _get_bool_env("CELERY_TASK_IGNORE_RESULT"),
    "result_expires": int(_get_env("CELERY_RESULT_EXPIRES")),
    # only the broker connection is retried on startup

    "broker_connection_retry_on_startup": _get_bool_env("CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP"),
}

This configuration means no automatic retry occurs for task execution failures. If a task raises an exception, Celery marks it as failed immediately, and the caller must explicitly re-queue the job if retry is desired.

LLM API Error Handling: Graceful Fallback Strategy

All LLM instantiation flows through api/internal/service/language_model_service.py. The load_language_model method wraps provider lookups, model creation, and parameter injection in a broad try / except block that captures any failure—from network timeouts to missing provider configurations.

# api/internal/service/language_model_service.py

def load_language_model(self, model_config: dict[str, Any]) -> BaseLanguageModel:
    try:
        provider_name = model_config.get("provider", "")
        model_name = model_config.get("model", "")
        parameters = model_config.get("parameters", {})

        provider = self.language_model_manager.get_provider(provider_name)
        model_entity = provider.get_model_entity(model_name)
        model_class = provider.get_model_class(model_entity.model_type)

        return model_class(**model_entity.attributes,
                           **parameters,
                           features=model_entity.features,
                           metadata=model_entity.metadata)
    except Exception as _:
        # any error → use default DeepSeek-Chat model

        return self.load_default_language_model()

When any exception occurs—whether from an unreachable provider, unknown model name, or authentication failure—the service immediately invokes load_default_language_model(). This returns a pre-configured DeepSeek Chat instance guaranteed to support tool calling and agent workflows:

# api/internal/service/language_model_service.py

@classmethod
def load_default_language_model(cls) -> BaseLanguageModel:
    return Chat(
        model="deepseek-chat",
        temperature=0.8,
        features=[ModelFeature.TOOL_CALL.value,
                  ModelFeature.AGENT_THOUGHT.value],
        metadata={},
    )

This fallback strategy ensures that downstream workflow nodes never crash due to missing LLM instances, even when external APIs are completely unavailable.

How Workflow Nodes Handle LLM Failures

The workflow execution layer in api/internal/core/workflow/nodes/llm/llm_node.py consumes the language model service and streams responses. Because LanguageModelService.load_language_model() already implements the fallback mechanism, the node itself contains no explicit retry logic or exception handling for API failures.

This design separation of concerns keeps the workflow engine lean: the node assumes it will always receive a valid BaseLanguageModel instance, while the service layer handles the complexity of provider resilience.

Task Definitions Without Retry Decorators

Background job definitions in api/internal/task/*.py (such as document_task.py or app_task.py) use bare @shared_task decorators without retry, autoretry_for, or max_retries parameters.

# api/internal/task/document_task.py

@shared_task
def build_documents(document_ids: list[UUID]) -> None:
    from app.http.app import injector
    from internal.service.indexing_service import IndexingService
    indexing_service = injector.get(IndexingService)
    indexing_service.build_documents(document_ids)

Any exception raised during execution propagates directly to Celery, which marks the task state as FAILURE. The platform relies on manual re-queueing or monitoring systems rather than automated retry loops for these background operations.

Summary

• LMForge configures Celery in api/config/config.py with broker-connection retries only, explicitly avoiding task-execution retry policies.
• Failed LLM API calls trigger an immediate fallback to a default DeepSeek Chat model via load_default_language_model() in api/internal/service/language_model_service.py.
• Celery tasks inherit default no-retry behavior, requiring manual intervention or re-queueing by callers when background jobs fail.
• Workflow nodes rely on service-layer resilience rather than implementing per-node retry logic, ensuring consistent behavior across the platform.

Frequently Asked Questions

Does LMForge automatically retry failed Celery tasks?

No. According to the source code in api/config/config.py, the platform does not configure task_autoretry_for, retry_backoff, or max_retries at the Celery app level. Individual tasks in api/internal/task/*.py use bare @shared_task decorators, meaning they execute once and mark as failed immediately upon raising an exception.

What happens when an LLM provider API is unreachable?

The load_language_model method in api/internal/service/language_model_service.py catches all exceptions—including network errors, authentication failures, and missing providers—and returns a default DeepSeek Chat model configured with TOOL_CALL and AGENT_THOUGHT features. This guarantees that agent workflows continue processing without interruption.

Can I enable automatic retries for Celery tasks in LMForge?

While the current implementation intentionally avoids retries, you could extend individual task definitions with autoretry_for=(Exception,) and retry_backoff=True decorators. However, this would deviate from the platform's design philosophy of fail-fast background jobs and graceful degradation via the LLM service layer.

Which default model is used when LLM API calls fail?

The fallback invokes load_default_language_model() which instantiates a DeepSeek Chat model with temperature 0.8 and metadata enabling tool usage and agent reasoning capabilities, as defined in api/internal/service/language_model_service.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}