# Celery Task Retry Policy and Error Handling for Failed LLM API Calls in LMForge

> Learn about LMForge's no-retry Celery policy and error handling for LLM API failures. Discover its fallback mechanism ensuring workflow continuity without retries.

- Repository: [Haohao/lmforge-end-to-end-llmops-platform-for-multi-model-agents](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents)
- Tags: how-to-guide
- Published: 2026-03-03

---

**LMForge implements a strict no-retry Celery execution policy coupled with a graceful fallback mechanism that substitutes a default DeepSeek Chat model whenever external LLM API calls fail, ensuring workflow continuity without automatic retry loops.**

LMForge is an open-source end-to-end LLMOps platform designed for orchestrating multi-model agent workflows. Understanding its **Celery task retry policy and error handling for failed LLM API calls** reveals a deliberate architectural choice: the platform prioritizes immediate service degradation over retry storms, using Celery solely as a lightweight asynchronous runner while embedding resilience directly into the LLM service layer.

## Celery Configuration: No Automatic Task Retries

In [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py), the Celery configuration dictionary explicitly omits task-execution retry policies. The implementation only enables connection-level resilience through the `broker_connection_retry_on_startup` flag, meaning tasks execute exactly once and fail fast on exceptions.

```python

# api/config/config.py

self.CELERY = {
    "broker_url": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_BROKER_DB'))}",
    "result_backend": f"redis://{self.REDIS_USERNAME}:{self.REDIS_PASSWORD}@{self.REDIS_HOST}:{self.REDIS_PORT}/{int(_get_env('CELERY_RESULT_BACKEND_DB'))}",
    "task_ignore_result": _get_bool_env("CELERY_TASK_IGNORE_RESULT"),
    "result_expires": int(_get_env("CELERY_RESULT_EXPIRES")),
    # only the broker connection is retried on startup

    "broker_connection_retry_on_startup": _get_bool_env("CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP"),
}

```

This configuration means **no automatic retry** occurs for task execution failures. If a task raises an exception, Celery marks it as failed immediately, and the caller must explicitly re-queue the job if retry is desired.

## LLM API Error Handling: Graceful Fallback Strategy

All LLM instantiation flows through [`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py). The `load_language_model` method wraps provider lookups, model creation, and parameter injection in a broad `try / except` block that captures any failure—from network timeouts to missing provider configurations.

```python

# api/internal/service/language_model_service.py

def load_language_model(self, model_config: dict[str, Any]) -> BaseLanguageModel:
    try:
        provider_name = model_config.get("provider", "")
        model_name = model_config.get("model", "")
        parameters = model_config.get("parameters", {})

        provider = self.language_model_manager.get_provider(provider_name)
        model_entity = provider.get_model_entity(model_name)
        model_class = provider.get_model_class(model_entity.model_type)

        return model_class(**model_entity.attributes,
                           **parameters,
                           features=model_entity.features,
                           metadata=model_entity.metadata)
    except Exception as _:
        # any error → use default DeepSeek-Chat model

        return self.load_default_language_model()

```

When any exception occurs—whether from an unreachable provider, unknown model name, or authentication failure—the service immediately invokes `load_default_language_model()`. This returns a pre-configured **DeepSeek Chat** instance guaranteed to support tool calling and agent workflows:

```python

# api/internal/service/language_model_service.py

@classmethod
def load_default_language_model(cls) -> BaseLanguageModel:
    return Chat(
        model="deepseek-chat",
        temperature=0.8,
        features=[ModelFeature.TOOL_CALL.value,
                  ModelFeature.AGENT_THOUGHT.value],
        metadata={},
    )

```

This fallback strategy ensures that downstream workflow nodes never crash due to missing LLM instances, even when external APIs are completely unavailable.

## How Workflow Nodes Handle LLM Failures

The workflow execution layer in [`api/internal/core/workflow/nodes/llm/llm_node.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/core/workflow/nodes/llm/llm_node.py) consumes the language model service and streams responses. Because `LanguageModelService.load_language_model()` already implements the fallback mechanism, the node itself contains **no explicit retry logic** or exception handling for API failures.

This design separation of concerns keeps the workflow engine lean: the node assumes it will always receive a valid `BaseLanguageModel` instance, while the service layer handles the complexity of provider resilience.

## Task Definitions Without Retry Decorators

Background job definitions in `api/internal/task/*.py` (such as [`document_task.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/document_task.py) or [`app_task.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/app_task.py)) use bare `@shared_task` decorators without `retry`, `autoretry_for`, or `max_retries` parameters.

```python

# api/internal/task/document_task.py

@shared_task
def build_documents(document_ids: list[UUID]) -> None:
    from app.http.app import injector
    from internal.service.indexing_service import IndexingService
    indexing_service = injector.get(IndexingService)
    indexing_service.build_documents(document_ids)

```

Any exception raised during execution propagates directly to Celery, which marks the task state as **FAILURE**. The platform relies on manual re-queueing or monitoring systems rather than automated retry loops for these background operations.

## Summary

- **LMForge configures Celery in [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py)** with broker-connection retries only, explicitly avoiding task-execution retry policies.
- **Failed LLM API calls trigger an immediate fallback** to a default DeepSeek Chat model via `load_default_language_model()` in [`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py).
- **Celery tasks inherit default no-retry behavior**, requiring manual intervention or re-queueing by callers when background jobs fail.
- **Workflow nodes rely on service-layer resilience** rather than implementing per-node retry logic, ensuring consistent behavior across the platform.

## Frequently Asked Questions

### Does LMForge automatically retry failed Celery tasks?

No. According to the source code in [`api/config/config.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/config/config.py), the platform does not configure `task_autoretry_for`, `retry_backoff`, or `max_retries` at the Celery app level. Individual tasks in `api/internal/task/*.py` use bare `@shared_task` decorators, meaning they execute once and mark as failed immediately upon raising an exception.

### What happens when an LLM provider API is unreachable?

The `load_language_model` method in [`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py) catches all exceptions—including network errors, authentication failures, and missing providers—and returns a default DeepSeek Chat model configured with `TOOL_CALL` and `AGENT_THOUGHT` features. This guarantees that agent workflows continue processing without interruption.

### Can I enable automatic retries for Celery tasks in LMForge?

While the current implementation intentionally avoids retries, you could extend individual task definitions with `autoretry_for=(Exception,)` and `retry_backoff=True` decorators. However, this would deviate from the platform's design philosophy of fail-fast background jobs and graceful degradation via the LLM service layer.

### Which default model is used when LLM API calls fail?

The fallback invokes `load_default_language_model()` which instantiates a DeepSeek Chat model with temperature 0.8 and metadata enabling tool usage and agent reasoning capabilities, as defined in [`api/internal/service/language_model_service.py`](https://github.com/haohao-end/lmforge-end-to-end-llmops-platform-for-multi-model-agents/blob/main/api/internal/service/language_model_service.py).