How to Handle Tool Failures and Implement Retries for Agent Actions in Embabel

Embabel Agent provides built-in failure detection via ToolCallOutcomes and configurable retry policies through the ActionRetryPolicy enum and Spring Expression Language (SpEL) expressions.

The embabel/embabel-agent framework offers robust primitives to handle tool failures and implement retries for agent actions in Embabel. By leveraging event listeners and quality-of-service (QoS) policies, you can ensure your agents gracefully recover from external tool failures without manual intervention.

Detecting Tool Failures with Event Listeners

Every tool invocation generates a ToolCallRequestEvent and a corresponding ToolCallResponseEvent. The static helper ToolCallOutcomes extracts results or errors from these responses using ToolCallOutcomes.error(event).

In EmbabelSpanEventListener.java, the framework inspects this error to determine whether an agent should re-plan or retry an action. When a tool call throws an exception, the listener records the failure and adds the "embabel.replan.reason" attribute with the value "stuck, retrying with new plan", signaling the planner to create a fresh execution plan.

Configuring Built-In Retry Policies

The ActionRetryPolicy enum in ActionRetryPolicy.kt defines two built-in strategies:

• FIRE_ONCE: Maps to maxAttempts = 1, executing the action exactly once regardless of failure.
• DEFAULT: Applies standard QoS settings with maxAttempts = 5 and exponential backoff starting at 10 seconds.

The DefaultActionQosProvider.kt consults this policy when building ActionQos for @Action annotated methods. If you specify FIRE_ONCE, the generated QoS forces a single attempt; otherwise, the framework applies the default retry configuration defined in ActionQos.kt.

Customizing Retries with SpEL Expressions

For externalized configuration, use the actionRetryPolicyExpression attribute on the @Action annotation. This SpEL expression resolves against the Spring Environment, allowing you to define retry parameters in application.yml or other property sources.

As demonstrated in ActionQosPropertyProviderTest.kt (lines 176-179), you can reference fully qualified QoS configurations:

@Action(actionRetryPolicyExpression = "\${retry-twice}")
fun fetchData(context: AgentProcess) {
    val result = context.toolCall("DataService", mapOf("id" to "123"))
}

With corresponding YAML configuration:

retry-twice:
  max-attempts: 2
  backoff-millis: 5000
  backoff-multiplier: 2.0
  backoff-max-interval: 20000

The Re-Planning Mechanism

When ToolCallOutcomes.error(event) returns an exception, the EmbabelSpanEventListener adds the embabel.replan.reason attribute to the span. As verified in EmbabelSpanEventListenerTest.java (lines 531-537), this attribute triggers the planner to abort the current plan and create a new one that respects the action's retry policy. The new plan generation occurs automatically without requiring custom error handling code in your agent logic.

Practical Implementation Examples

Basic Fire-Once Policy

Use FIRE_ONCE for actions that should not retry under any circumstances:

@Agent(description = "Demo agent")
class SimpleAgent {

    @Action(actionRetryPolicy = ActionRetryPolicy.FIRE_ONCE)
    fun riskyToolCall(context: AgentProcess) {
        val result = context.toolCall("UnreliableWebSearch", mapOf("query" to "latest news"))
        println("Result: $result")
    }
}

Externalized Configuration

Reference external properties for flexible deployment-specific retry logic:

@Agent(description = "Configurable retry example")
class ConfigurableAgent {

    @Action(actionRetryPolicyExpression = "\${my.retry.policy}")
    fun fetchWeather(context: AgentProcess) {
        val weather = context.toolCall("WeatherService", mapOf("city" to "London"))
        println("Weather: $weather")
    }
}

my:
  retry:
    policy:
      max-attempts: 3
      backoff-millis: 2000
      backoff-multiplier: 2.0
      backoff-max-interval: 15000
      idempotent: true

Custom Failure Listener

Implement ToolCallListener to add custom observability or alerting:

public class MyToolFailureListener implements ToolCallListener {
    
    @Override
    public void onToolCallResponse(ToolCallResponseEvent event) {
        Throwable error = ToolCallOutcomes.error(event);
        if (error != null) {
            log.warn("Tool '{}' failed: {}", event.getRequest().getToolName(), error.getMessage());
            // Additional custom logic here
        }
    }
}

Summary

• Tool failure detection relies on ToolCallOutcomes.error(event) within EmbabelSpanEventListener.java to extract exceptions from ToolCallResponseEvent instances.
• Retry policies are controlled via the ActionRetryPolicy enum (FIRE_ONCE or DEFAULT) processed by DefaultActionQosProvider.kt.
• Custom configurations use SpEL expressions (actionRetryPolicyExpression) to externalize ActionQos parameters in application.yml or properties files.
• Re-planning occurs automatically when the framework sets the embabel.replan.reason attribute after detecting tool errors, triggering the planner to create a fresh execution strategy.

Frequently Asked Questions

How does Embabel detect when a tool call fails?

Embabel generates ToolCallRequestEvent and ToolCallResponseEvent for every invocation. The ToolCallOutcomes.error(event) static method extracts any exception from the response event. The EmbabelSpanEventListener.java inspects this result to trigger retry or re-planning logic when errors are present.

What is the difference between FIRE_ONCE and DEFAULT retry policies?

FIRE_ONCE forces exactly one execution attempt regardless of outcome, while DEFAULT applies the standard QoS configuration with up to 5 retry attempts and exponential backoff starting at 10 seconds. These policies are defined in ActionRetryPolicy.kt and applied by DefaultActionQosProvider.kt when constructing the ActionQos for annotated methods.

Can I configure retry parameters in application.properties instead of code?

Yes. Use the actionRetryPolicyExpression attribute on @Action to reference a SpEL expression that resolves to an ActionQos configuration defined in your Spring properties. This allows externalized configuration of max-attempts, backoff-millis, and other parameters without modifying source code, as demonstrated in ActionQosPropertyProviderTest.kt.

What happens when all retry attempts are exhausted?

When the maximum attempts defined in the ActionQos are exhausted, the framework sets the embabel.replan.reason attribute to "stuck, retrying with new plan" and triggers the planner to create a fresh execution plan. This behavior is verified in EmbabelSpanEventListenerTest.java (lines 531-537), ensuring that stuck actions receive new planning rather than failing permanently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}