How to Handle Errors and Implement Retry Logic with AgentEventListener in Embabel

Embabel's AgentEventListener architecture provides built-in error handling through Spring Observability callbacks and configurable retry logic via @Action annotations, ensuring robust agent execution with automatic span management.

The Embabel agent framework provides a comprehensive observability layer through event listeners that hook into every stage of agent execution. By leveraging the AgenticEventListener interface and Spring's Observation API, developers can implement sophisticated error handling and retry strategies that prevent data loss and ensure trace completeness according to the embabel/embabel-agent source code.

Understanding the AgentEventListener Architecture

Embabel exposes three primary callback interfaces—AgenticEventListener, EmbeddingEventListener, and concrete implementations including EmbabelSpanEventListener, EmbabelMetricsEventListener, and MdcPropagationEventListener. These listeners integrate automatically via the auto-configuration class AgentPlatformAutoConfiguration, which wires them into the agent runtime's observation pipeline.

Observation Lifecycle Callbacks

All listeners extend Spring Observability's Observation contract, providing three core callbacks defined in the source:

• onStart(Observation.Context): Invoked at the beginning of a logical operation, such as a planning iteration, to initialize spans and record start-time attributes.
• onError(Observation.Context): Triggered when exceptions bubble up from user code or tool execution, capturing stack traces and error attributes.
• onStop(Observation.Context): Executes when the operation finishes, ensuring spans close regardless of success or failure states.

The EmbabelSpanEventListener implementation in embabel-agent-observability/src/main/java/com/embabel/agent/observability/tracing/EmbabelSpanEventListener.java guarantees that onStop runs even if onError throws an exception, preventing open spans and memory leaks. This behavior is validated in EmbabelSpanEventListenerTest.java at lines 115-124.

Error Handling Strategies

Robust error handling in Embabel relies on the onError callback to intercept exceptions before they propagate out of the agent context.

Safeguarding the Tracing Pipeline

The default EmbabelSpanEventListener implementation safeguards the tracing pipeline by wrapping all listener invocations in try-finally blocks. Even if a custom listener throws from onError, the surrounding onStop is guaranteed to run. This ensures that distributed traces remain intact and telemetry data is not lost during fault conditions.

Implementing Custom Error Handlers

To create bespoke error handling, implement AgenticEventListener and register it as a Spring bean. The bean is discovered automatically by AgentPlatformAutoConfiguration.

@Component
public class MyCustomListener implements AgenticEventListener {

    @Override
    public void onStart(Observation.Context context) {
        context.getLowCardinalityKeyValues()
               .add(KeyValue.of("my.custom.tag", "value"));
    }

    @Override
    public void onError(Observation.Context context) {
        Throwable t = context.getError();
        log.warn("Agent error: {}", t.getMessage(), t);
    }

    @Override
    public void onStop(Observation.Context context) {
        // Clean-up, emit metrics, etc.
    }
}

Because the listener executes within the same observation scope, any exception thrown from onError is caught by EmbabelSpanEventListener, and the span is still closed properly.

Implementing Retry Logic with AgentEventListener

Retry behavior in Embabel is driven by action-level annotations on agent methods, processed by the ActionRetryPolicy class in embabel-agent-api/src/main/java/com/embabel/agent/core/ActionRetryPolicy.java.

Action-Level Retry Annotations

The @Action annotation supports two retry configuration attributes:

• actionRetryPolicy: Accepts a policy constant such as ActionRetryPolicy.FIRE_ONCE to execute the action only once.
• actionRetryPolicyExpression: Evaluates a SpEL expression to obtain a retry count at runtime, enabling externalized configuration.

The test suite RetryActionAnnotationJavaTest in embabel-agent-api/src/test/java/com/embabel/agent/api/annotation/support/RetryActionAnnotationJavaTest.java demonstrates that after the first invocation fails with throw new RuntimeException(...), the framework automatically re-invokes the method up to the configured limit. After the final attempt, the exception propagates to the listener chain where onError records the failure.

Configuring Retry Policies

Define custom retry policies as Spring beans to override defaults. All @Action methods without explicit policies inherit the custom logic.

@Bean
public ActionRetryPolicy customRetryPolicy() {
    return ActionRetryPolicy.builder()
            .maxAttempts(Integer.MAX_VALUE)
            .initialDelay(Duration.ofMillis(200))
            .backoffFactor(2.0)
            .build();
}

This configuration implements unlimited retries with exponential back-off, applied automatically to compatible agent actions.

Complete Implementation Examples

The following examples demonstrate the interaction between retry logic and event listeners in production scenarios.

Example 1: Retry-Aware Agent Action

@Agent(description = "Demo agent with retry", planner = PlannerType.UTILITY)
public class DemoAgent {

    private final AtomicInteger attempts = new AtomicInteger();

    @Action(actionRetryPolicyExpression = "${retry-twice}")
    public String unreliableAction(String input) {
        int run = attempts.incrementAndGet();
        if (run < 2) {
            throw new IllegalStateException("Transient failure, try again");
        }
        return "Success on attempt " + run;
    }
}

The retry count resolves from the Spring property retry-twice, allowing environment-specific tuning without code changes.

Example 2: Monitoring Retries with a Custom Listener

@Component
public class RetryCountingListener implements AgenticEventListener {

    private final AtomicInteger retries = new AtomicInteger();

    @Override
    public void onError(Observation.Context ctx) {
        retries.incrementAndGet();
        log.info("Retry attempt #{}", retries.get());
    }

    public int getRetryCount() {
        return retries.get();
    }
}

Inject RetryCountingListener into monitoring components to expose retry metrics to dashboards or alerting systems.

Summary

• EmbabelSpanEventListener guarantees trace completion by ensuring onStop executes even when onError throws exceptions, preventing memory leaks in embabel-agent-observability.
• Error handling occurs through the AgenticEventListener interface's onError callback, which receives the Observation.Context containing the thrown exception.
• Retry logic is configured via @Action annotations using actionRetryPolicy or actionRetryPolicyExpression, processed by the ActionRetryPolicy class in embabel-agent-api.
• Custom listeners are automatically wired by AgentPlatformAutoConfiguration when registered as Spring beans, requiring no manual registration.
• Each retry attempt generates a fresh observation, allowing listeners to track individual retry cycles through the onStart and onError callbacks.

Frequently Asked Questions

How does Embabel ensure spans are closed when a listener throws an exception?

The EmbabelSpanEventListener in embabel-agent-observability/src/main/java/com/embabel/agent/observability/tracing/EmbabelSpanEventListener.java wraps the observation lifecycle in try-finally blocks. Even if onError throws, the onStop callback executes to close the span and clean up thread-local state, as verified by EmbabelSpanEventListenerTest.java at lines 115-124.

Can I implement exponential back-off for agent action retries?

Yes. Create a custom ActionRetryPolicy bean with .backoffFactor(2.0) and .initialDelay(Duration.ofMillis(200)) as implemented in embabel-agent-api/src/main/java/com/embabel/agent/core/ActionRetryPolicy.java. When registered as a Spring bean, this policy applies to all @Action methods that do not specify an explicit retry policy.

What is the difference between actionRetryPolicy and actionRetryPolicyExpression?

The actionRetryPolicy attribute accepts static policy constants like ActionRetryPolicy.FIRE_ONCE, while actionRetryPolicyExpression accepts a SpEL expression (such as ${retry-twice}) that resolves at runtime against Spring properties. Use the expression variant for externalized configuration across environments.

Where is the retry logic actually executed in the Embabel codebase?

Retry interception occurs through the annotation processing defined in ActionRetryPolicy and tested in embabel-agent-api/src/test/java/com/embabel/agent/api/annotation/support/RetryActionAnnotationJavaTest.java. The framework creates a new observation for each retry attempt, meaning AgentEventListener callbacks like onStart and onError fire for every individual attempt, not just the initial call.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}