How to Test Embabel Agents: A Two-Layer Testing Strategy

The recommended approach for testing Embabel agents uses a two-layer strategy: fast unit tests that mock external services using the FakeOperationContext utility, and separate integration tests that validate against real LLM providers using environment variables and Maven profiles.

The Embabel agent framework is deliberately architected for testability, enabling developers to verify agent logic without expensive external calls. According to the embabel-agent source code, the testing philosophy centers on deterministic unit tests for prompt construction and tool-group configuration, supplemented by occasional integration tests that exercise the full stack against live LLM APIs.

Two-Layer Testing Strategy for Embabel Agents

The framework enforces a strict separation between isolated logic tests and end-to-end validation. This approach ensures deterministic results during rapid development cycles while maintaining confidence in production behavior.

Unit Testing with FakeOperationContext

Unit tests run fast, require no external services, and focus exclusively on the agent’s internal logic. The testing support package provides utilities to simulate LLM interactions without network calls.

Key implementation details:

• Import utilities from embabel-agent-api/src/main/kotlin/com/embabel/agent/test/README.md (the unitTestUtils package)
• Use FakeOperationContext to record LLM invocations and inspect generated prompts
• Mock injected Spring beans and LLM responses using standard Mockito patterns
• Verify that prompts contain expected data and that correct tool groups are supplied to the LLM

The FakeOperationContext class captures every LLM invocation, allowing assertions on the prompt content and tool configuration. This utility is located in the test support package and provides methods like expectResponse() to simulate LLM outputs and getLlmInvocations() to retrieve recorded interactions.

public class StarNewsFinderTest {

    @Test
    void writeupPromptMustContainKeyData() {
        // Mock any required services
        HoroscopeService horoscopeService = mock(HoroscopeService.class);
        StarNewsFinder starNewsFinder = new StarNewsFinder(horoscopeService, 5);

        // Use the fake operation context provided by the testing support package
        var context = new FakeOperationContext();
        context.expectResponse(new com.embabel.example.horoscope.Writeup("Gonna be a good day"));

        // Prepare mock data
        NewsStory cockatoos = new NewsStory(
                "https://fake.com.au",
                "Cockatoo behavior",
                "Cockatoos are eating cabbages"
        );
        NewsStory emus = new NewsStory(
                "https://morefake.com.au",
                "Emu movements",
                "Emus are massing"
        );

        // Domain objects for the test
        StarPerson starPerson = new StarPerson("Lynda", "Scorpio");
        RelevantNewsStories relevantNewsStories = new RelevantNewsStories(Arrays.asList(cockatoos, emus));
        Horoscope horoscope = new Horoscope("This is a good day for you");

        // Execute the agent action
        starNewsFinder.writeup(starPerson, relevantNewsStories, horoscope, context);

        // Inspect the recorded prompt
        var prompt = context.getLlmInvocations().getFirst().getPrompt();

        // Verify that the prompt includes the expected data
        assertTrue(prompt.contains(starPerson.getName()));
        assertTrue(prompt.contains(starPerson.sign()));
        assertTrue(prompt.contains(cockatoos.getSummary()));
        assertTrue(prompt.contains(emus.getSummary()));

        // Verify that no tool groups were added (LLM call should be plain)
        var toolGroups = context.getLlmInvocations().getFirst().getInteraction().getToolGroups();
        assertTrue(toolGroups.isEmpty(), "The LLM should not have been given any tool groups");
    }
}

Integration Testing with Real LLM Providers

Integration tests exercise the complete stack including actual LLM providers. These tests are excluded from the default Maven lifecycle to maintain CI speed.

Requirements for integration testing:

• Set required API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY) as environment variables
• Use the integrationTestUtils package from embabel-agent-test-support/embabel-agent-test/src/main/kotlin/com/embabel/agent/test/integration/EmbabelMockitoIntegrationTest.kt
• Execute via Maven integration-test profile

The EmbabelMockitoIntegrationTest utility provides a default LLM configuration that returns valid random objects, simplifying setup for complex integration scenarios.

# Ensure required API keys are exported

export OPENAI_API_KEY=sk-…
export ANTHROPIC_API_KEY=…

# Run integration tests (skipping the Ollama IT which needs a local server)

mvn -Dtest='*IT,!LLMOllama*IT' -Dsurefire.failIfNoSpecifiedTests=false test

Key Testing Resources and File Locations

The following files define the testing infrastructure for Embabel agents:

• README.md – Contains the general testing documentation, including Maven commands and environment variable requirements
• embabel-agent-api/src/main/kotlin/com/embabel/agent/test/README.md – Documents the unitTestUtils and integrationTestUtils packages
• embabel-agent-api/src/test/kotlin/com/embabel/example/horoscope/java/StarNewsFinderTest.java – Reference implementation demonstrating prompt inspection and mock usage
• embabel-agent-test-support/embabel-agent-test/src/main/kotlin/com/embabel/agent/test/integration/EmbabelMockitoIntegrationTest.kt – Provides integration testing utilities with configurable LLM responses

Summary

• Unit tests should use FakeOperationContext to mock LLM interactions and verify prompt construction without external dependencies
• Integration tests require live API keys and run via specific Maven profiles to validate end-to-end behavior
• The testing support package in embabel-agent-api/src/main/kotlin/com/embabel/agent/test/ provides both unitTestUtils and integrationTestUtils for consistent test patterns
• Default Maven builds execute only unit tests, keeping CI pipelines fast while preserving the option for full-stack validation

Frequently Asked Questions

How do I mock LLM responses in Embabel agent unit tests?

Use the FakeOperationContext class from the testing support package. Instantiate it in your test, call expectResponse() to define the mock LLM output, then pass the context to your agent method. After execution, inspect getLlmInvocations() to verify prompt content and tool group configuration.

What environment variables are required for integration testing?

You must export valid API keys for the LLM providers you intend to test, specifically OPENAI_API_KEY and ANTHROPIC_API_KEY. The framework reads these at runtime to authenticate with live services during integration test execution.

Where are the testing utilities located in the repository?

The core testing documentation resides in embabel-agent-api/src/main/kotlin/com/embabel/agent/test/README.md. Integration-specific utilities are found in embabel-agent-test-support/embabel-agent-test/src/main/kotlin/com/embabel/agent/test/integration/EmbabelMockitoIntegrationTest.kt.

Why are integration tests excluded from the default Maven test run?

The default mvn test lifecycle runs only unit tests to maintain fast feedback loops during development. Integration tests require external API calls and are executed separately using Maven profiles or specific test selectors like *IT patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}