What Kind of Data Does embabel-agent Process? A Complete Guide to Multimodal Content Handling

embabel-agent processes multimodal messages combining plain text, images, and documents through a type-safe Kotlin API with strict validation and size limits.

The embabel-agent framework, developed by Embabel, provides a structured approach to handling diverse data types for LLM interactions. Its data model centers on the ContentPart hierarchy defined in embabel-agent-api/src/main/kotlin/com/embabel/chat/ContentPart.kt, enabling seamless integration of textual and binary content in a single conversation flow.

Core Data Types in embabel-agent

The framework recognizes three primary data kinds, each implemented as a sealed subtype of ContentPart. This design ensures exhaustive pattern matching and compile-time safety when processing multimodal inputs.

Plain Text (TextPart)

The simplest content type, TextPart wraps a non-empty string:

data class TextPart(val text: String) : ContentPart {
    init {
        require(text.isNotEmpty()) { "Text must not be empty" }
    }
}

According to the embabel-agent source code at lines 41-44, empty strings trigger an IllegalArgumentException. This prevents null or blank messages from propagating through the pipeline.

Images (ImagePart)

ImagePart handles binary image data with strict validation requirements:

Attribute	Specification
MIME type	Must satisfy isImageMimeType() (e.g., image/png, image/jpeg)
Payload	Non-empty ByteArray
Size limit	20 MiB maximum (MAX_IMAGE_SIZE)

The constructor validates all constraints at instantiation:

val imgBytes = Files.readAllBytes(Paths.get("diagram.png"))
val image = ImagePart("image/png", imgBytes)  // throws if >20 MiB or invalid MIME

Implementation details reside in lines 50-62 of ContentPart.kt. The MediaPart interface (sealed under ContentPart) provides the common contract for all binary types.

Documents (DocumentPart)

DocumentPart extends support to generic document formats:

Attribute	Specification
MIME type	Must satisfy isDocumentMimeType() (e.g., application/pdf, text/plain)
Payload	Non-empty ByteArray
Filename	Optional, but non-blank if provided
Size limit	20 MiB maximum (MAX_DOCUMENT_SIZE)

val pdfBytes = Files.readAllBytes(Paths.get("contract.pdf"))
val document = DocumentPart(
    mimeType = "application/pdf",
    data = pdfBytes,
    filename = "contract.pdf"  // optional parameter
)

The validation logic at lines 85-99 enforces these constraints consistently.

Validation and Type Safety Architecture

MIME Type Verification

embabel-agent delegates MIME type validation to helper functions in the common module:

• isImageMimeType(mimeType: String) — defined in MediaMimeTypeUtils.kt
• isDocumentMimeType(mimeType: String) — defined in MediaMimeTypeUtils.kt

These functions reference canonical lists maintained in MimeTypes.kt (embabel-agent-common/embabel-agent-common/src/main/kotlin/com/embabel/common/ai/media/). The separation of concerns allows supported formats to evolve without modifying core data classes.

Sealed Interface Extensibility

MediaPart is declared as a sealed interface beneath the ContentPart root. This architecture enables future expansion:

// Current hierarchy (simplified)
sealed interface ContentPart
sealed interface MediaPart : ContentPart  // mimeType: String, data: ByteArray
data class TextPart(...) : ContentPart
data class ImagePart(...) : MediaPart
data class DocumentPart(...) : MediaPart
// Future: AudioPart, VideoPart, etc.

The sealed constraint ensures all implementations are known at compile time, while the interface abstraction permits polymorphic handling of binary content.

Practical Usage: Composing Multimodal Messages

Individual ContentPart instances assemble into collections for LLM submission:

import com.embabel.chat.*

// Build a mixed-content message
val messageParts: List<ContentPart> = listOf(
    TextPart("Analyze this quarterly report and accompanying chart."),
    DocumentPart("application/pdf", reportBytes, "Q3_Financials.pdf"),
    ImagePart("image/png", chartBytes)
)

// Pass to agent for processing
val response = embabelAgent.send(messageParts)

Each element undergoes validation at construction, failing fast with descriptive exceptions rather than deferring errors to the LLM layer.

Key Implementation Files

Path	Responsibility
embabel-agent-api/src/main/kotlin/com/embabel/chat/ContentPart.kt	ContentPart sealed class, TextPart, ImagePart, DocumentPart definitions
embabel-agent-common/embabel-agent-common/src/main/kotlin/com/embabel/common/ai/media/MimeTypes.kt	Supported MIME type constants
embabel-agent-common/embabel-agent-common/src/main/kotlin/com/embabel/common/ai/media/MediaMimeTypeUtils.kt	isImageMimeType(), isDocumentMimeType() validation functions

Summary

• Text — TextPart with non-empty string requirement
• Images — ImagePart with 20 MiB limit and image-specific MIME validation
• Documents — DocumentPart with 20 MiB limit, optional filename, and document-specific MIME validation
• Architecture — Sealed ContentPart hierarchy with MediaPart interface for extensible binary handling
• Validation — Eager constructor checks using require() blocks and dedicated MIME type utilities

Frequently Asked Questions

What is the maximum file size for images and documents in embabel-agent?

Both images and documents are limited to 20 MiB per part, enforced by MAX_IMAGE_SIZE and MAX_DOCUMENT_SIZE constants. The ImagePart and DocumentPart constructors throw IllegalArgumentException if this threshold is exceeded.

Can embabel-agent process audio or video files?

Not currently in the core distribution. However, the MediaPart sealed interface architecture permits extension with new binary types. Implementing AudioPart or VideoPart would require defining appropriate MIME type validators in MediaMimeTypeUtils.kt and size constraints analogous to existing media parts.

How does embabel-agent handle unsupported MIME types?

The framework rejects unknown MIME types at instantiation. Image parts must satisfy isImageMimeType(); document parts must satisfy isDocumentMimeType(). These functions check against enumerated lists in MimeTypes.kt, preventing invalid content from reaching downstream processors.

What happens if I provide an empty filename for a DocumentPart?

The filename parameter is optional (nullable), but if supplied it must be non-blank. The constructor at ContentPart.kt:85-99 validates require(filename == null || filename.isNotBlank()), throwing an exception for empty or whitespace-only filenames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}