# How to Perform Image Generation with Multimodal LLMs in ChocolateLMLite

> Learn to perform image generation with multimodal LLMs in ChocolateLMLite. Discover how the generate image tool validates requests, calls LLM endpoints, and saves images as persona attachments.

- Repository: [Segment (gpsnmeajp)/chocolatelmlite](https://github.com/gpsnmeajp/chocolatelmlite)
- Tags: how-to-guide
- Published: 2026-03-02

---

**ChocolateLMLite enables multimodal image generation through the `generate_image` tool, which validates explicit user requests, calls external LLM endpoints, and persists generated images as persona attachments.**

ChocolateLMLite is an open-source framework that supports multimodal large language models capable of generating images on demand. According to the gpsnmeajp/chocolatelmlite source code, the system integrates external image generation APIs through a structured tool-calling mechanism. This guide explains the core components, configuration steps, and implementation details required to activate and use this feature.

## Core Architecture for Image Generation

The image generation pipeline in ChocolateLMLite relies on three primary classes that handle API communication, tool validation, and file persistence.

### ImageGenerater.cs: The API Wrapper

In [`src/ImageGenerater.cs`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/src/ImageGenerater.cs), the `ImageGenerater` class manages the external API request and response parsing. It constructs a JSON payload requesting both `image` and `text` modalities, POSTs to the configured endpoint, and processes the returned data. When the API returns a base64-encoded data URI, the class decodes the image and saves it via `FileManager.SaveAttachmentToActivePersona`. If the API returns a remote URL, that URL is passed directly to the user without local storage.

### Tools.cs: LLM Tool Integration

The `Tools.GenerateImage` method in [`src/Tools.cs`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/src/Tools.cs) exposes the `generate_image` function to the LLM. This method enforces two critical safety checks before executing: it verifies that `isRequestByUser` is `true` (ensuring the request originated from explicit user input rather than automated tool chains) and checks that `isImageGenerated` is `false` (preventing multiple generations within a single conversation turn). Upon successful validation, it invokes `ImageGenerater.GenerateImage`, stores the returned attachment ID in `lastAttachmentId`, and sets the generation flag to block subsequent calls until the next user turn.

### FileManager.cs: Attachment Persistence

Located in [`src/FileManager.cs`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/src/FileManager.cs), the `FileManager` class provides the storage layer for persona-specific data. The `SaveAttachmentToActivePersona` method writes decoded image bytes to `data/attachments/attachment_<id>.<ext>` and returns a unique attachment ID. This ID enables the frontend to reference and display the generated image in subsequent chat messages.

## Enabling Image Generation via Configuration

Image generation is disabled by default to prevent unintended API costs and usage. To activate the feature, modify the global settings in [`data/general.yaml`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/data/general.yaml):

```yaml
EnableImageGeneration: true
ImageGenerationEndpointUrl: "https://api.openrouter.ai/api/v1"
ImageGenerationApiKey: "sk-xxxxxxxxxxxxxxxxxxxx"
ImageGenerationModel: "google/gemini-2.5-flash-image"

```

The `EnableImageGeneration` field acts as the master switch. The endpoint URL must point to a compatible API that supports multimodal chat completions, while the model identifier must reference a version capable of returning `modalities: ["image","text"]` in its response structure.

## Request Flow and Execution

When a user sends a message requesting image creation (for example, "画像を生成してください。森の中の小さな家"), the following sequence executes:

1. **Function Detection**: The LLM identifies the image generation intent and emits a tool call for `generate_image`.
2. **Validation**: `Tools.GenerateImage` receives the prompt with `isRequestByUser = true` and verifies that no image has been generated yet in the current turn.
3. **API Communication**: `ImageGenerater.GenerateImage` wraps the prompt with a generation directive and POSTs to `{EndpointUrl}/chat/completions` with the `modalities` array set to `["image", "text"]`.
4. **Response Parsing**: The system inspects `choices[0].message.images[0].image_url.url` in the JSON response. Base64 data URIs trigger local storage via `FileManager`, while remote URLs return as text references.
5. **Delivery**: The textual description and optional attachment ID return to the client, where the frontend retrieves and renders the image from the persona's attachment store.

## Safety Mechanisms and Rate Limiting

ChocolateLMLite implements strict controls to prevent abuse and excessive API consumption:

- **User-Explicit Flag**: Generation only proceeds when `isRequestByUser` is `true`, blocking background or automated tool invocations.
- **One-Per-Turn Limit**: The `isImageGenerated` boolean prevents consecutive generation attempts within a single conversation turn, requiring a new user message to reset the state.
- **Credential Validation**: Missing endpoint URLs or API keys return immediate error responses without attempting network requests.

## Implementation Examples

### Configuration Setup

```yaml

# data/general.yaml

EnableImageGeneration: true
ImageGenerationEndpointUrl: "https://openrouter.ai/api/v1"
ImageGenerationApiKey: "sk-xxxxxxxxxxxxxxxxxxxx"
ImageGenerationModel: "google/gemini-2.5-flash-image"

```

### Tool Invocation Payload

When the LLM calls the generation tool, it sends this JSON structure:

```json
{
  "name": "generate_image",
  "arguments": {
    "prompt": "画像を生成してください。夕暮れの海辺に灯る灯台を描いて",
    "isRequestByUser": true
  }
}

```

### Backend Validation Logic

The `Tools.GenerateImage` method in [`src/Tools.cs`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/src/Tools.cs) performs preliminary checks before executing the external API call:

```csharp
if (!isRequestByUser)
    throw new InvalidOperationException("画像生成はユーザー要求時のみ実行可能です。");

if (isImageGenerated)
    throw new InvalidOperationException("画像は既に生成されています。");

var (textResponse, attachmentId) = await _imageGenerater.GenerateImage(prompt);

if (attachmentId.HasValue) {
    lastAttachmentId = attachmentId.Value;
    isImageGenerated = true;
}

return textResponse;

```

### Image Processing and Storage

The `ImageGenerater.GenerateImage` method handles the HTTP request and persistence logic:

```csharp
var requestBody = new {
    model = settings.ImageGenerationModel,
    messages = new[] {
        new { role = "user", content = prompt }
    },
    modalities = new[] { "image", "text" }
};

using var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromSeconds(settings.TimeoutSeconds);
var endpoint = $"{settings.ImageGenerationEndpointUrl.TrimEnd('/')}/chat/completions";

using var request = new HttpRequestMessage(HttpMethod.Post, endpoint);
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", settings.ImageGenerationApiKey);
request.Content = new StringContent(JsonSerializer.Serialize(requestBody),
                                 Encoding.UTF8, "application/json");

var response = await httpClient.SendAsync(request);
var json = await response.Content.ReadAsStringAsync();
using var doc = JsonDocument.Parse(json);
var images = doc.RootElement.GetProperty("choices")[0]
                     .GetProperty("message")
                     .GetProperty("images");

var imageUrl = images[0].GetProperty("image_url").GetProperty("url").GetString();

if (imageUrl.StartsWith("data:image/")) {
    var base64 = imageUrl.Substring(imageUrl.IndexOf(',') + 1);
    var bytes = Convert.FromBase64String(base64);
    var ext = Regex.Match(imageUrl, @"data:image/(.+?);base64").Groups[1].Value;
    var id = _fileManager.SaveAttachmentToActivePersona($"tmp.{ext}", bytes);
    return (textResponse, id);
}

```

### Frontend Response Handling

The backend returns a JSON payload that the frontend (in [`static/js/talk.js`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/static/js/talk.js)) processes:

```json
{
  "status": "completed",
  "response": "画像生成LLMからの応答内容: 夕暮れの灯台が映っています。",
  "attachmentId": 3
}

```

When `attachmentId` is present, the UI fetches the image from the `/attachment/{id}` endpoint and injects it into the chat view.

## Summary

- **ImageGenerater.cs** handles the HTTP transport to multimodal endpoints and decodes base64 responses into local attachments.
- **Tools.cs** exposes the `generate_image` tool to the LLM while enforcing user-validation and per-turn generation limits via `isRequestByUser` and `isImageGenerated`.
- **FileManager.cs** persists generated images to `data/attachments/` and returns unique IDs for frontend retrieval.
- Configuration requires setting `EnableImageGeneration: true` in [`data/general.yaml`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/data/general.yaml) with valid endpoint credentials and a compatible model identifier.
- The system distinguishes between base64-encoded images (stored locally) and remote URLs (referenced directly).
- Safety mechanisms ensure generation occurs only on explicit user request and limits output to one image per conversation turn.

## Frequently Asked Questions

### How do I enable image generation with multimodal LLMs in ChocolateLMLite?

Edit the [`data/general.yaml`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/data/general.yaml) file to set `EnableImageGeneration: true`, provide a valid `ImageGenerationEndpointUrl` (such as OpenRouter), and specify a compatible multimodal model like `google/gemini-2.5-flash-image`. The system requires an API key with sufficient credits for the target endpoint.

### What safety mechanisms prevent accidental image generation?

ChocolateLMLite implements two primary safeguards in [`src/Tools.cs`](https://github.com/gpsnmeajp/chocolatelmlite/blob/main/src/Tools.cs). First, the `isRequestByUser` flag ensures generation only proceeds when explicitly triggered by user input, blocking automated or cascading tool calls. Second, the `isImageGenerated` boolean limits each conversation turn to a single image generation, preventing rapid successive API calls.

### How are generated images stored and retrieved?

When the external API returns a base64-encoded image, `ImageGenerater.GenerateImage` decodes the payload and calls `FileManager.SaveAttachmentToActivePersona`, which writes the file to `data/attachments/attachment_<id>.<ext>`. The method returns an attachment ID that the frontend uses to fetch and display the image via the `/attachment/{id}` endpoint.

### Which multimodal models are compatible with this implementation?

The system works with any LLM that supports the OpenAI-compatible chat completions format with `modalities: ["image", "text"]` in the response. The reference implementation uses `google/gemini-2.5-flash-image` through OpenRouter, but other providers offering similar multimodal output formats should function with the same configuration structure.