How to Perform Image Generation with Multimodal LLMs in ChocolateLMLite

ChocolateLMLite enables multimodal image generation through the generate_image tool, which validates explicit user requests, calls external LLM endpoints, and persists generated images as persona attachments.

ChocolateLMLite is an open-source framework that supports multimodal large language models capable of generating images on demand. According to the gpsnmeajp/chocolatelmlite source code, the system integrates external image generation APIs through a structured tool-calling mechanism. This guide explains the core components, configuration steps, and implementation details required to activate and use this feature.

Core Architecture for Image Generation

The image generation pipeline in ChocolateLMLite relies on three primary classes that handle API communication, tool validation, and file persistence.

ImageGenerater.cs: The API Wrapper

In src/ImageGenerater.cs, the ImageGenerater class manages the external API request and response parsing. It constructs a JSON payload requesting both image and text modalities, POSTs to the configured endpoint, and processes the returned data. When the API returns a base64-encoded data URI, the class decodes the image and saves it via FileManager.SaveAttachmentToActivePersona. If the API returns a remote URL, that URL is passed directly to the user without local storage.

Tools.cs: LLM Tool Integration

The Tools.GenerateImage method in src/Tools.cs exposes the generate_image function to the LLM. This method enforces two critical safety checks before executing: it verifies that isRequestByUser is true (ensuring the request originated from explicit user input rather than automated tool chains) and checks that isImageGenerated is false (preventing multiple generations within a single conversation turn). Upon successful validation, it invokes ImageGenerater.GenerateImage, stores the returned attachment ID in lastAttachmentId, and sets the generation flag to block subsequent calls until the next user turn.

FileManager.cs: Attachment Persistence

Located in src/FileManager.cs, the FileManager class provides the storage layer for persona-specific data. The SaveAttachmentToActivePersona method writes decoded image bytes to data/attachments/attachment_<id>.<ext> and returns a unique attachment ID. This ID enables the frontend to reference and display the generated image in subsequent chat messages.

Enabling Image Generation via Configuration

Image generation is disabled by default to prevent unintended API costs and usage. To activate the feature, modify the global settings in data/general.yaml:

EnableImageGeneration: true
ImageGenerationEndpointUrl: "https://api.openrouter.ai/api/v1"
ImageGenerationApiKey: "sk-xxxxxxxxxxxxxxxxxxxx"
ImageGenerationModel: "google/gemini-2.5-flash-image"

The EnableImageGeneration field acts as the master switch. The endpoint URL must point to a compatible API that supports multimodal chat completions, while the model identifier must reference a version capable of returning modalities: ["image","text"] in its response structure.

Request Flow and Execution

When a user sends a message requesting image creation (for example, "画像を生成してください。森の中の小さな家"), the following sequence executes:

1. Function Detection: The LLM identifies the image generation intent and emits a tool call for generate_image.
2. Validation: Tools.GenerateImage receives the prompt with isRequestByUser = true and verifies that no image has been generated yet in the current turn.
3. API Communication: ImageGenerater.GenerateImage wraps the prompt with a generation directive and POSTs to {EndpointUrl}/chat/completions with the modalities array set to ["image", "text"].
4. Response Parsing: The system inspects choices[0].message.images[0].image_url.url in the JSON response. Base64 data URIs trigger local storage via FileManager, while remote URLs return as text references.
5. Delivery: The textual description and optional attachment ID return to the client, where the frontend retrieves and renders the image from the persona's attachment store.

Safety Mechanisms and Rate Limiting

ChocolateLMLite implements strict controls to prevent abuse and excessive API consumption:

• User-Explicit Flag: Generation only proceeds when isRequestByUser is true, blocking background or automated tool invocations.
• One-Per-Turn Limit: The isImageGenerated boolean prevents consecutive generation attempts within a single conversation turn, requiring a new user message to reset the state.
• Credential Validation: Missing endpoint URLs or API keys return immediate error responses without attempting network requests.

Implementation Examples

Configuration Setup

# data/general.yaml

EnableImageGeneration: true
ImageGenerationEndpointUrl: "https://openrouter.ai/api/v1"
ImageGenerationApiKey: "sk-xxxxxxxxxxxxxxxxxxxx"
ImageGenerationModel: "google/gemini-2.5-flash-image"

Tool Invocation Payload

When the LLM calls the generation tool, it sends this JSON structure:

{
  "name": "generate_image",
  "arguments": {
    "prompt": "画像を生成してください。夕暮れの海辺に灯る灯台を描いて",
    "isRequestByUser": true
  }
}

Backend Validation Logic

The Tools.GenerateImage method in src/Tools.cs performs preliminary checks before executing the external API call:

if (!isRequestByUser)
    throw new InvalidOperationException("画像生成はユーザー要求時のみ実行可能です。");

if (isImageGenerated)
    throw new InvalidOperationException("画像は既に生成されています。");

var (textResponse, attachmentId) = await _imageGenerater.GenerateImage(prompt);

if (attachmentId.HasValue) {
    lastAttachmentId = attachmentId.Value;
    isImageGenerated = true;
}

return textResponse;

Image Processing and Storage

The ImageGenerater.GenerateImage method handles the HTTP request and persistence logic:

var requestBody = new {
    model = settings.ImageGenerationModel,
    messages = new[] {
        new { role = "user", content = prompt }
    },
    modalities = new[] { "image", "text" }
};

using var httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromSeconds(settings.TimeoutSeconds);
var endpoint = $"{settings.ImageGenerationEndpointUrl.TrimEnd('/')}/chat/completions";

using var request = new HttpRequestMessage(HttpMethod.Post, endpoint);
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", settings.ImageGenerationApiKey);
request.Content = new StringContent(JsonSerializer.Serialize(requestBody),
                                 Encoding.UTF8, "application/json");

var response = await httpClient.SendAsync(request);
var json = await response.Content.ReadAsStringAsync();
using var doc = JsonDocument.Parse(json);
var images = doc.RootElement.GetProperty("choices")[0]
                     .GetProperty("message")
                     .GetProperty("images");

var imageUrl = images[0].GetProperty("image_url").GetProperty("url").GetString();

if (imageUrl.StartsWith("data:image/")) {
    var base64 = imageUrl.Substring(imageUrl.IndexOf(',') + 1);
    var bytes = Convert.FromBase64String(base64);
    var ext = Regex.Match(imageUrl, @"data:image/(.+?);base64").Groups[1].Value;
    var id = _fileManager.SaveAttachmentToActivePersona($"tmp.{ext}", bytes);
    return (textResponse, id);
}

Frontend Response Handling

The backend returns a JSON payload that the frontend (in static/js/talk.js) processes:

{
  "status": "completed",
  "response": "画像生成LLMからの応答内容: 夕暮れの灯台が映っています。",
  "attachmentId": 3
}

When attachmentId is present, the UI fetches the image from the /attachment/{id} endpoint and injects it into the chat view.

Summary

• ImageGenerater.cs handles the HTTP transport to multimodal endpoints and decodes base64 responses into local attachments.
• Tools.cs exposes the generate_image tool to the LLM while enforcing user-validation and per-turn generation limits via isRequestByUser and isImageGenerated.
• FileManager.cs persists generated images to data/attachments/ and returns unique IDs for frontend retrieval.
• Configuration requires setting EnableImageGeneration: true in data/general.yaml with valid endpoint credentials and a compatible model identifier.
• The system distinguishes between base64-encoded images (stored locally) and remote URLs (referenced directly).
• Safety mechanisms ensure generation occurs only on explicit user request and limits output to one image per conversation turn.

Frequently Asked Questions

How do I enable image generation with multimodal LLMs in ChocolateLMLite?

Edit the data/general.yaml file to set EnableImageGeneration: true, provide a valid ImageGenerationEndpointUrl (such as OpenRouter), and specify a compatible multimodal model like google/gemini-2.5-flash-image. The system requires an API key with sufficient credits for the target endpoint.

What safety mechanisms prevent accidental image generation?

ChocolateLMLite implements two primary safeguards in src/Tools.cs. First, the isRequestByUser flag ensures generation only proceeds when explicitly triggered by user input, blocking automated or cascading tool calls. Second, the isImageGenerated boolean limits each conversation turn to a single image generation, preventing rapid successive API calls.

How are generated images stored and retrieved?

When the external API returns a base64-encoded image, ImageGenerater.GenerateImage decodes the payload and calls FileManager.SaveAttachmentToActivePersona, which writes the file to data/attachments/attachment_<id>.<ext>. The method returns an attachment ID that the frontend uses to fetch and display the image via the /attachment/{id} endpoint.

Which multimodal models are compatible with this implementation?

The system works with any LLM that supports the OpenAI-compatible chat completions format with modalities: ["image", "text"] in the response. The reference implementation uses google/gemini-2.5-flash-image through OpenRouter, but other providers offering similar multimodal output formats should function with the same configuration structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}