Diverse Description Generation Framework: Leveraging LLMs for Caption Synthesis in RWKV-CLIP

The diverse description generation framework is a multi-source synthesis pipeline that leverages Large Language Models (LLMs) to generate rich image captions by combining web-based texts, synthetic captions, and visual detection tags, followed by tag-consistency refinement to eliminate hallucinations.

The deepglint/rwkv-clip repository introduces a sophisticated approach to vision-language pre-training that addresses the limitations of single-caption supervision. At its core, the diverse description generation framework synthesizes high-quality training data by orchestrating LLMs to process multiple information sources simultaneously. This methodology enables the creation of semantically rich descriptions that significantly enhance zero-shot retrieval and classification performance in vision-language models.

What Is the Diverse Description Generation Framework?

The diverse description generation framework is a structured pipeline designed to augment image-text datasets with high-quality captions derived from three complementary information sources. As documented in README.md (lines 22-35), the framework aggregates:

1. Web-based texts: Raw textual material scraped from the internet associated with the image
2. Synthetic captions: Automatically generated descriptions, often derived from CLIP-style prompts or template-based generation
3. Detection tags: Visual object tags extracted from the image (e.g., "dog", "car", "park") that provide concrete visual grounding

The framework feeds these sources into an LLM, allowing the model to generate candidate descriptions that synthesize information across modalities. Unlike traditional caption generation that relies on single-source supervision, this multi-source approach produces diverse descriptions that capture varied aspects of image content, improving the robustness of vision-language training.

How the Framework Synthesizes Captions Using LLMs

The caption synthesis process follows a three-stage architectural flow: prompt construction, LLM inference, and tag-consistency refinement. Each stage leverages specific components from the RWKV-CLIP codebase to ensure factual grounding and semantic richness.

Prompt Construction with Caption Templates

The framework utilizes structured caption templates to standardize LLM inputs. These templates are defined in utils/template.json (lines 2-4), which contains dataset-specific collections following formats such as "a photo of a {}." or "a photo of {}, a type of food."

During prompt construction, the system:

• Loads the appropriate template for the target dataset (e.g., CIFAR-10 or ImageNet variants)
• Fills template slots with candidate labels derived from detection tags
• Combines the filled template with web texts and synthetic captions to form a comprehensive prompt that instructs the LLM to generate a diverse description

LLM Inference and Raw Description Generation

With the constructed prompt, the framework interfaces with any compatible LLM (such as GPT-style models) through standard chat completion APIs. The LLM receives the multi-source context and generates candidate sentences that synthesize information from web texts, synthetic captions, and the visual concepts indicated by detection tags.

This generation step produces raw descriptions that are semantically rich but may contain hallucinated elements or details not supported by the actual image content.

Tag-Consistency Refinement to Reduce Hallucinations

The critical refinement stage ensures generated descriptions align with actual visual content. As implemented in the repository (README.md line 22), the framework applies a tag-consistency check that constrains the LLM output using the original detection tags.

The refinement process:

• Verifies that all detection tags appear in the generated description (or are semantically aligned)
• Re-prompts the LLM or appends missing tags when inconsistencies are detected
• Enforces visual grounding by ensuring the final description contains references to all detected objects

This constraint mechanism effectively reduces hallucinations—the common problem where language models invent objects not present in the image—while maintaining the semantic richness provided by the LLM.

Implementation: Building Diverse Descriptions with RWKV-CLIP

The following code examples demonstrate how to implement the diverse description generation pipeline using utilities from the deepglint/rwkv-clip repository. These snippets illustrate prompt construction, LLM integration, tag-consistency enforcement, and zero-shot inference.

Constructing LLM Prompts from Detection Tags

This example demonstrates loading caption templates from utils/template.json and constructing prompts for LLM inference:

import json
from PIL import Image

# Load caption templates from utils/template.json

with open("utils/template.json") as f:
    templates = json.load(f)

# Select appropriate template for your dataset

caption_template = templates["cifar10"][0]  # "a photo of a {}."

# Load image and define detection tags (visual grounding)

image_path = "figure/Diverse_description_generation_00.png"
image = Image.open(image_path)
detection_tags = ["dog", "park", "frisbee"]

# Build comprehensive prompt for the LLM

prompt = f"""You are given an image that contains {', '.join(detection_tags)}.
Generate a diverse description using the following template:
{caption_template}
Provide only the filled sentence."""

print("Prompt sent to LLM:")
print(prompt)

Enforcing Tag Consistency in Generated Captions

After receiving raw descriptions from the LLM, implement tag-consistency refinement to ensure visual grounding and reduce hallucinations:

def enforce_tag_consistency(text, detection_tags):
    """
    Ensures all detection tags appear in the generated description.
    Missing tags are appended to enforce visual grounding.
    """
    text_lower = text.lower()
    missing_tags = [tag for tag in detection_tags if tag not in text_lower]
    
    if missing_tags:
        # Append missing tags to maintain consistency with detection

        text = text.rstrip(".") + " and " + " and ".join(missing_tags) + "."
    
    return text

# Example usage with mock LLM response

raw_response = "a photo of a dog playing."
final_description = enforce_tag_consistency(raw_response, detection_tags)

print("Raw LLM output:", raw_response)
print("Refined description:", final_description)

# Output: "a photo of a dog playing and park and frisbee."

Zero-Shot Inference with Generated Descriptions

Use the refined diverse descriptions with the RWKV-CLIP model for zero-shot classification or retrieval:

import torch
import clip
from model.utils import create_RWKV_Model
from model_config.utils_notebook import load_model_configs
from open_clip.transform import image_transform
from PIL import Image

# Load model configuration from model_config/RWKV_CLIP_B32.json

args = load_model_configs('model_config/RWKV_CLIP_B32.json')
model = create_RWKV_Model(args, model_weight_path="Model_pretrained_weight.pt")
model.eval().to("cuda" if torch.cuda.is_available() else "cpu")

# Prepare image with transforms

transform = image_transform(args.input_size, False)
image = Image.open("figure/Diverse_description_generation_00.png")
img_tensor = transform(image).unsqueeze(0).to(model.device)

# Use the diverse description generated via LLM

diverse_caption = "a photo of a dog playing and park and frisbee."
text_tensor = clip.tokenize([diverse_caption]).to(model.device)

# Compute image-text similarity

with torch.no_grad():
    img_feat, txt_feat, _ = model(img_tensor, text_tensor)
    img_feat /= img_feat.norm(dim=-1, keepdim=True)
    txt_feat /= txt_feat.norm(dim=-1, keepdim=True)
    similarity = (100.0 * img_feat @ txt_feat.T).softmax(dim=-1)

print(f"Similarity score: {similarity.item():.4f}")

Summary

The diverse description generation framework in deepglint/rwkv-clip provides a robust methodology for creating high-quality training data for vision-language models:

• Multi-source aggregation: Combines web texts, synthetic captions, and detection tags to create comprehensive LLM prompts.
• Template-based standardization: Uses utils/template.json to maintain consistent prompt structures across different datasets.
• Hallucination mitigation: Implements tag-consistency refinement to ensure all detection tags appear in final descriptions, constraining LLM outputs to actual visual content.
• Enhanced training data: Produces the diverse descriptions released for the YFCC15M dataset, improving zero-shot performance in RWKV-CLIP.

Frequently Asked Questions

How does the diverse description generation framework prevent LLM hallucinations?

The framework implements a tag-consistency refinement stage that constrains LLM outputs using visual detection tags. After the LLM generates candidate descriptions, the system verifies that all detection tags (e.g., "dog", "park") appear in the text. If tags are missing, the framework either re-prompts the LLM or appends the missing tags to the output. This enforcement ensures that the final diverse description remains grounded in actual visual content rather than invented details.

What role do caption templates play in the diverse description generation framework?

Caption templates stored in utils/template.json provide the structural foundation for LLM prompt construction. These templates follow formats such as "a photo of a {}." or "a photo of {}, a type of food.", where placeholders are filled with candidate labels derived from detection tags. The templates ensure consistency across different datasets (like CIFAR-10 or ImageNet variants) and provide the LLM with clear instructions for how to format the diverse description, standardizing the output structure while allowing content variation.

Can the diverse description generation framework work with any LLM?

Yes, the framework is designed to be model-agnostic and can interface with any compatible LLM that supports chat completion APIs, including GPT-style models. The repository does not hardcode a specific LLM implementation; instead, it provides the prompt construction and refinement infrastructure. Users can integrate their preferred LLM client by replacing the placeholder API calls in the generation scripts, allowing the framework to leverage state-of-the-art language models while maintaining the tag-consistency constraints specific to the RWKV-CLIP training pipeline.

How are the generated diverse descriptions used in RWKV-CLIP training?

The diverse descriptions serve as high-quality textual supervision for training the RWKV-CLIP vision-language model. Specifically, the framework generates these descriptions for large-scale datasets like YFCC15M, creating multiple rich captions per image that combine web text, synthetic prompts, and detection tag information. During training, these descriptions provide more comprehensive semantic coverage than single captions, enabling the model to learn finer-grained associations between visual features and textual concepts. This improved textual supervision directly translates to better zero-shot retrieval and classification performance on downstream tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}