What Is the CLIP Interrogator in AUTOMATIC1111 and How Does It Reverse‑Engineer Prompts?

The CLIP Interrogator in AUTOMATIC1111's Stable Diffusion WebUI reverse‑engineers image prompts by pairing a BLIP captioning model that generates natural language descriptions with a CLIP vision‑language model that ranks textual tokens from category files by cosine similarity to the image embedding.

The CLIP Interrogator is a core feature of the AUTOMATIC1111/stable‑diffusion‑webui repository that enables users to analyze existing images and reconstruct probable text prompts used to generate them. This tool leverages multimodal deep learning to bridge the gap between visual content and textual descriptions, making it invaluable for prompt discovery, style analysis, and dataset annotation.

How the CLIP Interrogator Reverse‑Engineers Prompts

Core Architecture: BLIP and CLIP Integration

At the heart of the interrogator lies a dual‑model architecture defined in modules/interrogate.py. The BLIP (Bootstrapped Language‑Image Pre‑training) model generates a coherent natural language caption describing the image contents. Simultaneously, the CLIP (Contrastive Language‑Image Pre‑training) model—specifically the ViT‑L/14 variant by default—encodes both the image and candidate text tokens into a shared latent space where semantic similarity can be measured.

Category Files and Token Vocabularies

The interrogator relies on plain‑text category files stored in the interrogator content directory (shared.interrogator.content_dir). These include artists.txt, flavors.txt, mediums.txt, and movements.txt, which contain thousands of possible prompt fragments. According to the source code in modules/interrogate.py (lines 26‑38), if these files are missing, the function download_default_clip_interrogate_categories automatically fetches them from the clip‑interrogator repository. Users can extend functionality by adding custom .txt files to this directory.

Step‑by‑Step Pipeline

1. Model Loading and Initialization

The InterrogateModels.load method handles lazy initialization of both BLIP and CLIP models. BLIP is downloaded from official storage if absent, while CLIP loads via the clip Python package. Both models are moved to devices.device_interrogate and optionally cast to half‑precision (.half()) for GPU acceleration.

2. Caption Generation with BLIP

The generate_caption method processes the input PIL Image by resizing it to 384 × 384 and applying BLIP‑specific normalization. The BLIP decoder outputs a descriptive caption (e.g., "a portrait of a woman in a red dress") that forms the base of the reconstructed prompt.

3. CLIP Image Embedding Extraction

Within the interrogate method, the image undergoes CLIP preprocessing via self.clip_preprocess. The CLIP encode_image function generates a feature vector (image_features) that represents the image in the model's latent space.

4. Category Loading and Token Ranking

The categories method loads each .txt file into a Category named‑tuple containing the category name, top‑N limit, and item list. The rank method tokenizes these entries using clip.tokenize, encodes them with encode_text to produce text features, then computes cosine similarity between image_features and each text feature. Results are scaled by 100, softmax‑normalized, and ranked using torch.topk to extract the highest‑confidence matches.

5. Result Assembly and Formatting

The final output concatenates the BLIP caption with top‑ranked tokens from each category. If shared.opts.interrogate_return_ranks is enabled, confidence percentages are appended to each token (e.g., (artist:Claude Monet:0.921)).

Configuration Options and Performance Tuning

Several settings in modules/shared.py control interrogator behavior:

• interrogate_clip_dict_limit: Limits the number of categories processed per interrogation.
• interrogate_clip_skip_categories: Excludes specific category files from ranking.
• interrogate_clip_num_beams: Controls BLIP beam search width for caption diversity.
• interrogate_return_ranks: Toggles confidence score display in output strings.

Memory management is handled through modules/lowvram.py, which can offload models to CPU via send_everything_to_cpu() when unload() is called, freeing GPU memory for generation tasks.

Programmatic Usage Examples

The following examples demonstrate how to invoke the CLIP Interrogator programmatically within the WebUI environment.

# Example 1 – Simple one‑liner to get a prompt guess from an image file

from modules.interrogate import InterrogateModels
from PIL import Image
import modules.shared as shared
import modules.devices as devices

# Initialise the interrogator with the default category folder

interrogator = InterrogateModels(shared.interrogator.content_dir)

# Load an image (replace with your own path)

pil_img = Image.open("outputs/example.png").convert("RGB")

# Run the interrogation – returns a string like:

# "a portrait of a woman, (artist:Claude Monet:0.921), (style:Impressionism:0.874), ..."

prompt_guess = interrogator.interrogate(pil_img)

print(prompt_guess)

# Example 2 – Accessing the raw scores for further processing

from modules.interrogate import InterrogateModels
from PIL import Image
import torch
import modules.shared as shared
import modules.devices as devices

interrogator = InterrogateModels(shared.interrogator.content_dir)

img = Image.open("outputs/example.png").convert("RGB")

# The interrogate() method builds a human‑readable string.

# To get the underlying data, we can reuse its internal helpers:

interrogator.load()
caption = interrogator.generate_caption(img)

# Encode image once

clip_img = interrogator.clip_preprocess(img).unsqueeze(0).to(devices.device_interrogate)
with torch.no_grad():
    image_features = interrogator.clip_model.encode_image(clip_img)
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)

# Iterate categories manually

for cat in interrogator.categories():
    matches = interrogator.rank(image_features, cat.items, top_count=cat.topn)
    print(f"Category {cat.name}:")
    for txt, score in matches:
        print(f"  {txt:<30} {score:5.2f}%")
interrogator.unload()

Key Implementation Files

• modules/interrogate.py: Contains the core InterrogateModels class, implementing load(), generate_caption(), rank(), and category management logic.
• modules/devices.py: Defines device_interrogate and manages tensor placement for the interrogation pipeline.
• modules/shared.py: Stores global configuration options (shared.opts.interrogate_*) that control model behavior and output formatting.
• modules/lowvram.py: Provides memory management utilities such as send_everything_to_cpu() to offload interrogation models when not in use.

Summary

• The CLIP Interrogator combines BLIP for image captioning and CLIP ViT‑L/14 for semantic similarity ranking to reconstruct text prompts from images.
• It utilizes category files (artists.txt, flavors.txt, etc.) containing thousands of tokens that are ranked against the image embedding using cosine similarity.
• The pipeline executes in modules/interrogate.py through methods including load(), generate_caption(), and rank(), with configurable options for precision, category limits, and confidence display.
• Users can extend functionality by adding custom .txt files to the interrogator content directory or adjusting parameters like interrogate_clip_num_beams for caption quality.

Frequently Asked Questions

What models does the CLIP Interrogator use?

The interrogator utilizes two distinct models: BLIP (Bootstrapped Language‑Image Pre‑training) for generating natural language descriptions of images, and CLIP ViT‑L/14 (Vision Transformer) for encoding images and textual tokens into a comparable latent space. Both models are loaded on‑demand in modules/interrogate.py and can be configured to run on CPU or GPU via devices.device_interrogate.

How accurate is the CLIP Interrogator at guessing prompts?

Accuracy depends on image complexity and how well the visual concepts align with the pre‑defined categories in artists.txt, flavors.txt, and other vocabulary files. The system ranks tokens by cosine similarity confidence scores, but it reconstructs probable prompts rather than exact reverse‑engineering, as it cannot recover the specific seed, sampler, or original prompt syntax used during generation.

Can I add custom categories or artists to the interrogator?

Yes. The interrogator automatically loads any .txt files placed in the interrogator content directory (shared.interrogator.content_dir). Each file should contain one token or phrase per line. The download_default_clip_interrogate_categories function in modules/interrogate.py handles default file retrieval, but users can supplement or replace these with custom vocabularies for specialized domains.

Does the CLIP Interrogator work offline?

Yes, once the BLIP and CLIP models and category files are downloaded. The initial setup requires internet access to fetch model weights and the default category files from the clip‑interrogator repository, but subsequent operations function entirely offline using locally cached resources in the WebUI's models and interrogator directories.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}