# What Is the CLIP Interrogator in AUTOMATIC1111 and How Does It Reverse‑Engineer Prompts?

> Learn how the CLIP Interrogator in AUTOMATIC1111 reverse-engineers image prompts using BLIP and CLIP models to generate descriptive captions and discover optimal tokens for your AI art.

- Repository: [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
- Tags: deep-dive
- Published: 2026-02-24

---

**The CLIP Interrogator in AUTOMATIC1111's Stable Diffusion WebUI reverse‑engineers image prompts by pairing a BLIP captioning model that generates natural language descriptions with a CLIP vision‑language model that ranks textual tokens from category files by cosine similarity to the image embedding.**

The CLIP Interrogator is a core feature of the AUTOMATIC1111/stable‑diffusion‑webui repository that enables users to analyze existing images and reconstruct probable text prompts used to generate them. This tool leverages multimodal deep learning to bridge the gap between visual content and textual descriptions, making it invaluable for prompt discovery, style analysis, and dataset annotation.

## How the CLIP Interrogator Reverse‑Engineers Prompts

### Core Architecture: BLIP and CLIP Integration

At the heart of the interrogator lies a dual‑model architecture defined in [`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py). The **BLIP (Bootstrapped Language‑Image Pre‑training)** model generates a coherent natural language caption describing the image contents. Simultaneously, the **CLIP (Contrastive Language‑Image Pre‑training)** model—specifically the **ViT‑L/14** variant by default—encodes both the image and candidate text tokens into a shared latent space where semantic similarity can be measured.

### Category Files and Token Vocabularies

The interrogator relies on plain‑text category files stored in the interrogator content directory (`shared.interrogator.content_dir`). These include [`artists.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/artists.txt), [`flavors.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/flavors.txt), [`mediums.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/mediums.txt), and [`movements.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/movements.txt), which contain thousands of possible prompt fragments. According to the source code in [`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py) (lines 26‑38), if these files are missing, the function `download_default_clip_interrogate_categories` automatically fetches them from the clip‑interrogator repository. Users can extend functionality by adding custom `.txt` files to this directory.

## Step‑by‑Step Pipeline

### 1. Model Loading and Initialization

The `InterrogateModels.load` method handles lazy initialization of both BLIP and CLIP models. BLIP is downloaded from official storage if absent, while CLIP loads via the `clip` Python package. Both models are moved to `devices.device_interrogate` and optionally cast to half‑precision (`.half()`) for GPU acceleration.

### 2. Caption Generation with BLIP

The `generate_caption` method processes the input PIL Image by resizing it to **384 × 384** and applying BLIP‑specific normalization. The BLIP decoder outputs a descriptive caption (e.g., "a portrait of a woman in a red dress") that forms the base of the reconstructed prompt.

### 3. CLIP Image Embedding Extraction

Within the `interrogate` method, the image undergoes CLIP preprocessing via `self.clip_preprocess`. The CLIP `encode_image` function generates a **feature vector** (`image_features`) that represents the image in the model's latent space.

### 4. Category Loading and Token Ranking

The `categories` method loads each `.txt` file into a `Category` named‑tuple containing the category name, top‑N limit, and item list. The `rank` method tokenizes these entries using `clip.tokenize`, encodes them with `encode_text` to produce **text features**, then computes cosine similarity between `image_features` and each text feature. Results are scaled by 100, softmax‑normalized, and ranked using `torch.topk` to extract the highest‑confidence matches.

### 5. Result Assembly and Formatting

The final output concatenates the BLIP caption with top‑ranked tokens from each category. If `shared.opts.interrogate_return_ranks` is enabled, confidence percentages are appended to each token (e.g., `(artist:Claude Monet:0.921)`).

## Configuration Options and Performance Tuning

Several settings in [`modules/shared.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/shared.py) control interrogator behavior:

- **`interrogate_clip_dict_limit`**: Limits the number of categories processed per interrogation.
- **`interrogate_clip_skip_categories`**: Excludes specific category files from ranking.
- **`interrogate_clip_num_beams`**: Controls BLIP beam search width for caption diversity.
- **`interrogate_return_ranks`**: Toggles confidence score display in output strings.

Memory management is handled through [`modules/lowvram.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/lowvram.py), which can offload models to CPU via `send_everything_to_cpu()` when `unload()` is called, freeing GPU memory for generation tasks.

## Programmatic Usage Examples

The following examples demonstrate how to invoke the CLIP Interrogator programmatically within the WebUI environment.

```python

# Example 1 – Simple one‑liner to get a prompt guess from an image file

from modules.interrogate import InterrogateModels
from PIL import Image
import modules.shared as shared
import modules.devices as devices

# Initialise the interrogator with the default category folder

interrogator = InterrogateModels(shared.interrogator.content_dir)

# Load an image (replace with your own path)

pil_img = Image.open("outputs/example.png").convert("RGB")

# Run the interrogation – returns a string like:

# "a portrait of a woman, (artist:Claude Monet:0.921), (style:Impressionism:0.874), ..."

prompt_guess = interrogator.interrogate(pil_img)

print(prompt_guess)

```

```python

# Example 2 – Accessing the raw scores for further processing

from modules.interrogate import InterrogateModels
from PIL import Image
import torch
import modules.shared as shared
import modules.devices as devices

interrogator = InterrogateModels(shared.interrogator.content_dir)

img = Image.open("outputs/example.png").convert("RGB")

# The interrogate() method builds a human‑readable string.

# To get the underlying data, we can reuse its internal helpers:

interrogator.load()
caption = interrogator.generate_caption(img)

# Encode image once

clip_img = interrogator.clip_preprocess(img).unsqueeze(0).to(devices.device_interrogate)
with torch.no_grad():
    image_features = interrogator.clip_model.encode_image(clip_img)
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)

# Iterate categories manually

for cat in interrogator.categories():
    matches = interrogator.rank(image_features, cat.items, top_count=cat.topn)
    print(f"Category {cat.name}:")
    for txt, score in matches:
        print(f"  {txt:<30} {score:5.2f}%")
interrogator.unload()

```

## Key Implementation Files

- **[`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py)**: Contains the core `InterrogateModels` class, implementing `load()`, `generate_caption()`, `rank()`, and category management logic.
- **[`modules/devices.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/devices.py)**: Defines `device_interrogate` and manages tensor placement for the interrogation pipeline.
- **[`modules/shared.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/shared.py)**: Stores global configuration options (`shared.opts.interrogate_*`) that control model behavior and output formatting.
- **[`modules/lowvram.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/lowvram.py)**: Provides memory management utilities such as `send_everything_to_cpu()` to offload interrogation models when not in use.

## Summary

- The CLIP Interrogator combines **BLIP** for image captioning and **CLIP ViT‑L/14** for semantic similarity ranking to reconstruct text prompts from images.
- It utilizes **category files** ([`artists.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/artists.txt), [`flavors.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/flavors.txt), etc.) containing thousands of tokens that are ranked against the image embedding using cosine similarity.
- The pipeline executes in [`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py) through methods including `load()`, `generate_caption()`, and `rank()`, with configurable options for precision, category limits, and confidence display.
- Users can extend functionality by adding custom `.txt` files to the interrogator content directory or adjusting parameters like `interrogate_clip_num_beams` for caption quality.

## Frequently Asked Questions

### What models does the CLIP Interrogator use?

The interrogator utilizes two distinct models: **BLIP** (Bootstrapped Language‑Image Pre‑training) for generating natural language descriptions of images, and **CLIP ViT‑L/14** (Vision Transformer) for encoding images and textual tokens into a comparable latent space. Both models are loaded on‑demand in [`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py) and can be configured to run on CPU or GPU via `devices.device_interrogate`.

### How accurate is the CLIP Interrogator at guessing prompts?

Accuracy depends on image complexity and how well the visual concepts align with the pre‑defined categories in [`artists.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/artists.txt), [`flavors.txt`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/flavors.txt), and other vocabulary files. The system ranks tokens by cosine similarity confidence scores, but it reconstructs *probable* prompts rather than exact reverse‑engineering, as it cannot recover the specific seed, sampler, or original prompt syntax used during generation.

### Can I add custom categories or artists to the interrogator?

Yes. The interrogator automatically loads any `.txt` files placed in the `interrogator` content directory (`shared.interrogator.content_dir`). Each file should contain one token or phrase per line. The `download_default_clip_interrogate_categories` function in [`modules/interrogate.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/interrogate.py) handles default file retrieval, but users can supplement or replace these with custom vocabularies for specialized domains.

### Does the CLIP Interrogator work offline?

Yes, once the BLIP and CLIP models and category files are downloaded. The initial setup requires internet access to fetch model weights and the default category files from the clip‑interrogator repository, but subsequent operations function entirely offline using locally cached resources in the WebUI's `models` and `interrogator` directories.