How to Use SAM for Automatic Mask Generation: A Complete Guide

Use the SamAutomaticMaskGenerator class from the facebookresearch/segment-anything repository to generate segmentation masks for an entire image without manual prompts by sampling a grid of points and running batch inference with quality filtering.

The Segment Anything Model (SAM) provides a powerful pipeline for automatic mask generation that processes entire images without requiring user input prompts. This functionality is implemented in the SamAutomaticMaskGenerator class, which orchestrates point-grid sampling, multi-scale cropping, and intelligent filtering to produce high-quality object masks. Below is a comprehensive guide on how to use SAM for automatic mask generation, covering the underlying mechanics, implementation details, and configuration options.

How SamAutomaticMaskGenerator Works

The automatic mask generation pipeline operates through a sophisticated multi-step process defined in segment_anything/automatic_mask_generator.py. Understanding these stages helps optimize the generator for your specific use case.

Model Setup and Image Embedding

First, the SAM model is instantiated via sam_model_registry and wrapped in a SamPredictor for efficient embedding reuse. When processing begins, SamPredictor.set_image (defined in segment_anything/predictor.py) computes the image embedding once, allowing rapid mask prediction for multiple point prompts without recomputing backbone features.

Point Grid and Multi-Crop Strategy

The generator creates a grid of point prompts covering the entire image using build_all_layer_point_grids. When crop_n_layers is greater than 0, the image is split into overlapping crops via generate_crop_boxes, enabling detection of small objects that might be missed at full resolution. Each crop is processed independently with its own point grid scaled appropriately.

Mask Prediction and Quality Filtering

For each crop, points are processed in batches (controlled by points_per_batch). The SamPredictor.predict_torch method generates masks, IoU predictions, and low-resolution logits. Masks undergo rigorous quality filtering based on:

• Predicted IoU (pred_iou_thresh): Filters masks by model confidence
• Stability score (stability_score_thresh): Measures mask quality under threshold perturbations
• Box NMS (box_nms_thresh): Removes duplicate masks within each crop

Non-Maximum Suppression and Post-Processing

After processing all crops, a second NMS pass (crop_nms_thresh) resolves overlaps between crops, preferring masks from smaller crops to preserve fine details. If min_mask_region_area is specified, OpenCV removes small isolated regions and holes, followed by a final NMS pass to clean the results.

Basic Usage: Python Implementation

To use SAM for automatic mask generation in Python, instantiate the generator with a loaded model and call the generate method:

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry
import cv2
import numpy as np

# ① Load the SAM model (ViT-H variant)

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")

# ② Create the automatic mask generator

mask_generator = SamAutomaticMaskGenerator(sam)

# ③ Load your image (HxWx3 uint8 array)

image = cv2.imread("input_image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# ④ Generate masks automatically

masks = mask_generator.generate(image)

# ⑤ Inspect results

print(f"Generated {len(masks)} masks")
print(masks[0].keys())  # ['segmentation', 'bbox', 'area', 'predicted_iou', ...]

The generate method returns a list of dictionaries, where each entry contains the binary segmentation mask, bounding box coordinates (XYWH format), mask area, predicted IoU, stability score, and the point prompt that generated it.

Command-Line Interface for Batch Processing

For processing images without writing Python code, use the provided scripts/amg.py utility:

python scripts/amg.py \
    --checkpoint sam_vit_h_4b8939.pth \
    --model-type vit_h \
    --input path/to/images/ \
    --output annotations.json

This script wraps SamAutomaticMaskGenerator with default parameters and outputs a JSON file following the schema documented in the repository README. It supports both single images and directories for batch processing.

Advanced Configuration for Higher Recall

To maximize mask coverage for dataset annotation or dense object detection, tune the generator parameters when instantiating:

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=64,          # Denser grid for more masks

    crop_n_layers=2,             # Two levels of cropping for small objects

    crop_n_points_downscale_factor=2,  # Fewer points per crop layer

    pred_iou_thresh=0.7,         # Lower threshold to keep more masks

    stability_score_thresh=0.9,  # Slightly relaxed stability requirement

    box_nms_thresh=0.7,          # Aggressive duplicate removal per crop

    crop_nms_thresh=0.7,         # Aggressive duplicate removal across crops

    min_mask_region_area=100,    # Remove tiny artifacts

    output_mode="coco_rle",      # COCO-compatible run-length encoding

)

These settings trade inference speed for recall, generating masks for small or ambiguous objects that default parameters might miss. The output_mode="coco_rle" option returns masks in COCO RLE format, compatible with pycocotools for evaluation.

Summary

• SamAutomaticMaskGenerator in segment_anything/automatic_mask_generator.py provides fully automatic mask generation without user prompts.
• The pipeline samples point grids across the image and optional multi-scale crops, batch processes them through SamPredictor, and filters results via IoU thresholds, stability scores, and NMS.
• Basic usage requires only loading a model checkpoint, instantiating the generator, and calling generate(image).
• The scripts/amg.py CLI tool enables batch processing without Python scripting.
• Constructor parameters like points_per_side, crop_n_layers, and pred_iou_thresh allow tuning the speed-recall trade-off for specific applications.

Frequently Asked Questions

What is the difference between SamPredictor and SamAutomaticMaskGenerator?

SamPredictor (defined in segment_anything/predictor.py) is designed for interactive segmentation where you provide specific prompts such as points, boxes, or masks. It computes image embeddings once and allows rapid iteration on different prompts. SamAutomaticMaskGenerator (in segment_anything/automatic_mask_generator.py) is built for fully automatic segmentation—it generates its own point prompts in a grid pattern across the entire image, handles multi-scale cropping internally, and returns all valid masks without requiring user input.

How do I choose the right SAM model variant (vit_h, vit_l, vit_b) for automatic mask generation?

The choice depends on your speed-accuracy requirements and available GPU memory. ViT-H (vit_h) is the largest model with highest accuracy but slowest inference and requires the most VRAM. ViT-L (vit_l) offers a middle ground with good performance on most datasets. ViT-B (vit_b) is the fastest and most lightweight, suitable for real-time applications or edge devices with limited memory. For automatic mask generation, ViT-H is recommended when you need maximum recall for dataset annotation, while ViT-B works well for rapid prototyping.

Why am I getting too many overlapping masks or duplicate detections?

Overlapping masks are a natural result of SAM's prompt-based architecture—different point prompts may generate masks for the same object. The SamAutomaticMaskGenerator applies Non-Maximum Suppression (NMS) at multiple stages to mitigate this: box_nms_thresh filters duplicates within each crop, and crop_nms_thresh resolves overlaps between different crop layers. If you still see too many duplicates, increase these threshold values (e.g., from 0.7 to 0.9) to be more aggressive in removing overlapping boxes. Alternatively, reduce points_per_side to generate fewer initial prompts.

Can I use automatic mask generation on CPU-only machines?

Yes, but inference will be significantly slower. When loading the model, move it to CPU with sam.to(device="cpu") before passing it to SamAutomaticMaskGenerator. For CPU inference, consider using the ViT-B model variant and reducing points_per_side (e.g., to 16 or 32) and crop_n_layers (e.g., to 0 or 1) to minimize computation. The batch processing in predict_torch will still utilize vectorized operations, but expect processing times of several minutes per high-resolution image compared to seconds on GPU.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}