# Can SAM Segment All Objects in an Image? A Deep Dive into Meta's Segment Anything Model

> Discover if Meta's Segment Anything Model (SAM) can segment all objects in an image. Learn how the SamAutomaticMaskGenerator wrapper enables this capability.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: deep-dive
- Published: 2026-03-07

---

**Yes, SAM can segment all objects in an image, but only when using the `SamAutomaticMaskGenerator` wrapper; the core model itself requires explicit prompts (points, boxes, or masks) to produce segmentations.**

The Segment Anything Model (SAM), released by Meta's facebookresearch/segment-anything repository, was designed as a promptable segmentation system. While the base architecture excels at zero-shot transfer for specific prompts, the library includes a dedicated generator that enables full-image "segment everything" functionality by automatically creating and processing dense point prompts across the entire image.

## How the Core SAM Model Works (Prompt-Based)

At its foundation, SAM is not an automatic object detector. The core implementation in [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) defines the `Sam` class, whose `forward` method strictly requires prompts to generate masks.

### The Sam.forward Method

The `Sam.forward` method expects a `batched_input` list containing dictionaries with keys like `image`, `point_coords`, `point_labels`, `boxes`, or `mask_inputs`. Without these prompts, the model cannot determine which regions to segment.

```python

# Simplified conceptual view of Sam.forward

def forward(self, batched_input, multimask_output=True):
    # batched_input contains prompts (points, boxes, masks)

    # image_encoder produces embeddings

    # prompt_encoder processes the prompts

    # mask_decoder combines them to produce masks

    pass

```

This architecture means that to segment "everything," you need a strategy to generate prompts for all potential object locations automatically.

## Segmenting All Objects with SamAutomaticMaskGenerator

To solve the prompt dependency, the repository provides `SamAutomaticMaskGenerator` in [`segment_anything/automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py). This class implements a "segment everything" algorithm by sampling a dense grid of point prompts across the image and intelligently merging the results.

### Dense Point Grid Strategy

The generator creates a multi-layer grid of points using `build_all_layer_point_grids`. By default, it samples points in a grid pattern (controlled by `points_per_side`, typically 32 or 64 points per side). These points serve as foreground prompts for the underlying SAM model.

The `_process_batch` method runs SAM on batches of these points, generating candidate masks for each point location. This effectively turns the prompt-based model into an exhaustive object detection system.

### Filtering and Post-Processing

Raw point sampling produces overlapping and low-quality masks. The generator applies several filters:

- **Predicted IoU Thresholding**: Masks with `predicted_iou` below `pred_iou_thresh` (default 0.88) are discarded
- **Stability Scoring**: Masks must exceed a stability threshold to ensure consistent boundaries
- **Non-Maximum Suppression (NMS)**: Overlapping masks are deduplicated using box NMS
- **Small Region Removal**: The `postprocess_small_regions` function removes tiny disconnected components

These steps ensure the final output contains high-quality, non-overlapping masks covering distinct objects.

## Practical Limitations and Hyperparameters

While `SamAutomaticMaskGenerator` can segment most objects, performance depends on configuration and image characteristics.

### Grid Density and Small Objects

The `points_per_side` parameter directly impacts small object detection. A coarse grid (e.g., 16 points per side) may miss tiny objects located between grid points. Increasing to 64 or 128 improves recall but increases computation time quadratically.

```python

# Higher density for small objects

generator = SamAutomaticMaskGenerator(
    sam,
    points_per_side=64,  # Default is 32

    pred_iou_thresh=0.9,  # Stricter quality filter

    stability_score_thresh=0.95
)

```

### Quality Thresholds

Adjusting `pred_iou_thresh` and `stability_score_thresh` trades off between mask quality and coverage. Lower thresholds include more objects but may introduce false positives or fuzzy boundaries.

## Code Examples

### Automatic Mask Generation (Full Image)

This example demonstrates the complete pipeline for segmenting all objects in an image using the automatic generator:

```python
import torch
import numpy as np
import cv2
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

# Load the model

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

# Initialize the automatic mask generator

generator = SamAutomaticMaskGenerator(
    sam,
    points_per_side=32,
    pred_iou_thresh=0.88,
    stability_score_thresh=0.95,
    box_nms_thresh=0.7
)

# Load and convert image

image = cv2.imread("scene.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Generate masks for all objects

masks = generator.generate(image)

print(f"Found {len(masks)} objects")
print(f"First object area: {masks[0]['area']} pixels")
print(f"First object bbox: {masks[0]['bbox']}")  # [x, y, w, h]

```

### Single Object with Explicit Prompts

For comparison, here is the prompt-based approach using `SamPredictor` for segmenting a specific object:

```python
from segment_anything import sam_model_registry, SamPredictor
import numpy as np

# Setup

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)

# Define a point prompt (x, y) = (200, 300)

point = np.array([[200, 300]])
label = np.array([1])  # 1 = foreground

# Predict mask for this specific point

masks, scores, logits = predictor.predict(
    point_coords=point,
    point_labels=label,
    multimask_output=True
)

# Returns 3 masks (multimask_output=True), select based on scores

best_mask = masks[np.argmax(scores)]

```

## Key Source Files

Understanding the implementation requires familiarity with these specific files in the facebookresearch/segment-anything repository:

| File | Role |
|------|------|
| [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) | Core SAM class implementing the `forward` pass, image encoder integration, and prompt-based mask prediction |
| [`segment_anything/automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py) | `SamAutomaticMaskGenerator` class that implements full-image segmentation via dense point sampling and NMS filtering |
| [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) | `SamPredictor` wrapper for interactive, prompt-based segmentation with image embedding caching |
| [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) | Encodes point, box, and mask prompts into embeddings consumed by the mask decoder |
| [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) | Predicts masks and IoU scores from image and prompt embeddings |
| [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py) | Vision Transformer backbone that processes input images into feature embeddings |

## Summary

- **SAM is inherently prompt-based**: The core model in [`sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/sam.py) requires explicit points, boxes, or masks to generate segmentations via `Sam.forward`.
- **Automatic segmentation requires a wrapper**: The `SamAutomaticMaskGenerator` in [`automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/automatic_mask_generator.py) enables "segment everything" functionality by sampling dense point grids and filtering results.
- **Quality depends on configuration**: Parameters like `points_per_side`, `pred_iou_thresh`, and `stability_score_thresh` control the trade-off between coverage, accuracy, and computational cost.
- **Post-processing is essential**: The generator applies NMS, IoU filtering, and small region removal to produce clean, non-overlapping object masks.

## Frequently Asked Questions

### Does SAM require prompts to segment objects?

Yes, the core SAM model strictly requires prompts. The `Sam.forward` method in [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) accepts `batched_input` containing `point_coords`, `boxes`, or `mask_inputs` to specify which regions to segment. Without these prompts, the model cannot determine what to mask. However, the `SamAutomaticMaskGenerator` wrapper automates prompt generation using dense point grids, effectively removing the need for manual prompts.

### What is SamAutomaticMaskGenerator?

`SamAutomaticMaskGenerator` is a high-level utility class in [`segment_anything/automatic_mask_generator.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py) that implements full-image segmentation. It works by generating a dense grid of point prompts across the image (using `build_all_layer_point_grids`), running SAM on batches of these points via `_process_batch`, and then filtering the results using predicted IoU thresholds, stability scoring, and non-maximum suppression. This transforms the prompt-dependent SAM into a "segment everything" tool.

### Can SAM detect very small objects?

SAM can detect small objects, but detection depends on the point grid density configured in `SamAutomaticMaskGenerator`. The `points_per_side` parameter controls how many points are sampled across the image width and height. A coarse grid (e.g., 16 points per side) may miss tiny objects located between sample points, while a denser grid (64 or 128 points per side) improves small object recall at the cost of increased computation time and memory usage.

### Is SAM trained on specific object categories?

No, SAM was trained on a massive dataset (SA-1B) containing over 1 billion masks across 11 million images, covering a broad diversity of objects rather than specific predefined categories. This allows SAM to generalize to novel objects and categories it has never seen during training. However, while it can segment a wide variety of objects, it is not guaranteed to find every conceivable object in every image, particularly those with extreme ambiguity, very low contrast, or unusual visual properties.