# SAM Architecture Explained: The Three Core Components of Meta's Segment Anything Model

> Explore the SAM architecture and its three core components: Image Encoder, Prompt Encoder, and Mask Decoder. Understand how Meta's Segment Anything Model generates precise segmentation masks.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: architecture
- Published: 2026-03-07

---

**The SAM architecture consists of three neural modules—an Image Encoder (ViT), a Prompt Encoder, and a Mask Decoder—that work together to convert images and user prompts into high-quality segmentation masks.**

Developed by Meta's Fundamental AI Research (FAIR) team, the Segment Anything Model (SAM) is designed for flexible, promptable image segmentation. The [facebookresearch/segment-anything](https://github.com/facebookresearch/segment-anything) repository implements this architecture through a clean separation of concerns across three specialized neural network components.

## The Three Pillars of the SAM Architecture

SAM's design follows an encoder-decoder pattern split into three distinct stages. Each component handles a specific transformation of the input data, allowing the model to process images once while supporting multiple interactive queries.

### Image Encoder: Vision Transformer Backbone

The **Image Encoder** converts raw RGB images into dense feature representations using a Vision Transformer (ViT). In [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py), the `ImageEncoderViT` class processes an input image of size `H×W×3` and outputs a low-resolution embedding tensor of shape `256×64×64` (channels × height × width).

This component runs only once per image, making it the computationally heaviest part of the pipeline. The encoder uses a standard ViT architecture with patch embedding, positional encoding, and transformer blocks to capture global image context at reduced spatial resolution.

### Prompt Encoder: Sparse and Dense Embeddings

The **Prompt Encoder** handles user interactions by converting points, bounding boxes, and mask inputs into embedding vectors. Implemented in [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py), this module produces two distinct output types:

- **Sparse embeddings**: Token-wise representations for points and boxes that indicate *where* the user is interested
- **Dense embeddings**: Pixel-wise maps for partial mask inputs that provide *what* the mask should look like

Points are encoded using learned positional embeddings for foreground and background locations, while boxes are represented by pairs of point embeddings (top-left and bottom-right corners).

### Mask Decoder: Transformer-Based Prediction

The **Mask Decoder** fuses image and prompt embeddings to generate the final segmentation masks. Located in [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py), this lightweight transformer attends to the image features using the prompt embeddings as queries.

The decoder outputs three key tensors:
- `low_res_masks`: 256×256 resolution logits before upsampling
- `iou_predictions`: Quality scores estimating the predicted mask's intersection-over-union with the ground truth
- `mask_tokens`: Intermediate representations used to generate multiple mask variants

## Data Flow Through the SAM Architecture

Understanding how data moves through the SAM architecture clarifies the relationship between components. The pipeline follows this strict sequence:

1. **Image preprocessing**: Input images are resized to 1024×1024 and normalized
2. **Image encoding**: The ViT backbone generates `image_embeddings` (dense feature maps)
3. **Prompt encoding**: User inputs convert to `sparse_embeddings` (tokens) and `dense_embeddings` (maps)
4. **Mask decoding**: The transformer fusion produces `low_res_masks` and quality scores
5. **Post-processing**: Upsampling converts 256×256 logits to original image resolution with binary thresholding

This flow is orchestrated by the `Sam` class in [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py), which manages the end-to-end forward pass including preprocessing and post-processing steps.

## Implementing SAM: High-Level and Low-Level APIs

The repository provides two primary interfaces for working with the SAM architecture, catering to different use cases from research to production deployment.

### Using SamPredictor for Interactive Segmentation

For interactive applications where an image is loaded once but queried multiple times with different prompts, use the `SamPredictor` class from [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py). This high-level API caches the expensive image embedding, allowing rapid iteration on prompt combinations.

```python
import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2

# Load pre-trained model (vit_h, vit_l, or vit_b)

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
sam.eval()

predictor = SamPredictor(sam)

# Process image once through Image Encoder

image = cv2.imread("example.jpg")
predictor.set_image(image, image_format="BGR")

# Query repeatedly with different prompts

point = [[250, 400]]
label = [1]  # 1 = foreground, 0 = background

masks, scores, low_res = predictor.predict(
    point_coords=point,
    point_labels=label,
    multimask_output=False,
)

```

The `set_image` method triggers the Image Encoder (lines 34-90 in [`predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/predictor.py)), while `predict` invokes the Prompt Encoder and Mask Decoder (lines 92-144).

### Direct Module Access with the Sam Class

For batched inference or custom prompt engineering, instantiate the `Sam` class directly with its three sub-modules:

```python
from segment_anything.modeling import Sam, ImageEncoderViT, PromptEncoder, MaskDecoder

# Initialize components with model configuration

image_encoder = ImageEncoderViT(
    img_size=1024, patch_size=16, embed_dim=768,
    depth=12, num_heads=12, out_chans=256
)
prompt_encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),
    input_image_size=(1024, 1024),
    mask_in_chans=16
)
mask_decoder = MaskDecoder(
    transformer_dim=256,
    transformer=TwoWayTransformer(depth=2, heads=8, mlp_dim=2048, dim=256),
    num_multimask_outputs=3,
)

sam = Sam(image_encoder, prompt_encoder, mask_decoder)

# Batched forward pass

batched_input = [{
    "image": torch.randn(3, 1024, 1024),
    "original_size": (1024, 1024),
    "point_coords": torch.tensor([[[250, 400]]]),
    "point_labels": torch.tensor([[1]]),
}]
outputs = sam(batched_input, multimask_output=True)

```

This low-level approach mirrors the exact call chain in [`sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/sam.py) (lines 53-131), allowing direct manipulation of embeddings and intermediate representations.

## Key Source Files in the facebookresearch/segment-anything Repository

| File | Component | Purpose |
|------|-----------|---------|
| [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py) | Image Encoder | ViT backbone producing dense feature maps |
| [`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py) | Prompt Encoder | Embeddings for points, boxes, and masks |
| [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) | Mask Decoder | Transformer fusion and mask generation |
| [`segment_anything/modeling/sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/sam.py) | Orchestration | Top-level `Sam` class wiring components |
| [`segment_anything/predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/predictor.py) | High-level API | `SamPredictor` for cached interactive use |
| [`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py) | Utilities | Image resizing and normalization |

## Summary

- **SAM architecture** comprises three specialized neural modules: a Vision Transformer Image Encoder, a Prompt Encoder for user inputs, and a Mask Decoder for final predictions.
- The Image Encoder runs once per image, generating 256-channel embeddings at 64×64 resolution, while the Prompt and Mask Decoders execute per query.
- `SamPredictor` provides an optimized interface for interactive use cases by caching image embeddings, whereas the `Sam` class offers direct access for research and batched inference.
- All components are implemented in separate files under `segment_anything/modeling/`, following clean separation of concerns for maintainability and extension.

## Frequently Asked Questions

### What input resolution does SAM's Image Encoder expect?

The Image Encoder in [`image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/image_encoder.py) expects images resized to **1024×1024 pixels**. The predictor handles this automatically via `ResizeLongestSide` in [`segment_anything/utils/transforms.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/utils/transforms.py), padding images to maintain aspect ratio before feeding them to the ViT backbone. Output embeddings maintain a 16× downsampling ratio, resulting in 64×64 spatial features.

### How does SAM handle different types of prompts simultaneously?

The Prompt Encoder processes multiple prompt types in parallel. Points and boxes generate **sparse embeddings** (token vectors), while partial mask inputs create **dense embeddings** (pixel-aligned maps). In [`mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/mask_decoder.py), sparse embeddings attend to the image features via cross-attention, while dense embeddings are added element-wise to the image embeddings before the transformer blocks.

### What is the difference between Sam and SamPredictor?

`Sam` (in [`sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/sam.py)) is the raw PyTorch module implementing the **SAM architecture** with a `forward()` method accepting batched dictionaries. `SamPredictor` (in [`predictor.py`](https://github.com/facebookresearch/segment-anything/blob/main/predictor.py)) is a convenience wrapper that manages image preprocessing, caches image embeddings to avoid redundant computation, and converts between numpy arrays and tensors. Use `SamPredictor` for interactive applications; use `Sam` for research requiring direct tensor manipulation.

### Why does SAM generate three masks for a single prompt?

The Mask Decoder is designed to output multiple mask candidates to handle **ambiguous prompts**. When `multimask_output=True`, the decoder produces three masks representing different valid interpretations of the prompt (e.g., whole object vs. sub-part). Each mask receives an IoU score in `iou_predictions`, allowing downstream applications to select the highest quality prediction or present options to users.