SAM Architecture Explained: The Three Core Components of Meta's Segment Anything Model

The SAM architecture consists of three neural modules—an Image Encoder (ViT), a Prompt Encoder, and a Mask Decoder—that work together to convert images and user prompts into high-quality segmentation masks.

Developed by Meta's Fundamental AI Research (FAIR) team, the Segment Anything Model (SAM) is designed for flexible, promptable image segmentation. The facebookresearch/segment-anything repository implements this architecture through a clean separation of concerns across three specialized neural network components.

The Three Pillars of the SAM Architecture

SAM's design follows an encoder-decoder pattern split into three distinct stages. Each component handles a specific transformation of the input data, allowing the model to process images once while supporting multiple interactive queries.

Image Encoder: Vision Transformer Backbone

The Image Encoder converts raw RGB images into dense feature representations using a Vision Transformer (ViT). In segment_anything/modeling/image_encoder.py, the ImageEncoderViT class processes an input image of size H×W×3 and outputs a low-resolution embedding tensor of shape 256×64×64 (channels × height × width).

This component runs only once per image, making it the computationally heaviest part of the pipeline. The encoder uses a standard ViT architecture with patch embedding, positional encoding, and transformer blocks to capture global image context at reduced spatial resolution.

Prompt Encoder: Sparse and Dense Embeddings

The Prompt Encoder handles user interactions by converting points, bounding boxes, and mask inputs into embedding vectors. Implemented in segment_anything/modeling/prompt_encoder.py, this module produces two distinct output types:

• Sparse embeddings: Token-wise representations for points and boxes that indicate where the user is interested
• Dense embeddings: Pixel-wise maps for partial mask inputs that provide what the mask should look like

Points are encoded using learned positional embeddings for foreground and background locations, while boxes are represented by pairs of point embeddings (top-left and bottom-right corners).

Mask Decoder: Transformer-Based Prediction

The Mask Decoder fuses image and prompt embeddings to generate the final segmentation masks. Located in segment_anything/modeling/mask_decoder.py, this lightweight transformer attends to the image features using the prompt embeddings as queries.

The decoder outputs three key tensors:

• low_res_masks: 256×256 resolution logits before upsampling
• iou_predictions: Quality scores estimating the predicted mask's intersection-over-union with the ground truth
• mask_tokens: Intermediate representations used to generate multiple mask variants

Data Flow Through the SAM Architecture

Understanding how data moves through the SAM architecture clarifies the relationship between components. The pipeline follows this strict sequence:

1. Image preprocessing: Input images are resized to 1024×1024 and normalized
2. Image encoding: The ViT backbone generates image_embeddings (dense feature maps)
3. Prompt encoding: User inputs convert to sparse_embeddings (tokens) and dense_embeddings (maps)
4. Mask decoding: The transformer fusion produces low_res_masks and quality scores
5. Post-processing: Upsampling converts 256×256 logits to original image resolution with binary thresholding

This flow is orchestrated by the Sam class in segment_anything/modeling/sam.py, which manages the end-to-end forward pass including preprocessing and post-processing steps.

Implementing SAM: High-Level and Low-Level APIs

The repository provides two primary interfaces for working with the SAM architecture, catering to different use cases from research to production deployment.

Using SamPredictor for Interactive Segmentation

For interactive applications where an image is loaded once but queried multiple times with different prompts, use the SamPredictor class from segment_anything/predictor.py. This high-level API caches the expensive image embedding, allowing rapid iteration on prompt combinations.

import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2

# Load pre-trained model (vit_h, vit_l, or vit_b)

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")
sam.eval()

predictor = SamPredictor(sam)

# Process image once through Image Encoder

image = cv2.imread("example.jpg")
predictor.set_image(image, image_format="BGR")

# Query repeatedly with different prompts

point = [[250, 400]]
label = [1]  # 1 = foreground, 0 = background

masks, scores, low_res = predictor.predict(
    point_coords=point,
    point_labels=label,
    multimask_output=False,
)

The set_image method triggers the Image Encoder (lines 34-90 in predictor.py), while predict invokes the Prompt Encoder and Mask Decoder (lines 92-144).

Direct Module Access with the Sam Class

For batched inference or custom prompt engineering, instantiate the Sam class directly with its three sub-modules:

from segment_anything.modeling import Sam, ImageEncoderViT, PromptEncoder, MaskDecoder

# Initialize components with model configuration

image_encoder = ImageEncoderViT(
    img_size=1024, patch_size=16, embed_dim=768,
    depth=12, num_heads=12, out_chans=256
)
prompt_encoder = PromptEncoder(
    embed_dim=256,
    image_embedding_size=(64, 64),
    input_image_size=(1024, 1024),
    mask_in_chans=16
)
mask_decoder = MaskDecoder(
    transformer_dim=256,
    transformer=TwoWayTransformer(depth=2, heads=8, mlp_dim=2048, dim=256),
    num_multimask_outputs=3,
)

sam = Sam(image_encoder, prompt_encoder, mask_decoder)

# Batched forward pass

batched_input = [{
    "image": torch.randn(3, 1024, 1024),
    "original_size": (1024, 1024),
    "point_coords": torch.tensor([[[250, 400]]]),
    "point_labels": torch.tensor([[1]]),
}]
outputs = sam(batched_input, multimask_output=True)

This low-level approach mirrors the exact call chain in sam.py (lines 53-131), allowing direct manipulation of embeddings and intermediate representations.

Key Source Files in the facebookresearch/segment-anything Repository

File	Component	Purpose
segment_anything/modeling/image_encoder.py	Image Encoder	ViT backbone producing dense feature maps
segment_anything/modeling/prompt_encoder.py	Prompt Encoder	Embeddings for points, boxes, and masks
segment_anything/modeling/mask_decoder.py	Mask Decoder	Transformer fusion and mask generation
segment_anything/modeling/sam.py	Orchestration	Top-level Sam class wiring components
segment_anything/predictor.py	High-level API	SamPredictor for cached interactive use
segment_anything/utils/transforms.py	Utilities	Image resizing and normalization

Summary

• SAM architecture comprises three specialized neural modules: a Vision Transformer Image Encoder, a Prompt Encoder for user inputs, and a Mask Decoder for final predictions.
• The Image Encoder runs once per image, generating 256-channel embeddings at 64×64 resolution, while the Prompt and Mask Decoders execute per query.
• SamPredictor provides an optimized interface for interactive use cases by caching image embeddings, whereas the Sam class offers direct access for research and batched inference.
• All components are implemented in separate files under segment_anything/modeling/, following clean separation of concerns for maintainability and extension.

Frequently Asked Questions

What input resolution does SAM's Image Encoder expect?

The Image Encoder in image_encoder.py expects images resized to 1024×1024 pixels. The predictor handles this automatically via ResizeLongestSide in segment_anything/utils/transforms.py, padding images to maintain aspect ratio before feeding them to the ViT backbone. Output embeddings maintain a 16× downsampling ratio, resulting in 64×64 spatial features.

How does SAM handle different types of prompts simultaneously?

The Prompt Encoder processes multiple prompt types in parallel. Points and boxes generate sparse embeddings (token vectors), while partial mask inputs create dense embeddings (pixel-aligned maps). In mask_decoder.py, sparse embeddings attend to the image features via cross-attention, while dense embeddings are added element-wise to the image embeddings before the transformer blocks.

What is the difference between Sam and SamPredictor?

Sam (in sam.py) is the raw PyTorch module implementing the SAM architecture with a forward() method accepting batched dictionaries. SamPredictor (in predictor.py) is a convenience wrapper that manages image preprocessing, caches image embeddings to avoid redundant computation, and converts between numpy arrays and tensors. Use SamPredictor for interactive applications; use Sam for research requiring direct tensor manipulation.

Why does SAM generate three masks for a single prompt?

The Mask Decoder is designed to output multiple mask candidates to handle ambiguous prompts. When multimask_output=True, the decoder produces three masks representing different valid interpretations of the prompt (e.g., whole object vs. sub-part). Each mask receives an IoU score in iou_predictions, allowing downstream applications to select the highest quality prediction or present options to users.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}