# SAM Mask Decoder Functionality: Architecture and Implementation Guide

> Uncover the SAM mask decoder functionality. Learn its architecture and implementation for pixel-accurate segmentation masks and quality scores. A detailed guide for developers.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: architecture
- Published: 2026-03-07

---

**The SAM mask decoder transforms ViT-based image embeddings and sparse/dense prompt embeddings into pixel-accurate segmentation masks and IoU quality scores through a six-stage pipeline involving learned tokens, a lightweight transformer, and hypernetwork-based mask generation.**

The `MaskDecoder` class in the `facebookresearch/segment-anything` repository serves as the critical bridge between the image encoder's high-dimensional features and the final segmentation outputs. This component implements the "prompt-to-mask" design described in the original SAM paper, leveraging a shared transformer architecture and dynamic weight generation to produce masks from arbitrary user prompts. Understanding the SAM mask decoder functionality is essential for customizing segmentation workflows or extending the model for specialized computer vision tasks.

## Six-Stage Processing Pipeline

The decoder processes inputs through six distinct logical stages defined in [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py).

### 1. Token Preparation

At initialization (lines 49-52), the constructor instantiates learnable embedding weights for `self.iou_token` (1 × C) and `self.mask_tokens` (N+1 × C). These vectors represent the output tokens that the transformer will decode into final predictions, where **C** denotes the transformer dimension and **N** represents the number of multimask outputs.

### 2. Input Token Assembly

The `forward` method constructs `output_tokens` by stacking the IoU token and mask token embeddings, then tiles this sequence across the batch dimension (lines 21-24). This creates a uniform token sequence that concatenates with **sparse prompt embeddings** (points and boxes) to form the complete transformer input.

### 3. Fusion with Image Features

The decoder repeats image embeddings via `torch.repeat_interleave` to match the token batch size (line 26), then adds **dense prompt embeddings** (mask inputs) element-wise: `src = src + dense_prompt_embeddings`. This operation fuses spatial image features with optional mask prompts before transformer processing.

### 4. Transformer Decoding

A shared transformer module processes the fused sequence through the call `hs, src = self.transformer(src, pos_src, tokens)` at line 31. The output `hs` contains per-token hidden states where `hs[:, 0]` represents the **IoU token** and subsequent indices contain the **mask tokens** (lines 33-35).

### 5. Upscaling and Hypernetwork Mask Generation

The spatial output `src` reshapes to 2-D and undergoes two **transposed convolution** layers (lines 53-59) to upscale features from the transformer resolution to the final mask resolution. Simultaneously, each mask token passes through a small **hypernetwork MLP** (lines 60-65) to generate dynamic weights. These weights multiply against the upsampled feature map via matrix multiplication (lines 43-45) to produce raw mask logits.

### 6. IoU Quality Prediction

A separate MLP head processes the IoU token (`hs[:, 0]`) to predict scalar quality scores for each mask candidate (lines 67-70). This auxiliary prediction enables the model to rank multiple mask hypotheses by estimated intersection-over-union accuracy.

## Multimask Output Handling

The `forward` method generates all masks upfront using `self.num_mask_tokens = num_multimask_outputs + 1`, then selects outputs based on the `multimask_output` boolean flag (lines 101-108).

- **`multimask_output=False`**: Returns `slice(0, 1)` selecting only the single best mask
- **`multimask_output=True`**: Returns `slice(1, None)` providing multiple candidate masks (typically 3) for ambiguous prompts

This slicing mechanism allows the same forward pass to support both single-mask precision and multi-hypothesis diversity modes.

## Code Examples

### High-Level Inference with SamPredictor

```python
from segment_anything import sam_model_registry, SamPredictor
import cv2
import numpy as np

# Load a pretrained SAM checkpoint (e.g., "vit_h")

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

# Load an image

image = cv2.imread("dog.jpg")
predictor.set_image(image)

# Example prompt: a single point (x, y) with label=1 (foreground)

point = np.array([[250, 340]])
label = np.array([1])

# Get masks and IoU scores

masks, scores, logits = predictor.predict(
    point_coords=point,
    point_labels=label,
    multimask_output=True,        # returns 3 masks + the best one

)

print("Returned", masks.shape[0], "masks; scores:", scores)

```

Behind the scenes, `SamPredictor.predict` builds the sparse/dense prompt embeddings, forwards them through the **MaskDecoder**, and upsamples the mask logits to the original image size.

### Direct MaskDecoder Access

```python
import torch
from segment_anything.modeling.sam import Sam
from segment_anything.modeling.mask_decoder import MaskDecoder

# 1️⃣ Build the full SAM model (image encoder + transformer + mask decoder)

sam = Sam(image_encoder='vit_h', checkpoint='sam_vit_h_4b8939.pth')
mask_decoder: MaskDecoder = sam.mask_decoder

# 2️⃣ Obtain image embeddings from the image encoder (omitted for brevity)

# image_embeddings: B×C×H×W, image_pe: same shape positional encodings

# 3️⃣ Build prompt embeddings (sparse & dense) – see segment_anything/modeling/prompt_encoder.py

# For demonstration, we use zeros:

B = 1
C, H, W = mask_decoder.transformer_dim, 64, 64
image_embeddings = torch.randn(B, C, H, W)
image_pe = torch.randn_like(image_embeddings)
sparse_prompt = torch.randn(B, 2, C)   # e.g., point + box token

dense_prompt = torch.randn(B, C, H, W) # optional mask input

# 4️⃣ Forward through the decoder

masks, iou_pred = mask_decoder(
    image_embeddings=image_embeddings,
    image_pe=image_pe,
    sparse_prompt_embeddings=sparse_prompt,
    dense_prompt_embeddings=dense_prompt,
    multimask_output=False,
)

print("Mask shape:", masks.shape)          # (B, 1, H_out, W_out)

print("IoU predictions:", iou_pred)       # (B, 1)

```

**Note:** In practice, you should reuse the `PromptEncoder` ([`segment_anything/modeling/prompt_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/prompt_encoder.py)) to obtain `sparse_prompt_embeddings` and `dense_prompt_embeddings`. The high-level `SamPredictor` orchestrates this automatically.

## Summary

- The **MaskDecoder** in [`segment_anything/modeling/mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/mask_decoder.py) implements a six-stage pipeline converting embeddings to masks via learned tokens, transformer processing, and hypernetwork weight generation.
- **Two transposed convolutions** upscale transformer outputs to full-resolution masks, while **hypernetwork MLPs** generate dynamic mask-specific weights.
- The **IoU token** provides quality estimation through a dedicated prediction head, enabling automatic ranking of mask candidates.
- **Multimask slicing** (lines 101-108) supports both single-output precision and multi-hypothesis diversity through boolean flag control.
- The decoder integrates with **sparse prompts** (points/boxes) via token concatenation and **dense prompts** (input masks) via element-wise feature addition.

## Frequently Asked Questions

### How does the SAM mask decoder handle multiple mask candidates?

The decoder generates N+1 masks by default. When `multimask_output=False`, it returns only the first mask (index 0) via `slice(0, 1)`. When `multimask_output=True`, it returns masks 1 through N via `slice(1, None)`, providing diverse segmentations for ambiguous prompts as implemented at lines 101-108 of [`mask_decoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/mask_decoder.py).

### What is the purpose of hypernetworks in the mask decoder?

The hypernetworks are small MLPs defined in the `MLP` helper class that convert each mask token into dynamic convolution weights. According to the source code at lines 60-65, these weights multiply against upsampled image features through matrix multiplication (lines 43-45), enabling token-specific mask generation without requiring separate decoder networks for each output.

### How does the IoU token predict mask quality?

The IoU token corresponds to `hs[:, 0]` in the transformer output (lines 33-35). A dedicated MLP head processes this specific token to produce scalar IoU scores for each mask candidate (lines 67-70). These scores estimate how well the predicted mask aligns with actual object boundaries, allowing the system to select the highest quality segmentation from multiple candidates.

### What is the difference between sparse and dense prompt embeddings in the decoder?

**Sparse prompt embeddings** (points, boxes) concatenate with the IoU and mask tokens to form the transformer input sequence, processed at lines 21-24. **Dense prompt embeddings** (input masks) add element-wise to the repeated image embeddings before transformer processing via `src = src + dense_prompt_embeddings` at line 26, providing spatial guidance at full feature resolution rather than through the token sequence.