# How the SAM Image Encoder Works: ViT Architecture and Implementation

> Discover how the SAM image encoder, a Vision Transformer, creates dense embeddings from images using patch embedding, windowed attention, and a convolutional neck for Segment Anything.

- Repository: [Meta Research/segment-anything](https://github.com/facebookresearch/segment-anything)
- Tags: internals
- Published: 2026-03-07

---

**The SAM image encoder is a Vision Transformer (ViT) backbone that converts RGB images into dense embedding maps through patch embedding, transformer blocks with windowed attention, and a convolutional neck.**

The Segment Anything Model (SAM) from Meta's `facebookresearch/segment-anything` repository relies on a powerful image encoder to transform visual input into a feature space that enables zero-shot segmentation. Understanding how this **SAM image encoder** processes pixels through its hierarchical architecture is essential for customizing the model or optimizing inference performance. The implementation resides primarily in [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py) and follows the standard ViT paradigm with strategic modifications for computational efficiency.

## Architecture Overview

The encoder operates in three distinct stages to produce the final image embeddings consumed by the mask decoder.

### Patch Embedding Layer

The first stage converts raw pixels into sequence tokens using the `PatchEmbed` class defined at [lines 64-95](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L64-L95) of [`image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/image_encoder.py). This layer employs a 2D convolution with `kernel_size` and `stride` equal to `patch_size` (typically 16×16) to project non-overlapping image regions into vectors of dimension `embed_dim`.

```python
self.patch_embed = PatchEmbed(
    kernel_size=(patch_size, patch_size),
    stride=(patch_size, patch_size),
    in_chans=in_chans,
    embed_dim=embed_dim,
)

```

The output tensor maintains spatial organization with shape **B × H' × W' × C**, where H' and W' represent the downsampled spatial dimensions (e.g., 64×64 for a 1024×1024 input with patch size 16).

### Transformer Encoder Blocks

Following patch embedding, a stack of `Block` modules processes the tokens. The `ImageEncoderViT.__init__` method constructs these blocks at [lines 18-88](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L18-L88), with depth varying by model size (12, 24, or 32 layers). Each `Block` class ([lines 19-85](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L19-L85)) contains:

- A **self-attention** layer operating globally or within local windows
- Optional **relative positional embeddings** when `use_rel_pos=True`
- An MLP block with expansion ratio `mlp_ratio` (default 4×)

```python
x = self.norm1(x)
x = self.attn(x)                 # self-attention (global or window)

x = shortcut + x
x = x + self.mlp(self.norm2(x)) # feed-forward

```

### Convolutional Neck

After the final transformer layer, the `neck` module (defined at [lines 88-104](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L88-L104)) reshapes the token tensor from **B × H' × W' × C** to **B × C × H' × W'** and applies two 3×3 convolutions with `LayerNorm2d` normalization. This produces the final 256-channel image embedding that the prompt encoder and mask decoder query.

## Positional Encoding Mechanisms

The SAM image encoder employs two complementary strategies for spatial information.

### Absolute Positional Embeddings

Learnable absolute positional embeddings are stored in `self.pos_embed` (shape **1 × H' × W' × C**) and added to patch tokens immediately after embedding. This provides global spatial reference points throughout the network.

### Relative Positional Embeddings

When `use_rel_pos=True`, each `Block` initializes `rel_pos_h` and `rel_pos_w` tables within the `Attention` class ([lines 15-23](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L15-L23)). The `add_decomposed_rel_pos` function ([lines 33-61](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py#L33-L61)) injects these 1D relative position biases into the attention computation, enabling fine-grained spatial awareness without the memory cost of full 2D tables.

## Window Attention vs Global Attention

To balance computational efficiency and receptive field, the encoder implements a hybrid attention pattern controlled by `window_size` and `global_attn_indexes`.

**Window Attention**: For most layers, tokens are partitioned into non-overlapping windows (default size 14×14) using `window_partition`, restricting self-attention to local neighborhoods. This reduces complexity from quadratic in sequence length to quadratic in window size.

**Global Attention**: Specific layers indicated by `global_attn_indexes` (e.g., layers 2, 5, 8, 11 in the base configuration) maintain full global self-attention, allowing information integration across the entire image.

## Implementation Details and Code Examples

### Extracting Image Embeddings

To access the dense embeddings directly from the SAM image encoder:

```python
from segment_anything.build_sam import build_sam
import torch

# Build the default ViT-H model (depth 32, embed_dim 1280)

sam = build_sam()

# Preprocess dummy image (3×1024×1024)

img = torch.randn(3, 1024, 1024)
preprocessed = sam.preprocess(img).unsqueeze(0)  # (1, 3, 1024, 1024)

# Forward through image encoder only

with torch.no_grad():
    image_embedding = sam.image_encoder(preprocessed)  # (1, 256, 64, 64)

print(image_embedding.shape)  # torch.Size([1, 256, 64, 64])

```

### Complete Inference Pipeline

For full segmentation with point prompts:

```python
from segment_anything.build_sam import build_sam
import torch

sam = build_sam()
sam.eval()

image = torch.randn(3, 1024, 1024)
point_coords = torch.tensor([[[512, 512]]])  # Center point

point_labels = torch.tensor([[1]])           # Foreground

batched_input = [{
    "image": image,
    "original_size": (1024, 1024),
    "point_coords": point_coords,
    "point_labels": point_labels,
}]

with torch.no_grad():
    out = sam(batched_input, multimask_output=False)

print(out[0]["masks"].shape)  # (1, 1, 1024, 1024)

```

### Model Configuration

The `build_sam` factory function ([lines 55-79](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py#L55-L79)) assembles the encoder with architecture-specific hyperparameters:

```python
image_encoder=ImageEncoderViT(
    depth=encoder_depth,
    embed_dim=encoder_embed_dim,
    img_size=image_size,
    mlp_ratio=4,
    norm_layer=partial(torch.nn.LayerNorm, eps=1e-6),
    num_heads=encoder_num_heads,
    patch_size=vit_patch_size,
    qkv_bias=True,
    use_rel_pos=True,
    global_attn_indexes=encoder_global_attn_indexes,
    window_size=14,
    out_chans=prompt_embed_dim,
)

```

## Summary

- The **SAM image encoder** uses a Vision Transformer architecture to convert 1024×1024 RGB images into 256-channel feature maps of size 64×64.
- **Patch embedding** occurs via convolutional projection at stride 16, implemented in the `PatchEmbed` class.
- The transformer stack alternates between **window attention** (local, efficient) and **global attention** (full image, at specific layers) to manage computational cost.
- **Relative positional embeddings** provide spatial awareness through decomposed 1D tables rather than expensive 2D position matrices.
- A lightweight **convolutional neck** post-processes transformer outputs into the final image embedding format required by the mask decoder.

## Frequently Asked Questions

### What is the output shape of the SAM image encoder?

For the default input size of 1024×1024 pixels, the SAM image encoder outputs a tensor of shape **(Batch, 256, 64, 64)**. This represents 256 channels at 1/16th the original spatial resolution, produced by the convolutional neck in [`image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/image_encoder.py) after processing through the ViT backbone.

### How does window attention reduce computation in SAM?

Window attention restricts self-attention computation to non-overlapping local windows (typically 14×14 tokens) rather than the full sequence. This reduces complexity from O(N²) to O(N×w²), where w is the window size, significantly lowering memory usage and computation time for high-resolution images while maintaining local spatial coherence.

### What is the difference between the ViT-H, ViT-L, and ViT-B SAM variants?

The variants differ in transformer depth and embedding dimension: **ViT-H** (Huge) uses 32 layers with 1280 dimensions, **ViT-L** (Large) uses 24 layers with 1024 dimensions, and **ViT-B** (Base) uses 12 layers with 768 dimensions. All variants maintain the same patch size (16×16) and output 256-channel embeddings, but larger variants capture richer features at the cost of increased latency and memory.

### Where is the image encoder defined in the Segment Anything repository?

The core implementation resides in [`segment_anything/modeling/image_encoder.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/image_encoder.py), containing the `ImageEncoderViT` class, `PatchEmbed`, `Block`, and attention mechanisms. The factory function that instantiates the encoder with appropriate hyperparameters is located in [`segment_anything/build_sam.py`](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py) at [lines 55-79](https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/build_sam.py#L55-L79).