How the SAM Image Encoder Works: ViT Architecture and Implementation

The SAM image encoder is a Vision Transformer (ViT) backbone that converts RGB images into dense embedding maps through patch embedding, transformer blocks with windowed attention, and a convolutional neck.

The Segment Anything Model (SAM) from Meta's facebookresearch/segment-anything repository relies on a powerful image encoder to transform visual input into a feature space that enables zero-shot segmentation. Understanding how this SAM image encoder processes pixels through its hierarchical architecture is essential for customizing the model or optimizing inference performance. The implementation resides primarily in segment_anything/modeling/image_encoder.py and follows the standard ViT paradigm with strategic modifications for computational efficiency.

Architecture Overview

The encoder operates in three distinct stages to produce the final image embeddings consumed by the mask decoder.

Patch Embedding Layer

The first stage converts raw pixels into sequence tokens using the PatchEmbed class defined at lines 64-95 of image_encoder.py. This layer employs a 2D convolution with kernel_size and stride equal to patch_size (typically 16×16) to project non-overlapping image regions into vectors of dimension embed_dim.

self.patch_embed = PatchEmbed(
    kernel_size=(patch_size, patch_size),
    stride=(patch_size, patch_size),
    in_chans=in_chans,
    embed_dim=embed_dim,
)

The output tensor maintains spatial organization with shape B × H' × W' × C, where H' and W' represent the downsampled spatial dimensions (e.g., 64×64 for a 1024×1024 input with patch size 16).

Transformer Encoder Blocks

Following patch embedding, a stack of Block modules processes the tokens. The ImageEncoderViT.__init__ method constructs these blocks at lines 18-88, with depth varying by model size (12, 24, or 32 layers). Each Block class (lines 19-85) contains:

• A self-attention layer operating globally or within local windows
• Optional relative positional embeddings when use_rel_pos=True
• An MLP block with expansion ratio mlp_ratio (default 4×)

x = self.norm1(x)
x = self.attn(x)                 # self-attention (global or window)

x = shortcut + x
x = x + self.mlp(self.norm2(x)) # feed-forward

Convolutional Neck

After the final transformer layer, the neck module (defined at lines 88-104) reshapes the token tensor from B × H' × W' × C to B × C × H' × W' and applies two 3×3 convolutions with LayerNorm2d normalization. This produces the final 256-channel image embedding that the prompt encoder and mask decoder query.

Positional Encoding Mechanisms

The SAM image encoder employs two complementary strategies for spatial information.

Absolute Positional Embeddings

Learnable absolute positional embeddings are stored in self.pos_embed (shape 1 × H' × W' × C) and added to patch tokens immediately after embedding. This provides global spatial reference points throughout the network.

Relative Positional Embeddings

When use_rel_pos=True, each Block initializes rel_pos_h and rel_pos_w tables within the Attention class (lines 15-23). The add_decomposed_rel_pos function (lines 33-61) injects these 1D relative position biases into the attention computation, enabling fine-grained spatial awareness without the memory cost of full 2D tables.

Window Attention vs Global Attention

To balance computational efficiency and receptive field, the encoder implements a hybrid attention pattern controlled by window_size and global_attn_indexes.

Window Attention: For most layers, tokens are partitioned into non-overlapping windows (default size 14×14) using window_partition, restricting self-attention to local neighborhoods. This reduces complexity from quadratic in sequence length to quadratic in window size.

Global Attention: Specific layers indicated by global_attn_indexes (e.g., layers 2, 5, 8, 11 in the base configuration) maintain full global self-attention, allowing information integration across the entire image.

Implementation Details and Code Examples

Extracting Image Embeddings

To access the dense embeddings directly from the SAM image encoder:

from segment_anything.build_sam import build_sam
import torch

# Build the default ViT-H model (depth 32, embed_dim 1280)

sam = build_sam()

# Preprocess dummy image (3×1024×1024)

img = torch.randn(3, 1024, 1024)
preprocessed = sam.preprocess(img).unsqueeze(0)  # (1, 3, 1024, 1024)

# Forward through image encoder only

with torch.no_grad():
    image_embedding = sam.image_encoder(preprocessed)  # (1, 256, 64, 64)

print(image_embedding.shape)  # torch.Size([1, 256, 64, 64])

Complete Inference Pipeline

For full segmentation with point prompts:

from segment_anything.build_sam import build_sam
import torch

sam = build_sam()
sam.eval()

image = torch.randn(3, 1024, 1024)
point_coords = torch.tensor([[[512, 512]]])  # Center point

point_labels = torch.tensor([[1]])           # Foreground

batched_input = [{
    "image": image,
    "original_size": (1024, 1024),
    "point_coords": point_coords,
    "point_labels": point_labels,
}]

with torch.no_grad():
    out = sam(batched_input, multimask_output=False)

print(out[0]["masks"].shape)  # (1, 1, 1024, 1024)

Model Configuration

The build_sam factory function (lines 55-79) assembles the encoder with architecture-specific hyperparameters:

image_encoder=ImageEncoderViT(
    depth=encoder_depth,
    embed_dim=encoder_embed_dim,
    img_size=image_size,
    mlp_ratio=4,
    norm_layer=partial(torch.nn.LayerNorm, eps=1e-6),
    num_heads=encoder_num_heads,
    patch_size=vit_patch_size,
    qkv_bias=True,
    use_rel_pos=True,
    global_attn_indexes=encoder_global_attn_indexes,
    window_size=14,
    out_chans=prompt_embed_dim,
)

Summary

• The SAM image encoder uses a Vision Transformer architecture to convert 1024×1024 RGB images into 256-channel feature maps of size 64×64.
• Patch embedding occurs via convolutional projection at stride 16, implemented in the PatchEmbed class.
• The transformer stack alternates between window attention (local, efficient) and global attention (full image, at specific layers) to manage computational cost.
• Relative positional embeddings provide spatial awareness through decomposed 1D tables rather than expensive 2D position matrices.
• A lightweight convolutional neck post-processes transformer outputs into the final image embedding format required by the mask decoder.

Frequently Asked Questions

What is the output shape of the SAM image encoder?

For the default input size of 1024×1024 pixels, the SAM image encoder outputs a tensor of shape (Batch, 256, 64, 64). This represents 256 channels at 1/16th the original spatial resolution, produced by the convolutional neck in image_encoder.py after processing through the ViT backbone.

How does window attention reduce computation in SAM?

Window attention restricts self-attention computation to non-overlapping local windows (typically 14×14 tokens) rather than the full sequence. This reduces complexity from O(N²) to O(N×w²), where w is the window size, significantly lowering memory usage and computation time for high-resolution images while maintaining local spatial coherence.

What is the difference between the ViT-H, ViT-L, and ViT-B SAM variants?

The variants differ in transformer depth and embedding dimension: ViT-H (Huge) uses 32 layers with 1280 dimensions, ViT-L (Large) uses 24 layers with 1024 dimensions, and ViT-B (Base) uses 12 layers with 768 dimensions. All variants maintain the same patch size (16×16) and output 256-channel embeddings, but larger variants capture richer features at the cost of increased latency and memory.

Where is the image encoder defined in the Segment Anything repository?

The core implementation resides in segment_anything/modeling/image_encoder.py, containing the ImageEncoderViT class, PatchEmbed, Block, and attention mechanisms. The factory function that instantiates the encoder with appropriate hyperparameters is located in segment_anything/build_sam.py at lines 55-79.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}